The Sunspot Number series is one of the longest and most detailed available series in astrophysics. The series was first constructed in 1849 by Prof. Rudolf Wolf and a time series has been built in real time since then, involving a lot of observers who differ from each other in terms of their way of counting sunspots, different telescopes and eyesights. After the recalibration from 2015 (Clette et al., 2016), the series still suffers from various scale discrepancies which would benefit from a complete reconstruction (for example because of a jump in 1849). We will briefly present the data recovery efforts, which enabled the sunspot community to have many new datasets now at its disposal. One of the most reliable datasets is the logbook maintained exceptionally well by Prof. Wolf, known as the Mittheilungen (1610-1918) – which has been recently digitized at Royal Observatory of Belgium (ROB) and is constantly quality-controlled and consequently updated. In addition to that, Wolf’s original source books were recently digitized (1849-1877), and can help with the missing information in Mittheilungen. The ETH library in Zurich also recently started scanning sunspot observations tables from 1918 to 1980, and these are being converted to readable format by collaborations of the SILSO (Sunspot Index and Long-term Solar Observations) team at ROB. Here we will present the exploitation of all the available databases along with other available recounts of various observers to identify scale discrepancies or inhomogeneities that happened along the Sunspot Number series over time. We also introduce statistical techniques to stitch together all these recovered data and implement confidence bands or errors on daily Sunspot Numbers, an information that existing versions lack. The long-term aim is a complete reconstruction of the Sunspot Number series from the available raw data instead of a recalibration.