The magnetic field of the Sun drives its activity and variability. Understanding the evolution of the magnetic field is thus of great importance to dynamo and irradiance studies. While the evolution of the field in sunspot-bearing active regions has been extensively modelled, a realistic model of the evolution of small-scale magnetic regions is missing. We study the effect of small bipolar magnetic regions (BMRs) on the solar magnetic field by simulating the evolution of the total and open magnetic flux, the polar fields and the toroidal flux loss since 1874 with a surface flux transport model (SFTM). The emergence of all magnetic regions is described by a single power-law size distribution, whose exponent varies with solar activity, represented by the sunspot number. The spatial distribution of the emerging regions is described by statistical relationships derived from various observational data. We include regions as small as 2*10^20 Mx having lifetimes down to one day. The computed total magnetic flux is in good agreement with observations, even though in contrast to the earlier version of the SFTM we employ, we do not have a free parameter to adjust the total simulated flux to observations. The open flux, the polar fields and the toroidal flux loss are also consistent with observations and independent reconstructions. We find that small BMRs contribute about one third of the total and open flux at activity maxima, while at minima their contribution increases to roughly a half. Their impact on the polar fields and the toroidal flux loss is yet higher, with contributions from small BMRs and spot-containing active regions being comparable at all activity levels. Statistical differences between individual simulation runs are mostly attributed to large regions, while small BMRs have a stabilising effect on the simulations, especially for the polar field reversals. We conclude that small BMRs play an important role in the evolution of the solar magnetic field, increasing further at lower activity. Their inclusion in surface flux transport simulations will allow a better understanding of the generation of the poloidal field in the Babcock-Leighton dynamo and help constraining the secular variability of solar irradiance reconstructions.