Paleomagnetic poles used to compute apparent polar wander paths (APWPs) are strongly dispersed, which was recently shown to cause a large fraction (>50%) of these poles to be statistically distinct from the APWP to which they contributed, suggesting that current statistical approaches overestimate paleomagnetic resolution. Here, we analyze why coeval paleopoles are so dispersed, using the paleopoles behind the most recent global APWP and a compilation of paleomagnetic data obtained from <10 Ma volcanic rocks (PSV10). We find that paleopoles derived from sedimentary rocks, or from data sets underrepresenting paleosecular variation (PSV), are more dispersed and more frequently displaced. We show that paleopoles based on a smaller number of paleomagnetic sites are more dispersed than poles based on larger data sets, revealing that the degree to which PSV is averaged is an important contributor to the pole dispersion. We identify as fundamental problem, however, that the amount of paleomagnetic data used to calculate a paleopole, and thus the dispersion of coeval paleopoles, is essentially arbitrary. We therefore explore a different approach in which reference poles of APWPs are calculated from site-level data instead of paleopoles, thereby assigning larger weight to larger data sets. We introduce a bootstrap-based method for comparing a collection of paleomagnetic data with a reference data set on the same hierarchical level, whereby the uncertainty is weighted against the number of paleomagnetic sites. Finally, our study highlights that demonstrating smaller tectonic displacements requires larger paleomagnetic data sets, and that such data sets can strongly improve future APWPs.