In recent years, there has been a growing interest in using image super-resolution (SR) techniques in remote sensing. These techniques aim to reconstruct high-resolution (HR) imagery from low-resolution (LR) sources. Despite the development of sophisticated SR methodologies, determining what constitutes 'good' SR is still a matter of debate. Present-day literature often presents SR models through a strong computer vision perspective, heavily relying on synthetic datasets. Moreover, commonly used metrics often prioritize attributes that do not necessarily correspond to improvements in spatial resolution. To address this challenge, we present OpenSR-test, a comprehensive benchmark designed exclusively for evaluating SR of remote sensing images. Our framework incorporates specific quality metrics and curated cross-sensor datasets, each spanning various scale factors with consistent metadata. Utilizing OpenSR-test, we evaluate state-ofthe-art SR algorithms from a remote sensing perspective. The OpenSR-test framework and datasets are publicly available at https://esaopensr.github.io/opensr-test/.