Bayesian model selection (BMS) and Bayesian model justifiability analysis (BMJ) provide a statistically rigorous framework to compare competing conceptual models through the use of Bayesian model evidence (BME). However, BME-based analysis has two main limitations: (1) it’s powerless when comparing models with different data set sizes and/or types of data and (2) doesn’t allow to judge a model’s performance based on its posterior predictive capabilities. Thus, traditional BME-based approaches ignore useful data or models due to issue (1) or disregards Bayesian updating because of issue (2). To address these limitations, we advocate to include additional information-theoretic scores into BMS and BMJ analysis: expected log-predictive density (ELPD), relative entropy (RE) and information entropy (IE). Exploring the connection between Bayesian inference and information theory, we explicitly link BME and ELPD together with RE and IE to indicate the information flow in BMS and BMJ analysis. We show how to compute and interpret these scores alongside BME, and apply it in a model selection and similarity analysis framework. We test the methodology on a controlled 2D groundwater setup considering five competing conceptual models accompanied with different data sets. The results show how the information-theoretic scores complement BME by providing a more complete picture concerning the Bayesian updating process. Additionally, we present how both RE and IE can be used to objectively compare models that feature different data sets. Overall, the introduced Bayesian information-theoretic framework helps to avoid any potential loss of information and leads to an informed decision for model selection and similarity.