Developing and implementing a malaria early warning system that integrates public health surveillance with monitoring of related environmental factors is the goal of the Epidemic Prognosis Incorporating Disease and Environmental Monitoring for Integrated Assessment (EPIDEMIA) project. Collaborating with our Ethiopian partners on requirements, we developed the R package epidemiar to provide a generalized set of functions for disease forecasting, plus customized code including a Google Earth Engine script for environmental data and formatting scripts for distributable reports with maps and graphs. Since 2019, a local team at Bahir Dar University in Ethiopia has been using EPIDEMIA to produce weekly malaria forecasting reports. Intensive anti-malarial efforts in the Amhara region of Ethiopia have resulted in declining malaria incidence, with a 75% decrease in cases between 2013 and 2018 (561,101 to 137,445 cases). In this context of potentially changing malaria transmission patterns, continual model evaluation past the initial model development is warranted. We built model validation and assessment tools into the epidemiar R package for on-demand evaluation for any historical period. Validation statistics included Mean Error (ME), Mean Absolute Error (MAE), and proportion of observations that fell inside the prediction intervals. Evaluation can be made for one through n-week ahead predictions, and include comparisons with two naïve models: persistence of last known value, and average cases from that week of the year. Building validation into the early warning system provides more opportunities to learn about the model via the validation results. We can identify locations where the models perform best with district-level results. With on-demand implementation and time-range flexibility, we can also investigate how accuracy changes over time, which is of particular interest in places like Ethiopia with changing patterns and declining trends due to anti-malarial programs.