- In total this analysis provided a combined list of 50 themes relating to the use of forecasting and computer-generated simulations (and their reliability in anticipating disruptions) that was subsequently used to structure a survey assessing the relative importance of each validation category for different audiences. In particular this survey was intended to obtain empirical data on how academic, commercial, industrial, and public audiences relate to forecasts and the use of simulated disruptions as a means of building credibility for new concepts. Consequently, this survey focused on determining opinions relating to a) the modelling and simulation methods used to forecast future events, b) the perceived effectiveness of each validation theme for proving the value of a given forecast, and c) the requirements that must be satisfied to establish credibility of predicted disruptive changes.
- By then comparing the real-world perspectives provided in the survey to the categories identified from the literature analysis, this study will identify and rank the most effective means of validating forecasting techniques, and the prediction features that are most divisive to different audiences.
Qualitative research strategy adopted to review the perceived credibility of simulation validation techniques (best suited for handling subjective topics with responses that vary based on experience/beliefs of individual participants)
Methodology outline:
- Review of historical simulations and current credibility
- Identification of key trends/themes in simulation validation methods (literature analysis)
- Mapping of identified themes to specific simulation techniques
- Structuring of survey questions from initial categories and themes identified
- Obtaining real-world perspectives through survey/group model building on: a)Establishing credibility in new domains, b)Uses of modelling and simulation techniques, c)Belief in methods of validation for simulation
- Comparison of real-world perspectives to categories identified by literature analysis
Retrospective view of simulation challenges
- Generating forecasts using modelling and simulation techniques poses several different challenges. One of the most commonly encountered challenges relates to the sensitivity of forecast results on the initial modelling assumptions made in developing the simulation. This is illustrated in Fig. 4 to Fig. 7 in terms of the retrospective impact on air traffic forecasts observed as a consequence of changing economic and operating conditions. In the case of Fig. 4 and Fig. 5 the annual UK Department for Transport (DfT) forecasts are found to perform well versus actual air traffic growth during periods of relative stability, but significant errors appear when major shocks are encountered (with the average error shifting from 1.6% between 2003 and 2007, to greater than 30% between 2007 and 2012 following the economic crisis [11]). In this instance this shows that financial disruptions had a far bigger impact on forecasting accuracy than terrorist attacks for air transportation, demonstrating the uncertainty present in forecasting any sector closely linked to the state of the economy. Sensitivity studies conducted by DfT give an indication of the dependency, and very broad range of possible outcomes, for the forecast number of UK terminal passengers, arising solely from variations in assumed oil prices and national GDP (see Fig. 5). In a separate study, the UK Airports Commission have compared the impact of assuming unconstrained and constrained traffic growth on forecasts of airport capacity, as shown in Fig. 6 and Fig. 7 [7]. From this analysis it was identified that under an assumption of no capacity constraints (i.e. continued infrastructure development without obstacles), there was a relatively minor impact on the total traffic predicted across the airports studied. However, with the introduction of capacity constraints into the model there are some notable changes in forecast results (see Fig. 7): whilst there is a small percent of growth continuing at heavily constrained airports in this condition (due to some continued development in capability), these airports are now expanding behind overall market growth. Traffic lost at these constrained airports now spills over to adjacent airports, leading to greater traffic growth in these regions whilst the aggregated traffic levels remain approximately the same as the unconstrained growth trend (in line with general economic trends) [7]. As such, both of these sensitivity studies illustrate strong divergence from reality based on the initial modelling assumptions made at the time.