In this instance Agent-Based Modelling (ABM) and System Dynamics (SD) have been selected as the focus for identifying validation themes. These methods have been selected for the review since they are increasingly used in projections of technology substitution patterns, market disruptions, and the development of planning tools for large-scale events difficult to replicate under normal laboratory conditions \cite{Peres_2010,Chatterjee_1990,Pel_2011,RN949} (see chapter 2). To identify and categorise the many procedures available for demonstrating the validity of these forecasting methods, a multiple stage literature review was conducted, as shown in Fig. \ref{595505}. This process was applied to systematically detect the most frequently occurring terms linked to the validity of each modelling technique, as well as the credibility of disruptions predicted using these approaches. The inductive approach taken to identify validation themes combined a quantitative referencing of closely-situated key-words within the selected literature sources (in order to narrow the search results to the most relevant paragraphs of the text) with a qualitative cross-check of any over-lapping phrases, so as to eliminate any unnecessary duplicates. Additionally, a retrospective appraisal of several historical simulations (such as the annual air traffic forecasts made by the Department for Transport \cite{govuk}) is provided in section \ref{487159} to illustrate some of the known challenges already observed when reviewing previous attempts at modelling the future. Using the procedure outlined in Fig. \ref{595505} (discussed in more detail in section \ref{618494}) provided a combined list of 50 themes relating to the use of computer-generated simulations in forecasting, including themes more specific to their reliability in modelling disruptive changes. These themes were subsequently used to structure a survey assessing the relative importance of each validation category for different audiences. In doing so this survey was intended to obtain empirical data on how academic, commercial, industrial, and public audiences relate to forecasts built on computer modelling, and how the credibility of forecasts may be affected when these same models project disruptive changes. Consequently, this survey focused on determining opinions relating to a) the modelling and simulation methods used to forecast future events, b) the perceived effectiveness of each validation theme for proving the value of a given forecast, and c) the requirements that must be satisfied to establish credibility of predicted disruptive changes. By comparing the real-world perspectives obtained in the survey to the categories identified from the literature analysis, this exploratory review provides an identification and ranking of the most effective means of validating forecasting techniques, as well as the prediction features that are most divisive to different audiences, outlined in section \ref{788541}.

Retrospective view of simulation challenges

Generating forecasts through modelling and simulation poses several different challenges. One of the most commonly encountered challenges relates to the sensitivity of forecast results on the initial modelling assumptions made in developing the simulation. This is illustrated in Fig. \ref{112371} to Fig. \ref{817587} in terms of the degree of variability observed in air traffic forecasts as a consequence of changing initial economic and operating condition assumptions. In the case of Fig. \ref{112371} and Fig. \ref{444497} the annual UK Department for Transport (DfT) forecasts are found to perform well versus actual air traffic growth during periods of relative stability, but significant errors appear when major shocks are encountered (with the average error shifting from 1.6% between 2003 and 2007, to greater than 30% between 2007 and 2012 following the economic crisis \cite{govuk}). In this instance this shows that financial disruptions had a far bigger impact on forecasting accuracy than terrorist attacks for air transportation, demonstrating the uncertainty present in forecasting any sector closely linked to the state of the economy. Sensitivity studies conducted by DfT give an indication of the dependency, and very broad range of possible outcomes, for the forecast number of UK terminal passengers, arising solely from variations in assumed oil prices and national GDP (see Fig. \ref{444497}). As such, the real-life outcome appears more in line with the low GDP and high oil price scenario shown retrospectively in Fig. \ref{444497}, illustrating the importance of selecting the right initial conditions. In a separate study, the UK Airports Commission have compared the impact of assuming unconstrained and constrained traffic growth on forecasts of airport capacity, as shown in Fig. \ref{780282} and Fig. \ref{817587} \cite{8bk9uk}. From this analysis it was identified that under an assumption of no capacity constraints (i.e. continued infrastructure development without obstacles), there was a relatively minor impact on the total traffic predicted across the airports studied. However, with the introduction of capacity constraints into the model there are some notable changes in forecast results (see Fig. \ref{817587}): whilst there is a small percent of growth continuing at heavily constrained airports in this condition (due to some continued development in capability), these airports are now expanding behind overall market growth. Traffic lost at these constrained airports now spills over to adjacent airports, leading to greater traffic growth in these regions whilst the aggregated traffic levels remain approximately the same as the unconstrained growth trend (in line with general economic trends) \cite{8bk9uk}. As such, both of these sensitivity studies illustrate strong divergence from reality based on the initial modelling assumptions made at the time.