One of the first questions posed by any form of market or technology forecasting is how credible are the final predictions made by the model? This brings to attention the question of model validity
Research strategy for identifying and ranking validation themes
Qualitative research strategy adopted to review the perceived credibility of simulation validation techniques (best suited for handling subjective topics with responses that vary based on experience/beliefs of individual participants)
Methodology outline:
- Review of historical simulations and current credibility
- Identification of key trends/themes in simulation validation methods (literature analysis)
- Mapping of identified themes to specific simulation techniques
- Structuring of survey questions from initial categories and themes identified
- Obtaining real-world perspectives through survey/group model building on: a)Establishing credibility in new domains, b)Uses of modelling and simulation techniques, c)Belief in methods of validation for simulation
- Comparison of real-world perspectives to categories identified by literature analysis
Example applications of modelling and simulation in aviation forecasts
Airbus, Boeing, Embraer, Bombardier publish forecasts for the next 20 years:
- Air transport (RPK) growth widely predicted, with GDP and urbanisation as key drivers
- Forecasts are superficially similar, but the differences are significant
- Fuel prices remain manageable by incremental efficiency improvements, without major disruptive changes
- Historically: growth is resilient to transient economic variations (still valid for Future?): a) 4-5% year-on-year in terms of RPK is forecast by most studies (Airbus GMF 4.7% average), b) Global traffic (RPK) doubles roughly every 15 years
Questions raised for Modelling and Simulation:
- Could aircraft types offered modify the networks and the markets?: a) Airports Commission suggests more efficient hub feeder aircraft can positively influence the development of a hub-and-spoke network, b) Co-operation (alliances) and harmonisation of climate legislation are also deemed to be key, c) Small short range aircraft can create a city pair sooner than larger aircraft
- How will disruptive technology modify market forecasts?
- Can models be a useful means to evaluate disruptive concepts?
+ additional notes on corresponding slide
Simulation vs. modelling assumptions
Copy images from slide and expand bullet points in terms of relevance of forecasting techniques:
- Assuming no capacity constraints has a relatively minor impact on total traffic
- A small % of growth will continue at constrained airports due to development in capability, but behind market growth
- Unconstrained growth continues in line with general economic trends, with some regional variation
- If there are capacity constraints in effect, other airports will experience greater traffic growth
Simulation vs. historical forecasts
Copy images from slide and expand bullet points in terms of relevance of forecasting techniques:
- Department for Transport forecasts perform well vs. actual air traffic growth during periods of relative stability, but significant errors appear when major shocks are encountered: a) 2003 – 2007 average error = 1.6%, b) 2007 – 2012 average error >30% (economic crisis)
- Financial disruptions had far bigger impact on forecasting accuracy than terrorist attacks: demonstrates uncertainty in forecasting any sector closely linked to state of economy
- Very broad range of possible outcomes can be predicted by just varying oil price and GDP variation
Simulation vs. historical disruptions
Copy images from slide and expand bullet points in terms of relevance of forecasting techniques:
Complexity of capturing real-life trends:
<Insert images>
Challenges:
- Model granularity required to capture real-world effects vs. model complexity
- Identification of model variables and methods that require different levels of granularity
- Availability of data to extract accurate models of real-world effects
- Definition of boundary between influences internal and external to modelled system
- Capturing of emergent properties
- Calibration of models to multiple phenomena (robustness and generalisability)
Sensitivity to assumed initial conditions:
<Insert images>
Challenges:
- Sensitivity studies are necessary to determine robustness of results
- Simulations are often a good indicator of trends rather than exact values
Identification of validation themes for simulation methods
See slides and copy across from IEMS paper
Categorisation and ranking of validation themes for simulation methods
Copy across from IEMS paper
Scientometric and bibliometric analysis methods
See notes in ‘Verification and validation plan.txt’, notes on ' Scientometric and bibliometric analysis methods ' slide, papers listed in ‘Papers to go through in detail for thesis.txt’, and papers in ‘Measures of technological progress’ folder
Is some of this already covered in the literature review???
Statistical comparisons of time series
This study considers 23 technologies, defined in Table 2 in chapter 5, where literature evidence has been identified to classify the particular mode of technology substitution observed. The evidence and process used in this categorisation is outlined in detail in section XX of chapter 2. Using bibliometric analysis methods it is possible to extract a variety of historical trends for any technologies of interest, effectively generating a collection of time series data points associated with a given technology (these multidimensional time series datasets are referred to here as 'technology profiles'). This raises the question of how best to compare dissimilar bibliometric technology profiles in an unbiased manner in order to investigate whether literature based technology substitution groupings can be determined using a classification system built on the assumptions given in section \ref{771448}. In particular comparisons of technology time series can be subject to one or more areas of dissimilarity: time series may be based on different number of observations (e.g. covering different time spans), be out of phase with each other, may be subject to long-term and shorter term cyclic trends, be at different stages through the Technology Life Cycle (or be fluctuating between different stages) \cite{little1981strategic}, or be representative of dissimilar industries. As such, a body of work already exists on the statistical comparison of time series, and in particular time series classification methods \cite{lin2012pattern}. Most modern time series pattern recognition and classification techniques emerging from the machine learning and data science domains broadly fall within the categories of supervised, semi-supervised, or unsupervised learning approaches. The distinction between these categories is based on the amount of training information provided to the classifier in each case. In supervised learning, training time series are provided with known classification labels, whilst training time series with both known and unknown classification labels are used in semi-supervised learning. By contrast, unsupervised learning approaches are not provided with any classification labels, and as such are required to determine groupings independently (e.g. clustering) \cite{lin2012pattern}. Table \ref{table:time_series_pattern_recognition_techniques} below provides an overview of time series pattern recognition techniques commonly used (this list is not exhaustive):