This comparative analysis of highest rankings suggests that some validation themes are seen as prerequisites for credible forecasting across all occupational groups. These include demonstrations of methodological rigour, providing evidence of traceability, elaborating the researcher’s subjectivity, and exploring the human factors that may or may not have an influence on the simulation. However, at the other end of the spectrum this analysis suggests academic communities place increased importance on error-checking and using hindsight as a basis for foresight, whilst commercial and public sector workers are more focused on impact locality and informativeness. This would fit with the intended rigour of academic processes as well as the day-to-day decision based nature of commercial and public sectors environments. Similarly, the focus on ‘Context’ within Industrial workplaces agrees with the highly product based mentality of this occupational subset (i.e. the need for targeting activities at an ‘end’ application). Although the compiled survey results agree well with the simple retrospective view of simulation challenges (presented in section \ref{487159}) in that the sensitivity of simulations to initial model conditions, competing environmental dynamics, and development assumptions are a high priority for audiences, this additional level of detail helps to more clearly define modelling expectations and requirements.
Consequences for the technology classification and substitutions models
From the exploratory review presented, there are several implications that can be taken forward in subsequent modelling and simulation activities. Firstly, beyond simply reproducing historical conditions, there is also a clear need to explore the sensitivity of any models constructed to assumed initial conditions. This provides a means to examine the degree of variability and uncertainty associated with any starting assumptions made, and to correctly identify the influence of these assumptions in determining model behaviour. To account for the impact of this variability on any conclusions made statistical ranking, benchmarking, and permutation testing methods are applied to the technology classification model developed in chapter 5, whilst sensitivity studies are presented for a range of both assumed and derived simulation parameters in the technology substitution model developed in chapter 6. Related to assumptions regarding initial conditions is the need to clearly define what influences will be considered internal and external to the model (i.e. the model scope), so as to provide the intended audience with appropriate expectations of model limitations. An initial outline of model scope appears from the problem structuring exercises presented in chapter 3. Taken in the more specific context of the technology classification model, this means recognising that the statistical approach taken is suitable for correlation analysis, but does not by itself provide a detailed causal understanding of events (the causal influence of socio-economic effects is examined through a historical review of the extracted technology data in chapter 5, but this refers to the input datasets rather than the model). Meanwhile for the technology substitution model the scope is established through the definition of essential model verification criteria in chapter 6, along with a detailed description of the model structure and incorporated features. Following on from the historical review of technology development datasets presented in chapter 5, an examination of the adoption data and model components corresponding to each technology is provided in chapter 6 to ensure that the datasets used in the model are representative of observed real-life effects.
The provisions listed above for the models developed in subsequent chapters could all be regarded as contributing towards methodological rigour, one of the key validation themes identified as being a prerequisite for all of the audiences considered. However, additional measures are required to satisfy the other prevalent validation themes identified. The review of the philosophical and methodological stances taken in this work, presented in chapter 3, is therefore required to provide background on the researcher's paradigm. Beyond this, these reflections provide an indication of any other human factors that could adversely influence the work beyond those typically accounted for by human-error (which are addressed separately through error-checking procedures employed in subsequent chapters). Traceability meanwhile is particularly important when attempting to reproduce any work based on modelling and simulation. In this study this is addressed through the disclosure of assumptions, datasets, methods, and tools used in the analysis where possible, amongst reviews of any evidence considered in structuring modelling decisions. For this reason, a comprehensive series of appendices is also included to enable the greatest extent of model reconstruction to an interested reader. Ultimately though, this is only considering those themes identified as being of most critical significance to demonstrating validity in modelling and simulation, but as Fig. \ref{222026} suggests, many other facets exist.
Statistical comparisons of time series
Having discussed the more general challenges presented to validity arising from modelling, this chapter now turns to specific challenges presented by time series data. This study considers 23 technologies, defined in Table 2 in chapter 5, where literature evidence has been identified to classify the particular mode of technology substitution observed. The evidence and process used in this categorisation is outlined in detail in section XX of chapter 2. Using bibliometric analysis methods it is possible to extract a variety of historical trends for any technologies of interest, effectively generating a collection of time series data points associated with a given technology (these multidimensional time series datasets are referred to here as 'technology profiles'). This raises the question of how best to compare dissimilar bibliometric technology profiles in an unbiased manner in order to investigate whether literature based technology substitution groupings can be determined using a classification system built on the assumptions given in section \ref{771448}. In particular comparisons of technology time series can be subject to one or more areas of dissimilarity: time series may be based on different number of observations (e.g. covering different time spans), be out of phase with each other, may be subject to long-term and shorter term cyclic trends, be at different stages through the Technology Life Cycle (or be fluctuating between different stages) \cite{little1981strategic}, or be representative of dissimilar industries. As such, a body of work already exists on the statistical comparison of time series, and in particular time series classification methods \cite{lin2012pattern}. Most modern time series pattern recognition and classification techniques emerging from the machine learning and data science domains broadly fall within the categories of supervised, semi-supervised, or unsupervised learning approaches. The distinction between these categories is based on the amount of training information provided to the classifier in each case. In supervised learning, training time series are provided with known classification labels, whilst training time series with both known and unknown classification labels are used in semi-supervised learning. By contrast, unsupervised learning approaches are not provided with any classification labels, and as such are required to determine groupings independently (e.g. clustering) \cite{lin2012pattern}. Table \ref{table:time_series_pattern_recognition_techniques} below provides an overview of time series pattern recognition techniques commonly used (this list is not exhaustive):