6、Conclusions and discussions.

Three kinds of subsampling-ANOVA schemes (single-, multiple- and full-subsampling) have been proposed and analyzed in this study. The applicability of different subsampling ANOVA schemes are illustrated through one simplified model and a rainfall-runoff conceptual model. To evaluate the performance of different subsampling ANOVA schemes, the traditional sobol’s method is also used as benchmark in the study. The main purpose is to investigate the influence of different subsampling ANOVA schemes on sensitivity analyses results. Based on the case studies, some findings can be concluded:
1) The subsampling effectively diminishes the bias introduced by the biased variance estimator. In the application of subsampling ANOVA method, the parameter’s individual sensitivity is related to the subsampling scheme. The subsampling process will reduce the subsampled parameter’s individual sensitivity and increase the non-subsampled parameter’s individual sensitivity. In other words, the difference of sampling densities among parameters has great influence on quantification of parametric sensitivities in hydrologic modeling.
2) For full–subsampling ANOVA method, the deviation decreased with the parameters levels increased. The variation of the obtained parameters sensitivities is small and the order of parameters influences (i.e. sensitivity) would not change for three 3 or more parameter levels.
3) Compared with sobol’s method, the subsampling ANOVA methods can significantly reduce the calculation requirements while achieve similar calculation accuracy. Particularly, in order to get reliable parameter sensitivity results, the full-subsampling scheme is necessary, and the 3 or more parameter levels are recommended.
In this study, the sobol’s method is considered as the benchmark to evaluate the performance of the developed subsampling ANOVA approaches. Even though the subsampling ANOVA approaches may not produce better results than the sobol’s method, the proposed subsampling ANOVA approaches, especially for the full-subsampling ANOVA method, have their own essential strengths. Firstly, the sobol’s algorithm has high computational cost and the number of model evaluations required for the sobol’s indices to converge increases rapidly with the number of parameters, making its efficiency questionable for complex hydrological models (Herman et al., 2013, Zhang et al., 2013, Khorashadi Zadeh et al., 2017, Shin et al., 2013). In comparison, the subsampling ANOVA approaches can effectively reduce the computational demands and generate reliable results (as shown in Table 1 and Table 3). The number of model evaluations is equal to the number of combinations with all the parameter levels. However, as indicated in this paper, the full-subsampling ANOVA approach can generate acceptable results with three or four levels for each parameter. Thus, the computational cost would be reduced greatly. Secondly, besides sensitivity analysis for parameters with continuous values (Qi et al., 2016c), the single-subsampling ANOVA algorithms has already been applied to analyze the sensitivity of discrete or non-numeric elements such as the statistical post processing scheme, precipitation products and the hydrological model (Bosshard et al., 2013, Qi et al., 2016a, Qi et al., 2016b). Consequently, the developed multiple-/full-subsampling ANOVA approaches can also handle with sensitivity analysis for both numeric and non-numeric variables. However, the sobol’s approach can only deal with numeric variables.
The approaches proposed in this study just serve as a first basis for the application of subsampling ANOVA in hydrological model sensitivity analysis under multiple uncertainties. The number of levels would probably be higher to ensure robustness with a more complex model. The subsampling ANOVA algorithms can not only reduce the computing cost greatly, but also analyze the sensitivity of discrete or non-numeric elements. Further research is encouraged to examine the applicability of the subsampling ANOVA approaches in other non-numeric elements sensitivity analysis.