Relative model score: a scoring rule for evaluating ensemble simulations with application to microbial soil respiration modeling

This paper defines a new scoring rule, namely relative model score (RMS), for evaluating ensemble simulations of environmental models. RMS implicitly incorporates the measures of ensemble mean accuracy, prediction interval precision, and prediction interval reliability for evaluating the overall model predictive performance. RMS is numerically evaluated from the probability density functions of ensemble simulations given by individual models or several models via model averaging. We demonstrate the advantages of using RMS through an example of soil respiration modeling. The example considers two alternative models with different fidelity, and for each model Bayesian inverse modeling is conducted using two different likelihood functions. This gives four single-model ensembles of model simulations. For each likelihood function, Bayesian model averaging is applied to the ensemble simulations of the two models, resulting in two multi-model prediction ensembles. Predictive performance for these ensembles is evaluated using various scoring rules. Results show that RMS outperforms the commonly used scoring rules of log-score, pseudo Bayes factor based on Bayesian model evidence (BME), and continuous ranked probability score (CRPS). RMS avoids the problem of rounding error specific to log-score. Being applicable to any likelihood functions, RMS has broader applicability than BME that is only applicable to the same likelihood function of multiple models. By directly considering the relative score of candidate models at each cross-validation datum, RMS results in more plausible model ranking than CRPS. Therefore, RMS is considered as a robust scoring rule for evaluating predictive performance of single-model and multi-model prediction ensembles.     

Assessing the impact of land use change on hydrology by ensemble modeling (LUCHEM). I: Model intercomparison with current land use

This paper introduces the project on ‘Assessing the impact of land use change on hydrology by ensemble modeling (LUCHEM)’ that aims at investigating the envelope of predictions on changes in hydrological fluxes due to land use change. As part of a series of four papers, this paper outlines the motivation and setup of LUCHEM, and presents a model intercomparison for the present-day simulation results. Such an intercomparison provides a valuable basis to investigate the effects of different model structures on model predictions and paves the ground for the analysis of the performance of multi-model ensembles and the reliability of the scenario predictions in companion papers. In this study, we applied a set of 10 lumped, semi-lumped and fully distributed hydrological models that have been previously used in land use change studies to the low mountainous Dill catchment, Germany. Substantial differences in model performance were observed with Nash–Sutcliffe efficiencies ranging from 0.53 to 0.92. Differences in model performance were attributed to (1) model input data, (2) model calibration and (3) the physical basis of the models. The models were applied with two sets of input data: an original and a homogenized data set. This homogenization of precipitation, temperature and leaf area index was performed to reduce the variation between the models. Homogenization improved the comparability of model simulations and resulted in a reduced average bias, although some variation in model data input remained. The effect of the physical differences between models on the long-term water balance was mainly attributed to differences in how models represent evapotranspiration. Semi-lumped and lumped conceptual models slightly outperformed the fully distributed and physically based models. This was attributed to the automatic model calibration typically used for this type of models. Overall, however, we conclude that there was no superior model if several measures of model performance are considered and that all models are suitable to participate in further multi-model ensemble set-ups and land use change scenario investigations.     

Evaluating the contribution of multi-model combination to streamflow hindcasting by empirical and conceptual models

The contribution of multi-model combination to daily streamflow hindcasting was evaluated through the HBV (Hydrologiska Byråns Vattenbalansavdelning) and RNN (recurrent neural networks) models with 100 ensemble members generated with different initial conditions for both. In the calibration phase, the analysis showed that the HBV and RNN models with 20 members have better accuracy and require less calibration time. The combination of two models, however, did not provide significant improvements when 80 more members were added in the combination. In the validation phase, the results indicated that both HBV and RNN models with 20 members not only accurately produce reliable and stable streamflow hindcasting, but also effectively simulate the timing and the value of peak flows. From the consistency of calibration and validation results, the study provides an important contribution, namely, that ensemble size is not sensitive to the type of hydrological model in terms of streamflow hindcasting.     

Investigating the application of climate models in flood projection across the UK

Regional climate models (RCMs) have emerged as the preferred tool in hydrological impact assessment at the catchment scale. The direct application of RCM precipitation output is still not recommended; instead, a number of alternative methods have been proposed. One method that has been used is the change factor methodology, which typically uses changes to monthly mean or seasonal precipitation totals to develop change scenarios. However, such simplistic approaches are subject to significant caveats. In this paper, 18 RCMs covering the UK from the ENSEMBLES and UKCP09 projects are analysed across different catchments. The ensembles' ability in capturing monthly total and extreme precipitation is outlined to explore how the ability to make confident statements about future flood risk varies between different catchments. The suitability of applying simplistic change factor approaches in flood impact studies is also explored. We found that RCM ensembles do have some skill in simulating observed monthly precipitation; however, seasonal patterns of bias were evident across each of the catchments. Moreover, even apparently good simulations of extreme rainfall can mis‐estimate the magnitude of flood‐generating rainfall events in ways that would significantly affect flood risk management. For future changes in monthly mean precipitation, we observe the clear ‘drier summers/wetter winters’ signal used to develop current UK policy, but when we look instead at flood‐generating rainfall, this seasonal signal is less clear and greater increases are projected. Furthermore, the confidence associated with future projections varies from catchment to catchment and season to season as a result of the varying ability of the RCM ensembles, and in some cases, future flood risk projections using RCM outputs may be highly problematic. Copyright © 2013 John Wiley & Sons, Ltd.     

Performance comparison of multiple and single surrogate models for pumping optimization of coastal aquifers

Single and multiple surrogate models were compared for single-objective pumping optimization problems of a hypothetical and a real-world coastal aquifer. Different instances of radial basis functions and kriging surrogates were utilized to reduce the computational cost of direct optimization with variable density and salt transport models. An adaptive surrogate update scheme was embedded in the operations of an evolutionary algorithm to efficiently control the feasibility of optimal solutions in pumping optimization problems with multiple constraints. For a set of independent optimization runs, results showed that multiple surrogates, either by selecting the best or by using ensembles, did not necessarily outperform the single surrogate approach. Nevertheless, the ensemble with optimal weights produced slightly better results than selecting only the best surrogates or applying a simple averaging approach. For all cases, the computational cost, by using single or multiple surrogate models, was reduced by up to 90% of the direct optimization.     

Multi-model averaging for continuous streamflow prediction in ungauged basins

ABSTRACT

This paper assesses the possibility of using multi-model averaging techniques for continuous streamflow prediction in ungauged basins. Three hydrological models were calibrated on the Nash-Sutcliffe Efficiency metric and were used as members of four multi-model averaging schemes. Model weights were estimated through optimization on the donor catchments. The averaging methods were tested on 267 catchments in the province of Québec, Canada, in a leave-one-out cross-validation approach. It was found that the best hydrological model was practically always better than the others used individually or in a multi-model framework, thus no averaging scheme performed statistically better than the best single member. It was also found that the robustness and adaptability of the models were highly influential on the models’ performance in cross-verification. The results show that multi-model averaging techniques are not necessarily suited for regionalization applications, and that models selected in such studies must be chosen carefully to be as robust as possible on the study site.

Editor M.C. Acreman; Associate editor S. Grimaldi     

Ensemble prediction of regional droughts using climate inputs and the SVM–copula approach

In this study, the climate teleconnections with meteorological droughts are analysed and used to develop ensemble drought prediction models using a support vector machine (SVM)–copula approach over Western Rajasthan (India). The meteorological droughts are identified using the Standardized Precipitation Index (SPI). In the analysis of large‐scale climate forcing represented by climate indices such as El Niño Southern Oscillation, Indian Ocean Dipole Mode and Atlantic Multidecadal Oscillation on regional droughts, it is found that regional droughts exhibits interannual as well as interdecadal variability. On the basis of potential teleconnections between regional droughts and climate indices, SPI‐based drought forecasting models are developed with up to 3 months' lead time. As traditional statistical forecast models are unable to capture nonlinearity and nonstationarity associated with drought forecasts, a machine learning technique, namely, support vector regression (SVR), is adopted to forecast the drought index, and the copula method is used to model the joint distribution of observed and predicted drought index. The copula‐based conditional distribution of an observed drought index conditioned on predicted drought index is utilized to simulate ensembles of drought forecasts. Two variants of drought forecast models are developed, namely a single model for all the periods in a year and separate models for each of the four seasons in a year. The performance of developed models is validated for predicting drought time series for 10 years' data. Improvement in ensemble prediction of drought indices is observed for combined seasonal model over the single model without seasonal partitions. The results show that the proposed SVM–copula approach improves the drought prediction capability and provides estimation of uncertainty associated with drought predictions. Copyright © 2013 John Wiley & Sons, Ltd.     

Improving the predictions of a MIKE SHE catchment‐scale application by using a multi‐criteria approach

This paper suggests a multi‐criteria protocol for appropriately evaluating the predictions of hydrologic models during calibration and evaluation stages. The protocol includes different statistical, analytical and visual criteria such as analysis of peak and low flows, cumulative volumes, extreme value statistics, performance statistics, etc. Furthermore, the protocol assesses the physical consistency of model predictions by filtering the total observed hydrograph into different flow‐components (baseflow, interflow and overland flow) and using these filtered data in the calibration and evaluation processes. Based on the distributed modelling of a medium size catchment, it is shown that application of the suggested protocol, and in particular the use of the filtered flow‐components in model calibration, enhances the physical consistency of model predictions, adding considerable value to the calibration process. Copyright © 2007 John Wiley & Sons, Ltd.     

Climate impacts on river flow: projections for the Medway catchment,UK, with UKCP09 and CATCHMOD

The potential impact of climate change on areas of strategic importance for water resources remains a concern. Here, river flow projections for the River Medway, above Teston in southeast England are presented, which is just such an area of strategic importance. The river flow projections use climate inputs from the Hadley Centre Regional Climate Model (HadRM3) for the time period 1960–2080 (a subset of the early release UKCP09 projections). River flow predictions are calculated using CATCHMOD, the main river flow prediction tool of the Environment Agency (EA) of England and Wales. In order to use this tool in the best way for climate change predictions, model setup and performance are analysed using sensitivity and uncertainty analysis. The model's representation of hydrological processes is discussed and the direct percolation and first linear storage constant parameters are found to strongly affect model results in a complex way, with the former more important for low flows and the latter for high flows. The uncertainty in predictions resulting from the hydrological model parameters is demonstrated and the projections of river flow under future climate are analysed. A clear climate change impact signal is evident in the results with a persistent lowering of mean daily river flows for all months and for all projection time slices. Results indicate that a projection of lower flows under future climate is valid even taking into account the uncertainties considered in this modelling chain exercise. The model parameter uncertainty becomes more significant under future climate as the river flows become lower. This has significant implications for those making policy decisions based on such modelling results. Copyright © 2010 John Wiley & Sons, Ltd.     

Combining climate model output via model correlations

In climate science, collections of climate model output, usually referred to as ensembles, are commonly used devices to study uncertainty in climate model experiments. The ensemble members may reflect variation in initial conditions, different physics implementations, or even entirely different climate models. However, there is a need to deliver a unified product based on the ensemble members that reflects the information contained in whole of the ensemble. We propose a technique for creating linear combinations of ensemble members where the weights are constructed from estimates of variation and correlation both within and between ensemble members. At the heart of this approach is a Bayesian hierarchical model that allows for estimation of the correlation between ensemble members as well as the study of the impact of uncertainty in the parameter estimates of the hierarchical model on the weights. The approach is demonstrated on an ensemble of regional climate model (RCM) output.     

A comparison of three approaches to spatial generalization of rainfall–runoff models

The generalization of the parameters of rainfall–runoff models, to enable application at ungauged sites, is an important and ongoing area of research. This paper compares the performance of three alternative methods of generalization, for two parameter‐sparse conceptual models (PDM and TATE), specifically for use in flood frequency estimation using continuous simulation. Two of the methods are based on fitting regression relationships between catchment properties and calibrated parameter values, using weighted or sequential regression (with weights based on estimates of calibration uncertainty), and the third is based on the use of pooling groups, defined through measures of site‐similarity based on catchment properties. The study uses a relatively large sample of catchments in Britain. For the PDM, the site‐similarity method performs best, but not greatly better than either regression method, so there may be cases where the use of regression would be preferable. For the TATE model, weighted regression performs best (with a very similar level of performance to that of the PDM with site‐similarity), whereas site‐similarity performs worst (due to poor performance for catchments with higher baseflow), indicating that the choice of model and generalization method should not be separated. The use of sequential regression, which was developed to try to allow for parameter interdependence, shows no clear advantage for either model. Other than the poor performance of the TATE model with site‐similarity for catchments with a higher baseflow index, there are no clear relationships between performance of any model/method and catchment type. Copyright © 2006 John Wiley & Sons, Ltd.     

Rainfall‐runoff modelling using artificial neural networks technique: a Blue Nile catchment case study

A rainfall‐runoff model based on an artificial neural network (ANN) is presented for the Blue Nile catchment. The best geometry of the ANN rainfall‐runoff model in terms of number of hidden layers and nodes is identified through a sensitivity analysis. The Blue Nile catchment (about 300 000 km²) in the Nile basin is selected here as a case study. The catchment is classified into seven subcatchments, and the mean areal precipitation over those subcatchments is computed as a main input to the ANN model. The available daily data (1992–99) are divided into two sets for model calibration (1992–96) and for validation (1997–99). The results of the ANN model are compared with one of physical distributed rainfall‐runoff models that apply hydraulic and hydrologic fundamental equations in a grid base. The results over the case study area and the comparative analysis with the physically based distributed model show that the ANN technique has great potential in simulating the rainfall‐runoff process adequately. Because the available record used in the calibration of the ANN model is too short, the ANN model is biased compared with the distributed model, especially for high flows. Copyright © 2005 John Wiley & Sons, Ltd.     

Assessment of the adequacy of the regional relationships between catchment attributes and catchment response dynamics,calibrated by a multi‐objective approach

This study aimed to evaluate the effectiveness of the regionalization method on the basis of a combination of a parsimonious model structure and a multi‐objective calibration technique. For this study, 12 gauged catchments in the Republic of Korea were used. The parsimonious model structure, requiring minimal input data, was used to avoid adverse effects arising from model complexity, over‐parameterization and data requirements. The IHACRES rainfall‐runoff model was applied to represent the dynamic response characteristics of catchments in Korea. A multi‐objective approach was adopted to reduce the predictive uncertainty arising from the calibration of a rainfall‐runoff model, by increasing the amount of information retrieved from the available data. The regional relationships (or models) between the model parameters and the catchment attributes were established via a multiple regression approach, incorporating correlation analysis and stepwise regression on linear and logarithmic scales. The impacts of the parameters, calibrated by the multi‐objective approach, on the adequacy of regional relationships were assessed by comparison with impacts obtained by the single‐objective approach. The regional relationships were well defined, despite limited available data. The drainage area, the effective soil depth, the mean catchment slope and the catchment gradient appeared to be the main factors for describing the hydrologic response characteristics in the areas studied. The overall model performance of the regional models based on the multi‐objective approach was good, producing reasonable results for high and low flows and for the overall water balance, simultaneously. The regional models based on the single‐objective approach yielded accurate predictions in high flows but showed limited predictive capability for low flows and the overall water balance. This was due to the optimal model parameter estimates when using a single‐objective measure. The parameters calibrated by the single‐objective approach decreased the predictability of the regional models. Copyright © 2013 John Wiley & Sons, Ltd.     

Assessing hydrological model predictive uncertainty using stochastically generated geological models

In distributed and coupled surface water–groundwater modelling, the uncertainty from the geological structure is unaccounted for if only one deterministic geological model is used. In the present study, the geological structural uncertainty is represented by multiple, stochastically generated geological models, which are used to develop hydrological model ensembles for the Norsminde catchment in Denmark. The geological models have been constructed using two types of field data, airborne geophysical data and borehole well log data. The use of airborne geophysical data in constructing stochastic geological models and followed by the application of such models to assess hydrological simulation uncertainty for both surface water and groundwater have not been previously studied. The results show that the hydrological ensemble based on geophysical data has a lower level of simulation uncertainty, but the ensemble based on borehole data is able to encapsulate more observation points for stream discharge simulation. The groundwater simulations are in general more sensitive to the changes in the geological structure than the stream discharge simulations, and in the deeper groundwater layers, there are larger variations between simulations within an ensemble than in the upper layers. The relationship between hydrological prediction uncertainties measured as the spread within the hydrological ensembles and the spatial aggregation scale of simulation results has been analysed using a representative elementary scale concept. The results show a clear increase of prediction uncertainty as the spatial scale decreases. Copyright © 2015 John Wiley & Sons, Ltd.     

Predicting dry‐season flows with a monthly rainfall–runoff model: Performance for gauged and ungauged catchments

Hydrologic models are useful to understand the effects of climate and land‐use changes on dry‐season flows. In practice, there is often a trade‐off between simplicity and accuracy, especially when resources for catchment management are scarce. Here, we evaluated the performance of a monthly rainfall–runoff model (dynamic water balance model, DWBM) for dry‐season flow prediction under climate and land‐use change. Using different methods with decreasing amounts of catchment information to set the four model parameters, we predicted dry‐season flow for 89 Australian catchments and verified model performance with an independent dataset of 641 catchments in the United States. For the Australian catchments, model performance without catchment information (other than climate forcing) was fair; it increased significantly as the information to infer the four model parameters increased. Regressions to infer model parameters from catchment characteristics did not hold for catchments in the United States, meaning that a new calibration effort was needed to increase model performance there. Recognizing the interest in relative change for practical applications, we also examined how DWBM could be used to simulate a change in dry‐season flow following land‐use change. We compared results with and without calibration data and showed that predictions of changes in dry‐season flow were robust with respect to uncertainty in model parameters. Our analyses confirm that climate is a strong driver of dry‐season flow and that parsimonious models such as DWBM have useful management applications: predicting seasonal flow under various climate forcings when calibration data are available and providing estimates of the relative effect of land use on seasonal flow for ungauged catchments.     

Ensemble flood forecasting: A review

Operational medium range flood forecasting systems are increasingly moving towards the adoption of ensembles of numerical weather predictions (NWP), known as ensemble prediction systems (EPS), to drive their predictions. We review the scientific drivers of this shift towards such ‘ensemble flood forecasting’ and discuss several of the questions surrounding best practice in using EPS in flood forecasting systems. We also review the literature evidence of the ‘added value’ of flood forecasts based on EPS and point to remaining key challenges in using EPS successfully.     

Assessment of a seasonal calibration technique using multiple objectives in rainfall–runoff analysis

The hydrological response characteristics for the catchments in the Republic of Korea are related to a strong seasonality in the rainfall and streamflow distributions with distinct wet and dry seasons. This study aims to improve a model's ability to predict streamflows by minimizing information loss from the available data during the calibration processes. This study assesses calibration techniques incorporating a multi‐objective approach and seasonal calibration. The lumped conceptual rainfall–runoff model IHACRES was applied to selected catchments in Korea. The model was calibrated based on three different methods: the classical approach using a single performance statistic (the single‐objective method), the multi‐objective approach (the multi‐objective method (I)) and the combined approach incorporating multi‐objective and seasonal calibrations (the multi‐objective method (II)). In the multi‐objective approach, the ‘best fit’ models in the calibration period were selected by considering the trade‐offs among multiple statistics. During seasonal calibration, the calibration period was divided into four seasons to investigate whether these calibrated models can improve the model performance with regards to seasonal climate, rainfall and streamflow distributions. The adequacy of the three different calibration methods was assessed through comparison of the variability of model performance in high and low flows and water balance for the entire period and for each seasonal period. The multi‐objective methods yielded more accurate and consistent predictions for high and low flows and water balance simultaneously, compared to the single‐objective method. In particular, the multi‐objective method (II) produces the best modelling capacity to capture the non‐stationary nature of the hydrological response under different climate conditions. The pattern of improvement with the multi‐objective method (II) was generally consistent through the seasons, with the exception of the winter period in the regions partially affected by snow. This exception is due to a potential limitation of the IHACRES model in reflecting the impact of snow on the catchment hydrology. Copyright © 2013 John Wiley & Sons, Ltd.