Group-based estimation of missing hydrological data: II. Application to streamflows

Abstract

The group approach that treats hydrological data as groups rather than as single-valued observations was proposed in a companion paper. Various models representing four techniques are briefly presented and applied to single series and bi-series cases, respectively, in this paper. The techniques represented by these models are regression, time series analysis, partitioning modelling, and artificial neural networks. The utility of the models for estimating missing streamflow data using the group approach is investigated. It turns out that the group approach is valid for estimating missing values, and possibly other applications, when data are significantly auto-correlated.     

Satellite-based rainfall estimation for river flow forecasting in Africa. I: Rainfall estimates and hydrological forecasts

Abstract

Reliable, real-time river flow forecasting in Africa on a time scale of days can provide enormous humanitarian and economic benefits. This study investigates the feasibility of using daily rainfall estimates based on cold cloud duration (CCD) derived from Meteosat thermal infrared imagery as input to a simple rainfall—runoff model and also whether such estimates can be improved by the inclusion of information from numerical weather prediction (NWP) analysis models. The Bakoye catchment in Mali, West Africa has been used as a test area. The data available for the study covered the main months of the rainy season for three years. The rainfall estimates were initially validated against gauge data. Improvements in quality were observed when information relating to African Easterly Wave phase and storm type was included in a multiple linear regression (MR) algorithm. The estimates were also tested by using them as input to a rainfall—runoff model. When contemporaneous calibrations from raingauges were available for calibration, both CCD-only and MR rainfall estimates gave more accurate river flow forecasts than when using raingauge data alone. In the absence of contemporaneous calibrations, the performance was reduced but the MR did better than the CCDonly input in all years. The use of satellite-derived vegetation index did not improve the quality of the river flow forecasts.     

Evaluation of efficiency of different estimation methods for missing climatological data

Reliable estimation of missing data is an important task for meteorologists, hydrologists and environment protection workers all over the world. In recent years, artificial intelligence techniques have gained enormous interest of many researchers in estimating of missing values. In the current study, we evaluated 11 artificial intelligence and classical techniques to determine the most suitable model for estimating of climatological data in three different climate conditions of Iran. In this case, 5 years (2001–2005) of observed data at target and neighborhood stations were used to estimate missing data of monthly minimum temperature, maximum temperature, mean air temperature, relative humidity, wind speed and precipitation variables. The comparison includes both visual and parametric approaches using such statistic as mean absolute errors, coefficient of efficiency and skill score. In general, it was found that although the artificial intelligence techniques are more complex and time-consuming models in identifying their best structures for optimum estimation, but they outperform the classical methods in estimating missing data in three distinct climate conditions. Moreover, the in-filling done by artificial neural network rivals that by genetic programming and sometimes becomes more satisfactory, especially for precipitation data. The results also indicated that multiple regression analysis method is the suitable method among the classical methods. The results of this research proved the high importance of choosing the best and most precise method in estimating different climatological data in Iran and other arid and semi-arid regions.     

Entropy estimation of hydrological extremes

The method of Relative Entropy with Fractile constraints (REF method) is explained and applied to model extreme compound hydrological phenomena, such as extreme sea levels under storm conditions. Also presented is a simple method of Tail Entropy Approximation (TEA), which amounts to a correction of traditional statistical estimates for extreme observations.Distribution assumptions are necessary but downplayed in the REF method, relegating the prior distribution to the role of an extrapolation function. The estimates are objective in an information-theoretical sense. They also satisfy a strict requirement of self-consistency that is generally not satisfied by standard statistical methods: invariance under monotonic transformations of the random variable.Historical records of storm surge levels in the Netherlands and annual maximum tidal heights for Sheerness, UK, are used as examples. Comparison is made with distributions obtained using other methods.It is concluded that the tail entropy approximation provides simple, objective estimates of extremes in the tail beyond the range of observations.     

Application of hydrological and hydraulic models for hydrological data transfer

This paper presents an application of hydrological and hydraulic models for transferring instantaneous discharges from a water gauge station to budgeting sites on rivers. Calculations were done using the following models: MIKE NAM rainfall-runoff model and a hydrodynamic MIKE 11 HD model. The simulations were carried out for the catchment of Warta River and its tributaries for the multiyear period 1999–2009.     

A general Bayesian framework for calibrating and evaluating stochastic models of annual multi-site hydrological data

Multi-site simulation of hydrological data are required for drought risk assessment of large multi-reservoir water supply systems. In this paper, a general Bayesian framework is presented for the calibration and evaluation of multi-site hydrological data at annual timescales. Models included within this framework are the hidden Markov model (HMM) and the widely used lag-1 autoregressive (AR(1)) model. These models are extended by the inclusion of a Box–Cox transformation and a spatial correlation function in a multi-site setting. Parameter uncertainty is evaluated using Markov chain Monte Carlo techniques. Models are evaluated by their ability to reproduce a range of important extreme statistics and compared using Bayesian model selection techniques which evaluate model probabilities. The case study, using multi-site annual rainfall data situated within catchments which contribute to Sydney’s main water supply, provided the following results: Firstly, in terms of model probabilities and diagnostics, the inclusion of the Box–Cox transformation was preferred. Secondly the AR(1) and HMM performed similarly, while some other proposed AR(1)/HMM models with regionally pooled parameters had greater posterior probability than these two models. The practical significance of parameter and model uncertainty was illustrated using a case study involving drought security analysis for urban water supply. It was shown that ignoring parameter uncertainty resulted in a significant overestimate of reservoir yield and an underestimation of system vulnerability to severe drought.     

LH-moment estimation of Wakeby distribution with hydrological applications

This article discusses the method of higher-order L-moment (LH-moment) estimation for the Wakeby distribution (WAD), and describes and formulates details of parameter estimation using LH-moments for WAD. Monte Carlo simulation is performed, to illustrate the performance of the LH-moment method via heavy-tail quantiles (over all quantiles) using WAD. The LH-moment method proves as useful and effective as the L-moment approach in handling data that follow WAD, and it is then applied to annual maximum flood and wave height data.     

Review of the Kalman-type hydrological data assimilation

ABSTRACT

There is great potential in Data Assimilation (DA) for the purposes of uncertainty identification, reduction and real-time correction of hydrological models. This paper reviews the latest developments in Kalman filters (KFs), particularly the Extended KF (EKF) and the Ensemble KF (EnKF) in hydrological DA. The hydrological DA targets, methodologies and their applicability are examined. The recent applications of the EKF and EnKF in hydrological DA are summarized and assessed critically. Furthermore, this review highlights the existing challenges in the implementation of the EKF and EnKF, especially error determination and joint parameter estimation. A detailed review of these issues would benefit not only the Kalman-type DA but also provide an important reference to other hydrological DA types.

Editor D. Koutsoyiannis; Associate editor F. Pappenberger     

Story-by-story estimation of the stiffness parameters of laterally-torsionally coupled buildings using forced or ambient vibration data: I. Formulation and verification

A new parameter estimation algorithm is described for identifying the stiffness properties of torsionally coupled shear buildings from their linear response due to ambient excitations or during low-amplitude forced-vibration tests. The algorithm is based on the time-domain equations of motion, and yields estimates of the stiffness properties using a measure of the equilibrium of forces acting on each floor over a time interval. The banded structure of the stiffness matrix — a property intrinsic to torsion-shear buildings — is exploited to decompose the initial inverse problem into several problems of reduced size. This decomposition allows the identification of lateral and torsional stiffnesses of individual stories, independent of the others. The algorithm utilizes vibration data where input excitation is known/measured, which is typical for forced-vibration tests and earthquakes. If the ambient vibrations of the structure are adequately uncorrelated to the (unknown) external forces that induce such vibrations, then the algorithm can also be modified for output-only system identification. The proposed algorithm is verified — and its various attributes are investigated — using simulation data from the ‘Analytical Phase I’ of the IASC (International Association for Structural Control)-ASCE (American Society of Civil Engineers) benchmark studies. The companion article is devoted to the algorithm's application to experimental data, using data from the ‘Experimental Phase’ of the same benchmark studies. Copyright © 2011 John Wiley & Sons, Ltd.     

Development of a modelling methodology for the investigation of riparian hydrological processes

The riparian zone is in intimate contact with the river and, as such, is a critical zone for understanding hydrological problems. This paper presents a general modelling methodology for the assessment of riparian hydrological processes. It is applicable to a wide range of riparian spaces and incorporates current expertise in numerical methods. A core part of the modelling methodology is the random walk particle method (RWPM). We develop an RWPM as part of the ESTEL2D subsurface flow model, test it against analytical solutions and apply it to the simulation of parcels of water as they move through the riparian zone. The modelling methodology provides a new opportunity to assess fundamental hydrological process issues such as the proportioning of pre‐event and event water storm runoff, and reversals of flow in floodplains. Copyright © 2005 John Wiley & Sons, Ltd.     

The Shaw chondrite,I. The case of the missing metal

The mineralogy, elemental and isotopic composition of the Shaw meteorite indicate that it is a highly metamorphosed L-group chondrite which has lost a portion of its metal and sulfide. The metal which remains has an unusual composition relative to that in other L-group chondrites. It is enriched in Ga, Ge, Ir, Mo, Os, Pt, Re and Ru and depleted in As, Au, Cu and Sb. A comparison of the relative enrichments and depletions in Shaw to those observed in San Cristobal, the extreme end-member of group IAB iron meteorites, shows that the metal phases in these two meteorites have complementary compositions. This implies that the metal in Shaw represents the residual solid of a partial melting process while the missing metal, which drained away, may have gone to form an iron meteorite, like San Cristobal.     

Parametric uncertainty assessment of hydrological models: coupling UNEEC-P and a fuzzy general regression neural network

Due to the complicated nature of environmental processes, consideration of uncertainty is an important part of environmental modelling. In this paper, a new variant of the machine learning-based method for residual estimation and parametric model uncertainty is presented. This method is based on the UNEEC-P (UNcertainty Estimation based on local Errors and Clustering – Parameter) method, but instead of multilayer perceptron uses a “fuzzified” version of the general regression neural network (GRNN). Two hydrological models are chosen and the proposed method is used to evaluate their parametric uncertainty. The approach can be classified as a hybrid uncertainty estimation method, and is compared to the group method of data handling (GMDH) and ordinary kriging with linear external drift (OKLED) methods. It is shown that, in terms of inherent complexity, measured by Akaike information criterion (AIC), the proposed fuzzy GRNN method has advantages over other techniques, while its accuracy is comparable. Statistical metrics on verification datasets demonstrate the capability and appropriate efficiency of the proposed method to estimate the uncertainty of environmental models.     

Study of discontinuities in hydrological data using catastrophe theory

Abstract

Modelling and prediction of hydrological processes (e.g. rainfall–runoff) can be influenced by discontinuities in observed data, and one particular case may arise when the time scale (i.e. resolution) is coarse (e.g. monthly). This study investigates the application of catastrophe theory to examine its suitability to identify possible discontinuities in the rainfall–runoff process. A stochastic cusp catastrophe model is used to study possible discontinuities in the monthly rainfall–runoff process at the Aji River basin in Azerbaijan, Iran. Monthly-averaged rainfall and flow data observed over a period of 20 years (1981–2000) are analysed using the Cuspfit program. In this model, rainfall serves as a control variable and runoff as a behavioural variable. The performance of this model is evaluated using four measures: correlation coefficient, log-likelihood, Akaike information criterion (AIC) and Bayesian information criterion (BIC). The results indicate the presence of discontinuities in the rainfall–runoff process, with a significant sudden jump in flow (cusp signal) when rainfall reaches a threshold value. The performance of the model is also found to be better than that of linear and logistic models. The present results, though preliminary, are promising in the sense that catastrophe theory can play a possible role in the study of hydrological systems and processes, especially when the data are noisy.

Citation Ghorbani, M. A., Khatibi, R., Sivakumar, B. & Cobb, L. (2010) Study of discontinuities in hydrological data using catastrophe theory. Hydrol. Sci. J. 55(7), 1137–1151.     

A note on the applicability of log-Gumbel and log-logistic probability distributions in hydrological analyses: I. Known pdf

Abstract

Two probability density functions (pdf), popular in hydrological analyses, namely the log-Gumbel (LG) and log-logistic (LL), are discussed with respect to (a) their applicability to hydrological data and (b) the drawbacks resulting from their mathematical properties. This paper—the first in a two-part series—examines a classical problem in which the considered pdf is assumed to be the true distribution. The most significant drawback is the existence of the statistical moments of LG and LL for a very limited range of parameters. For these parameters, a very rapid increase of the skewness coefficient, as a function of the coefficient of variation, is observed (especially for the log-Gumbel distribution), which is seldom observed in the hydrological data. These probability distributions can be applied with confidence only to extreme situations. For other cases, there is an important disagreement between empirical data and theoretical distributions in their tails, which is very important for the characterization of the distribution asymmetry. The limited range of shape parameters in both distributions makes the analyses (such as the method of moments), that make use of the interpretation of moments, inconvenient. It is also shown that the often-used L-moments are not sufficient for the characterization of the location, scale and shape parameters of pdfs, particularly in the case where attention is paid to the tail part of probability distributions. The maximum likelihood method guarantees an asymptotic convergence of the estimators beyond the domain of the existence of the first two moments (or L-moments), but it is not sensitive enough to the upper tails shape.     

Regionalisation of hydrological responses under land-use change and variable data quality

ABSTRACT

Estimating river flows at ungauged sites is generally recognised as an important area of research. In countries or regions with rapid land development and sparse hydrological gauging networks, three particular challenges may arise—data scarcity, data quality, and hydrological non-stationarity. Using data from 44 gauged sub-catchments of the upper Ping catchment in northern Thailand from the period 1995–2006, three relevant flow response indices (runoff coefficient, base flow index and seasonal elasticity of flow) were regionalised by regression against available catchment properties. The runoff coefficient was the most successfully regionalised, followed by base flow index and lastly the seasonal elasticity. The non-stationarity (represented by the differences between two 6-year sub-periods) was significant both in the flow response indices and in land use indices; however relationships between the two sets of indices were weak. The regression equations derived from regionalisation were not helpful in predicting the non-stationarity in the flow indices except somewhat for the runoff coefficient. A partly subjective data quality scoring system was devised, and showed the clear influence of rainfall and flow data quality on regionalisation uncertainty. Recommendations towards improving data support for hydrological regionalisation in Thailand include more relevant soils databases, improved records of abstractions and investment in the gauge network. Widening of the regionalisation beyond the upper Ping and renewed efforts at using remotely sensed rainfall data are other possible ways forward.

EDITOR Z.W. Kundzewicz ASSOCIATE EDITOR T. Wagener     

Effects of missing data on zonal kinetic energy calculations

Summary The effects of missing data upon zonal kinetic energy calculations are investigated by constructing clusters of alternate stations in the vicinity of 59 primary stations in middle latitudes. Reports from these stations are then taken to approximate the missing winds at the primary stations. In the mid-troposphere as much as about seventy percent of the missing primary station reports were eliminated through the use of the alternates. This adjustment failed to detect any large systematic bias in the observed mean wind fields calculated from an original 206 stations of which the 59 were a subset. The missing data had only a marginal influence on other quantitative results.The research reported in this paper was sponsored by the U.S. National Science Foundation under Grant No. GA-1310X.     

Statistical corrections of spatially interpolated missing precipitation data estimates

Spatial interpolation methods used for estimation of missing precipitation data generally under and overestimate the high and low extremes, respectively. This is a major limitation that plagues all spatial interpolation methods as observations from different sites are used in local or global variants of these methods for estimation of missing data. This study proposes bias‐correction methods similar to those used in climate change studies for correcting missing precipitation estimates provided by an optimal spatial interpolation method. The methods are applied to post‐interpolation estimates using quantile mapping, a variant of equi‐distant quantile matching and a new optimal single best estimator (SBE) scheme. The SBE is developed using a mixed‐integer nonlinear programming formulation. K‐fold cross validation of estimation and correction methods is carried out using 15 rain gauges in a temperate climatic region of the U.S. Exhaustive evaluation of bias‐corrected estimates is carried out using several statistical, error, performance and skill score measures. The differences among the bias‐correction methods, the effectiveness of the methods and their limitations are examined. The bias‐correction method based on a variant of equi‐distant quantile matching is recommended. Post‐interpolation bias corrections have preserved the site‐specific summary statistics with minor changes in the magnitudes of error and performance measures. The changes were found to be statistically insignificant based on parametric and nonparametric hypothesis tests. The correction methods provided improved skill scores with minimal changes in magnitudes of several extreme precipitation indices. The bias corrections of estimated data also brought site‐specific serial autocorrelations at different lags and transition states (dry‐to‐dry, dry‐to‐wet, wet‐to‐wet and wet‐to‐dry) close to those from the observed series. Bias corrections of missing data estimates provide better serially complete precipitation time series useful for climate change and variability studies in comparison to uncorrected filled data series. Copyright © 2013 John Wiley & Sons, Ltd.     

含噪声数据反演的概率描述

根据贝叶斯理论给出了对含噪声地球物理数据处理的具体流程和方法,主要包括似然函数估计和后验概率计算.我们将数据向量的概念扩展为数据向量的集合,通过引入数据空间内的信赖度,把数据噪声转移到模型空间的概率密度函数上,即获得了反映数据本身的不确定性的似然函数.该方法由于避免了处理阶段数据空间内的人工干预,因而可以保证模型空间中的概率密度单纯反映数据噪声,具有信息保真度高、保留可行解的优点.为了得到加入先验信息的后验分布,本文提出了使用加权矩阵的概率分析法,该方法在模型空间直接引入地质信息,对噪声引起的反演多解性有很强的约束效果.整个处理流程均以大地电磁反演为例进行了展示.     

Similarity search and pattern discovery in hydrological time series data mining

The rapid development of data mining provides a new method for water resource management, hydrology and hydroinformatics research. In the paper, based on data mining theory and technology, we analyse hydrological daily discharge time series of the Shaligunlanke Station in the Tarim River Basin in China from the year 1961 to 2000. Firstly, according to the four monthly statistics, namely mean monthly discharge, monthly maximum discharge, monthly amplitude and monthly standard deviation, K‐mean clustering was used to segment the annual process of the daily discharge. The clustering result showed that the annual process of the daily discharge can be divided into five segments: snowmelt period I (April), snowmelt period II (May), rainfall period I (June–August), rainfall period II (September) and dry period (October–December and January–March). Secondly, dynamic time warping (DTW), which is a different distance metric method from the traditional Euclidian distance metric, was used to look for similarities in the discharge process. On the basis of the similarity matrix, the similar discharge processes can be mined in each period. Thirdly, agglomerative hierarchical clustering was used to cluster and discover the discharge patterns in terms of the autoregressive model. It was found that the discharge had a close relationship with the temperature and the precipitation, and the discharge processes were more similar under the same climatic condition. Our study shows that data mining is a feasible and efficient approach to discover the hidden information in the historical hydrological data and mining the implicative laws under the hydrological process. Copyright © 2010 John Wiley & Sons, Ltd.