Data assimilation for distributed hydrological catchment modeling via ensemble Kalman filter

Catchment scale hydrological models are critical decision support tools for water resources management and environment remediation. However, the reliability of hydrological models is inevitably affected by limited measurements and imperfect models. Data assimilation techniques combine complementary information from measurements and models to enhance the model reliability and reduce predictive uncertainties. As a sequential data assimilation technique, the ensemble Kalman filter (EnKF) has been extensively studied in the earth sciences for assimilating in-situ measurements and remote sensing data. Although the EnKF has been demonstrated in land surface data assimilations, there are no systematic studies to investigate its performance in distributed modeling with high dimensional states and parameters. In this paper, we present an assessment on the EnKF with state augmentation for combined state-parameter estimation on the basis of a physical-based hydrological model, Soil and Water Assessment Tool (SWAT). Through synthetic simulation experiments, the capability of the EnKF is demonstrated by assimilating the runoff and other measurements, and its sensitivities are analyzed with respect to the error specification, the initial realization and the ensemble size. It is found that the EnKF provides an efficient approach for obtaining a set of acceptable model parameters and satisfactory runoff, soil water content and evapotranspiration estimations. The EnKF performance could be improved after augmenting with other complementary data, such as soil water content and evapotranspiration from remote sensing retrieval. Sensitivity studies demonstrate the importance of consistent error specification and the potential with small ensemble size in the data assimilation system.     

Bias aware Kalman filters: Comparison and improvements

This paper reviews two different approaches that have been proposed to tackle the problems of model bias with the Kalman filter: the use of a colored noise model and the implementation of a separate bias filter. Both filters are implemented with and without feedback of the bias into the model state. The colored noise filter formulation is extended to correct both time correlated and uncorrelated model error components. A more stable version of the separate filter without feedback is presented. The filters are implemented in an ensemble framework using Latin hypercube sampling. The techniques are illustrated on a simple one-dimensional groundwater problem. The results show that the presented filters outperform the standard Kalman filter and that the implementations with bias feedback work in more general conditions than the implementations without feedback.     

On ensemble representation of the observation-error covariance in the Ensemble Kalman Filter

Evensen (2003) presents a modification of the Ensemble Kalman Filter (EnKF), in which the observation-error and background-error covariance matrices are both represented by ensembles, in contrast to the usual practice, where only the background error is so represented. It is shown that this modification can cause the ensemble to collapse to a single member, in the common situation where the number of observations is more than twice the number of ensemble members, and to be rank-deficient when the number of observations is greater than or equal to the ensemble size. It is also shown that some further modifications to the scheme, presented by Evensen as offering numerical efficiencies, can prevent this collapse. However, these latter modifications are shown in some simple numerical examples to require tuning to produce acceptable results, which are nevertheless inferior to those of the standard EnKF.Acknowledgements The author acknowledges useful discussions with Peter Steinle, and other participants at the EnKF workshop held in BMRC in November, 2003.     

Inverse methods in hydrogeology: Evolution and recent trends

Parameter identification is an essential step in constructing a groundwater model. The process of recognizing model parameter values by conditioning on observed data of the state variable is referred to as the inverse problem. A series of inverse methods has been proposed to solve the inverse problem, ranging from trial-and-error manual calibration to the current complex automatic data assimilation algorithms. This paper does not attempt to be another overview paper on inverse models, but rather to analyze and track the evolution of the inverse methods over the last decades, mostly within the realm of hydrogeology, revealing their transformation, motivation and recent trends. Issues confronted by the inverse problem, such as dealing with multiGaussianity and whether or not to preserve the prior statistics are discussed.     

The Ensemble Kalman Filter: theoretical formulation and practical implementation

The purpose of this paper is to provide a comprehensive presentation and interpretation of the Ensemble Kalman Filter (EnKF) and its numerical implementation. The EnKF has a large user group, and numerous publications have discussed applications and theoretical aspects of it. This paper reviews the important results from these studies and also presents new ideas and alternative interpretations which further explain the success of the EnKF. In addition to providing the theoretical framework needed for using the EnKF, there is also a focus on the algorithmic formulation and optimal numerical implementation. A program listing is given for some of the key subroutines. The paper also touches upon specific issues such as the use of nonlinear measurements, in situ profiles of temperature and salinity, and data which are available with high frequency in time. An ensemble based optimal interpolation (EnOI) scheme is presented as a cost-effective approach which may serve as an alternative to the EnKF in some applications. A fairly extensive discussion is devoted to the use of time correlated model errors and the estimation of model bias.Responsible Editor: Jörg-Olaf Wolff     

Impacts of localisation in the EnKF and EnOI: experiments with a small model

The performance of an inexpensive, ensemble-based optimal interpolation (EnOI) scheme that uses a stationary ensemble of model anomalies to approximate forecast error covariances, is compared with that of an ensemble Kalman filter (EnKF). The model to which the methods are applied is a pair of “perfect”, one-dimensional, linear advection equations for two related variables. While EnOI is sub-optimal, it can give results that are comparable to those of the EnKF. The computational cost of EnOI is typically about times less than that of EnKF, where is the ensemble size. We suggest that EnOI may provide a practical and cost-effective alternative to the EnKF for some applications where computational cost is a limiting factor. We demonstrate that when the ensemble size is smaller than the dimension of the model’s sub-space, both the EnKF and EnOI may require localisation around each observation to eliminate effects of sampling error and to increase the effective number of independent ensemble members used to construct an analysis. However, localisation can degrade an analysis if the length-scales of the localising function are too short. We demonstrate that, as the length-scale of the localising function is decreased, localisation can significantly compromise the model’s dynamical balances. We also find that localisation artificially amplifies high frequencies for applications of the EnKF. Based on our experiments, for applications where localisation is necessary, the length-scales of the localisation should be larger than the decorrelation length-scales of the variables being updated.     

基于热层电离层耦合数据同化的热层参量估计

本文采用高效集合卡尔曼滤波（EnKF）算法和背景场热层电离层理论模式NCAR-TIEGCM,开发了热层电离层数据同化系统.基于全球空地基GNSS电离层斜TEC观测、CHAMP和TIMED/GUVI热层参量观测构型设计了系列观测系统模拟实验,对热层参量进行估计.实验结果表明,（1）通过集合卡尔曼滤波算法同化电离层TEC观测能够较好地优化热层参量.（2）中性质量密度优化效果在整个同化阶段均有提升,提升百分比能达到40%.（3）积分氧氮比在同化阶段也能得到较好的优化,但在电子密度水平梯度变化剧烈区域效果较差.最后本文对中性质量密度进行了预报评估,结果表明,由于中性成分优化,在地磁平静条件下其预报时间尺度可长达24 h.

     

Fast ensemble smoothing

Smoothing is essential to many oceanographic, meteorological, and hydrological applications. There are two predominant classes of smoothing problems. The first is fixed-interval smoothing, where the objective is to estimate model states within a time interval using all available observations in the interval. The second is fixed-lag smoothing, where the objective is to sequentially estimate model states over a fixed or indefinitely growing interval by restricting the influence of observations within a fixed window of time ahead of the evolving estimation time. In this paper, we use an ensemble-based approach to fixed-interval and fixed-lag smoothing, and synthesize two algorithms. The first algorithm is a fixed-interval smoother whose computation time is linear in the interval. The second algorithm is a fixed-lag smoother whose computation time is independent of the lag length. The complexity of these algorithms is presented, shown to improve upon existing implementations and verified with identical-twin experiments conducted with the Lorenz-95 system. Results suggest that ensemble methods yield efficient fixed-interval and fixed-lag smoothing solutions in the sense that the additional increment for smoothing is a small fraction of either filtering or model propagation costs in a practical ensemble application. We also show that fixed-interval smoothing can perform as fast as fixed-lag smoothing, and it may not be necessary to use a fixed-lag approximation for computational savings alone.     

Calibration framework for a Kalman filter applied to a groundwater model

The paper presents a novel approach to the setup of a Kalman filter by using an automatic calibration framework for estimation of the covariance matrices. The calibration consists of two sequential steps: (1) Automatic calibration of a set of covariance parameters to optimize the performance of the system and (2) adjustment of the model and observation variance to provide an uncertainty analysis relying on the data instead of ad-hoc covariance values. The method is applied to a twin-test experiment with a groundwater model and a colored noise Kalman filter. The filter is implemented in an ensemble framework. It is demonstrated that lattice sampling is preferable to the usual Monte Carlo simulation because its ability to preserve the theoretical mean reduces the size of the ensemble needed. The resulting Kalman filter proves to be efficient in correcting dynamic error and bias over the whole domain studied. The uncertainty analysis provides a reliable estimate of the error in the neighborhood of assimilation points but the simplicity of the covariance models leads to underestimation of the errors far from assimilation points.     

Dual states estimation of a subsurface flow-transport coupled model using ensemble Kalman filtering

Modeling the spread of subsurface contaminants requires coupling a groundwater flow model with a contaminant transport model. Such coupling may provide accurate estimates of future subsurface hydrologic states if essential flow and contaminant data are assimilated in the model. Assuming perfect flow, an ensemble Kalman filter (EnKF) can be used for direct data assimilation into the transport model. This is, however, a crude assumption as flow models can be subject to many sources of uncertainty. If the flow is not accurately simulated, contaminant predictions will likely be inaccurate even after successive Kalman updates of the contaminant model with the data. The problem is better handled when both flow and contaminant states are concurrently estimated using the traditional joint state augmentation approach. In this paper, we introduce a dual estimation strategy for data assimilation into a one-way coupled system by treating the flow and the contaminant models separately while intertwining a pair of distinct EnKFs, one for each model. The presented strategy only deals with the estimation of state variables but it can also be used for state and parameter estimation problems. This EnKF-based dual state-state estimation procedure presents a number of novel features: (i) it allows for simultaneous estimation of both flow and contaminant states in parallel; (ii) it provides a time consistent sequential updating scheme between the two models (first flow, then transport); (iii) it simplifies the implementation of the filtering system; and (iv) it yields more stable and accurate solutions than does the standard joint approach. We conducted synthetic numerical experiments based on various time stepping and observation strategies to evaluate the dual EnKF approach and compare its performance with the joint state augmentation approach. Experimental results show that on average, the dual strategy could reduce the estimation error of the coupled states by 15% compared with the joint approach. Furthermore, the dual estimation is proven to be very effective computationally, recovering accurate estimates at a reasonable cost.     

Comparison of deterministic ensemble Kalman filters for assimilating hydrogeological data

Groundwater models are critical decision support tools for water resources management and environmental remediation. However, limitations in site characterization data and conceptual models can adversely affect the reliability of groundwater models. Therefore, there is a strong need for continuous model uncertainty reduction. Ensemble filters have recently emerged as promising high-dimensional data assimilation techniques. Two general categories of ensemble filters exist in the literature: perturbation-based and deterministic. Deterministic ensemble filters have been extensively studied for their better performance and robustness in assimilating oceanographic and atmospheric data. In hydrogeology, while a number of previous studies demonstrated the usefulness of the perturbation-based ensemble Kalman filter (EnKF) for joint parameter and state estimation, there have been few systematic studies investigating the performance of deterministic ensemble filters. This paper presents a comparative study of four commonly used deterministic ensemble filters for sequentially estimating the hydraulic conductivity parameter in low- and moderately high-dimensional groundwater models. The performance of the filters is assessed on the basis of twin experiments in which the true hydraulic conductivity field is assumed known. The test results indicate that the deterministic ensemble Kalman filter (DEnKF) is the most robust filter and achieves the best performance at relatively small ensemble sizes. Deterministic ensemble filters often make use of covariance inflation and localization to stabilize filter performance. Sensitivity studies demonstrate the effects of covariance inflation, localization, observation density, and conditioning on filter performance.     

An application of Ensemble Kalman Filter in integral-balance subsurface modeling

Data assimilation method provides a framework to decrease the uncertainty of hydrological modeling by sequentially incorporating observations into numerical model. Such a process involves estimating statistical moments of different order based on the evolution of conditional probability distribution function. Because of the nonlinearity of many hydrological dynamics, explicit and analytical solutions for moments of state distribution are often impossible. Evensen [J Geophys Res 99(c5): 10143–10162 (1994)] introduced Ensemble Kalman Filtering (EnKF) method to address such problems. We test and evaluate the performance of EnKF in fusing model predictions and observations for a saturated–unsaturated integral-balance subsurface model. We find EnKF improve the model predictions, and also we conclude a good estimate of state variance is essential for the success of EnKF.     

Data assimilation in groundwater modelling: ensemble Kalman filter versus ensemble smoothers

Groundwater modelling calls for an effective and robust data integrating method to fill the gap between the model and observation data. The ensemble Kalman filter (EnKF), a real‐time data assimilation method, has been increasingly applied in multiple disciplines such as petroleum engineering and hydrogeology. In this approach, a groundwater model is updated sequentially with measured data such as hydraulic head and concentration. As an alternative to the EnKF, the ensemble smoother (ES) has been proposed for updating groundwater models using all the data together, with much less computational cost. To further improve the performance of the ES, an iterative ES has been proposed for continuously updating the model by assimilating measurements together. In this work, we compare the performance of the EnKF, the ES, and the iterative ES using a synthetic example in groundwater modelling. Hydraulic head data modelled on the basis of the reference conductivity field are used to inversely estimate conductivities at unsampled locations. Results are evaluated in terms of the characterization of conductivity and groundwater flow predictions. It is concluded that (a) the iterative ES works better than the standard ES because of its continuous updating and (b) the iterative ES could achieve results comparable with those of the EnKF, with less computational cost. These findings show that the iterative ES should be paid much more attention for data assimilation in groundwater modelling.     

State estimation of tidal hydrodynamics using ensemble Kalman filter

This paper presents a coupling of an ensemble Kalman filter (EnKF) with a discontinuous Galerkin-based, two-dimensional circulation model (DG ADCIRC-2DDI) to improve the state estimation of tidal hydrodynamics including water surface elevations and depth-integrated velocities. The methodology in this paper using EnKF perturbs the modeled hydrodynamics and bottom friction parameterization in the model while assimilating data with inherent error, and demonstrates a capability to apply EnKF within DG ADCIRC-2DDI for data assimilation. Parallel code development presents a unique aspect of the approach taken and is briefly described in the paper, followed by an application to a real estuarine system, the lower St. Johns River in north Florida, for the state estimation of tidal hydrodynamics. To test the value of gauge observations for improving state estimation, a tide modeling case study is performed for the lower St. Johns River successively using one of the four available tide gauging stations in model-data comparison. The results are improved simulations of water surface elevations and depth-integrated velocities using DG ADCIRC-2DDI with EnKF, both locally where data are available and non-locally where data are not available. The methodology, in general, is extensible to other modeling and data applications, for example, the use of remote sensing data, and specifically, can be readily applied as is to study other tidal systems.     

基于集合卡尔曼滤波的湖泊富营养化模型Delft3D-BLOOM数据同化

富营养化模型是进行湖泊水环境质量预测和管理的重要工具,然而模型客观存在的误差一直是应用者关心的重要问题.数据同化作为连接观测数据与数值模型的重要方法,可以有效提高模型的准确性.集合卡尔曼滤波(En KF)是众多数据同化算法中应用最为广泛的一种,可进行非线性系统的数据同化,并能有效降低数据同化的计算量.本研究以太湖作为具体实例,选择Delft3D-BLOOM作为富营养化模型,在数值实验确定En KF集合数为100、观测误差方差为1%、模拟误差方差为10%的基础上分别进行模型状态变量同化以及状态变量与关键参数同步同化.结果显示,仅同化状态变量时,模型预测精度有所增加;同时同化状态变量和关键参数时,可显著提升模型在湖泊水环境质量预测中的精度.该研究为应用集合卡尔曼滤波以提高复杂的湖库富营养化模型模拟精度提供了有效的方法.     

Investigation of flow and transport processes at the MADE site using ensemble Kalman filter

In this work the ensemble Kalman filter (EnKF) is applied to investigate the flow and transport processes at the macro-dispersion experiment (MADE) site in Columbus, MS. The EnKF is a sequential data assimilation approach that adjusts the unknown model parameter values based on the observed data with time. The classic advection–dispersion (AD) and the dual-domain mass transfer (DDMT) models are employed to analyze the tritium plume during the second MADE tracer experiment. The hydraulic conductivity (K), longitudinal dispersivity in the AD model, and mass transfer rate coefficient and mobile porosity ratio in the DDMT model, are estimated in this investigation. Because of its sequential feature, the EnKF allows for the temporal scaling of transport parameters during the tritium concentration analysis. Inverse simulation results indicate that for the AD model to reproduce the extensive spatial spreading of the tritium observed in the field, the K in the downgradient area needs to be increased significantly. The estimated K in the AD model becomes an order of magnitude higher than the in situ flowmeter measurements over a large portion of media. On the other hand, the DDMT model gives an estimation of K that is much more comparable with the flowmeter values. In addition, the simulated concentrations by the DDMT model show a better agreement with the observed values. The root mean square (RMS) between the observed and simulated tritium plumes is 0.77 for the AD model and 0.45 for the DDMT model at 328 days. Unlike the AD model, which gives inconsistent K estimates at different times, the DDMT model is able to invert the K values that consistently reproduce the observed tritium concentrations through all times.     

Hydrological data assimilation with the Ensemble Square-Root-Filter: Use of streamflow observations to update model states for real-time flash flood forecasting

The objective of the study is to evaluate the potential of a data assimilation system for real-time flash flood forecasting over small watersheds by updating model states. To this end, the Ensemble Square-Root-Filter (EnSRF) based on the Ensemble Kalman Filter (EnKF) technique was coupled to a widely used conceptual rainfall-runoff model called HyMOD. Two small watersheds susceptible to flash flooding from America and China were selected in this study. The modeling and observational errors were considered in the framework of data assimilation, followed by an ensemble size sensitivity experiment. Once the appropriate model error and ensemble size was determined, a simulation study focused on the performance of a data assimilation system, based on the correlation between streamflow observation and model states, was conducted. The EnSRF method was implemented within HyMOD and results for flash flood forecasting were analyzed, where the calibrated streamflow simulation without state updating was treated as the benchmark or nature run. Results for twenty-four flash-flood events in total from the two watersheds indicated that the data assimilation approach effectively improved the predictions of peak flows and the hydrographs in general. This study demonstrated the benefit and efficiency of implementing data assimilation into a hydrological model to improve flash flood forecasting over small, instrumented basins with potential application to real-time alert systems.     

基于Gauss-Markov卡尔曼滤波的电离层数值同化现报预报系统的构建—以中国及周边地区为例的观测系统模拟试验

本文给出了一个基于Gauss-Markov卡尔曼滤波的电离层数据同化系统的初步构建和试验结果.我们选择中国及周边地区部分涉及电离层观测的台站(包括子午工程台站、中国地壳形变网和部分IGS台站)作为观测系统进行模拟试验,背景场利用IRI模式,观测值则由NeQuick模式计算得到.我们的同化结果表明,采用Kalman滤波算法,把部分斜TEC同化到背景模式当中,能够获得较好的同化结果,说明我们设计的算法可行、所选择的各种参数比较合理,采用Gauss-Markov假设进行短期预报也取得了较合理的结果.本项研究经过进一步的改进和完善,可以用来对中国地区的电离层进行现报和短期预报,一方面满足相关空间工程应用,另一方面可以提升现有观测系统的科学意义.     

基于离散余弦变换的集合Kalman滤波方法对河流相油藏渗透率场的反演

针对河流相油藏参数场的非高斯性及油藏生产数据与地质模型之间的非线性关系,提出了耦合离散余弦变换的半迭代集合Kalman滤波方法对河流相油藏进行生产历史拟合及反演.将河流相油藏非高斯渗透率场利用离散余弦变换变为类似高斯分布的DCT(Discrete cosine transform)系数,利用能量压缩特性对DCT系数进行截断,使得反演渗透率场的分布变得光滑,较好地解决了河流相油藏的非高斯分布问题.半迭代集合Kalman滤波法很好地解决了油藏生产数据与地质模型之间的非线性问题.理论证明了半迭代集合Kalman滤波与EnRML方法的等效性.比较了标准EnKF、耦合DCT的EnKF及耦合DCT的半迭代EnKF法对河流相油藏渗透率场的反演.讨论了DCT系数保留模式及系数保留数量对耦合DCT的半迭代集合Kalman滤波方法反演结果的影响,结果表明三角形系数保留模式可以用更少的系数保留参数场的地质特征,当保留2.5%的DCT系数时,反演结果已经充分反映了河流相油藏的地质特征.