On ensemble representation of the observation-error covariance in the Ensemble Kalman Filter

Evensen (2003) presents a modification of the Ensemble Kalman Filter (EnKF), in which the observation-error and background-error covariance matrices are both represented by ensembles, in contrast to the usual practice, where only the background error is so represented. It is shown that this modification can cause the ensemble to collapse to a single member, in the common situation where the number of observations is more than twice the number of ensemble members, and to be rank-deficient when the number of observations is greater than or equal to the ensemble size. It is also shown that some further modifications to the scheme, presented by Evensen as offering numerical efficiencies, can prevent this collapse. However, these latter modifications are shown in some simple numerical examples to require tuning to produce acceptable results, which are nevertheless inferior to those of the standard EnKF.Acknowledgements The author acknowledges useful discussions with Peter Steinle, and other participants at the EnKF workshop held in BMRC in November, 2003.     

Prediction and predictability of a catastrophic local extreme precipitation event through cloud-resolving ensemble analysis and forecasting with Doppler radar observations

Local extreme rain usually resulted in disasters such as flash floods and landslides. Upon today, it is still one of the most difficult tasks for operational weather forecast centers to predict those events accurately. In this paper, we simulate an extreme precipitation event with ensemble Kalman filter (EnKF) assimilation of Doppler radial-velocity observations, and analyze the uncertainties of the assimilation. The results demonstrate that, without assimilation radar data, neither a single initialization of deterministic forecast nor an ensemble forecast with adding perturbations or multiple physical parameterizations can predict the location of strong precipitation. However, forecast was significantly improved with assimilation of radar data, especially the location of the precipitation. The direct cause of the improvement is the buildup of a deep mesoscale convection system with EnKF assimilation of radar data. Under a large scale background favorable for mesoscale convection, efficient perturbations of upstream mid-low level meridional wind and moisture are key factors for the assimilation and forecast. Uncertainty still exists for the forecast of this case due to its limited predictability. Both the difference of large scale initial fields and the difference of analysis obtained from EnKF assimilation due to small amplitude of initial perturbations could have critical influences to the event's prediction. Forecast could be improved through more cycles of EnKF assimilation. Sensitivity tests also support that more accurate forecasts are expected through improving numerical models and observations.     

Validation and application of an ensemble Kalman filter in the Selat Pauh of Singapore

The effectiveness of an ensemble Kalman filter (EnKF) is assessed in the Selat Pauh of Singapore using observing system simulation experiment. Perfect model experiments are first considered. The perfect model experiments examine the EnKF in reducing the initial perturbations with no further errors than those in the initial conditions. Current velocity at 15 observational sites from the true ocean is assimilated every hour into the false ocean. While EnKF reduces the initial velocity error during the first few hours, it fails after one tidal cycle (approximately 12 h) due to the rapid convergence of the ensemble members. Successively, errors are introduced in the surface wind forcing. A random perturbation ε is applied independently to each ensemble member to maintain the ensemble spread. The assimilation results showed that the success of EnKF depends critically on the presence of ε, yet it is not sensitive to the magnitude of ε, at least in the range of weak to moderate perturbations. Although all experiments were made with EnKF only, the results could be applicable in general to all other ensemble-based data assimilation methods.     

Impacts of localisation in the EnKF and EnOI: experiments with a small model

The performance of an inexpensive, ensemble-based optimal interpolation (EnOI) scheme that uses a stationary ensemble of model anomalies to approximate forecast error covariances, is compared with that of an ensemble Kalman filter (EnKF). The model to which the methods are applied is a pair of “perfect”, one-dimensional, linear advection equations for two related variables. While EnOI is sub-optimal, it can give results that are comparable to those of the EnKF. The computational cost of EnOI is typically about times less than that of EnKF, where is the ensemble size. We suggest that EnOI may provide a practical and cost-effective alternative to the EnKF for some applications where computational cost is a limiting factor. We demonstrate that when the ensemble size is smaller than the dimension of the model’s sub-space, both the EnKF and EnOI may require localisation around each observation to eliminate effects of sampling error and to increase the effective number of independent ensemble members used to construct an analysis. However, localisation can degrade an analysis if the length-scales of the localising function are too short. We demonstrate that, as the length-scale of the localising function is decreased, localisation can significantly compromise the model’s dynamical balances. We also find that localisation artificially amplifies high frequencies for applications of the EnKF. Based on our experiments, for applications where localisation is necessary, the length-scales of the localisation should be larger than the decorrelation length-scales of the variables being updated.     

Assimilating transient groundwater flow data via a localized ensemble Kalman filter to calibrate a heterogeneous conductivity field

A localized ensemble Kalman filter (EnKF) method is developed to assimilate transient flow data to calibrate a heterogeneous conductivity field. To update conductivity value at a point in a study domain, instead of assimilating all the measurements in the study domain, only limited measurement data in an area around the point are used for the conductivity updating in the localized EnKF method. The localized EnKF is proposed to solve the problems of the filter divergence usually existing in a data assimilation method without localization. The developed method is applied, in a synthetical two dimensional case, to calibrate a heterogeneous conductivity field by assimilating transient hydraulic head data. The simulations by the data assimilation with and without localized EnKF are compared. The study results indicate that the hydraulic conductivity field can be updated efficiently by the localized EnKF, while it cannot be by the EnKF. The covariance inflation and localization are found to solve the problem of the filter divergence efficiently. In comparison with the EnKF method without localization, the localized EnKF method needs smaller ensemble size to achieve stabilized results. The simulation results by the localized EnKF method are much more sensitive to conductivity correlation length than to the localization radius. The developed localized EnKF method provides an approach to improve EnKF method in conductivity calibration.     

Parameter estimation of subsurface flow models using iterative regularized ensemble Kalman filter

A new parameter estimation algorithm based on ensemble Kalman filter (EnKF) is developed. The developed algorithm combined with the proposed problem parametrization offers an efficient parameter estimation method that converges using very small ensembles. The inverse problem is formulated as a sequential data integration problem. Gaussian process regression is used to integrate the prior knowledge (static data). The search space is further parameterized using Karhunen–Loève expansion to build a set of basis functions that spans the search space. Optimal weights of the reduced basis functions are estimated by an iterative regularized EnKF algorithm. The filter is converted to an optimization algorithm by using a pseudo time-stepping technique such that the model output matches the time dependent data. The EnKF Kalman gain matrix is regularized using truncated SVD to filter out noisy correlations. Numerical results show that the proposed algorithm is a promising approach for parameter estimation of subsurface flow models.     

An effective method based on dynamic sampling for data assimilation in a global wave model

The ensemble Kalman filter (EnKF) performs well because that the covariance of background error is varying along time. It provides a dynamic estimate of background error and represents the reasonable statistic characters of background error. However, high computational cost due to model ensemble in EnKF is employed. In this study, two methods referred as static and dynamic sampling methods are proposed to obtain a good performance and reduce the computation cost. Ensemble adjustment Kalman filter (EAKF) method is used in a global surface wave model to examine the performance of EnKF. The 24-h interval difference of simulated significant wave height (SWH) within 1 year is used to compose the static samples for ensemble errors, and these errors are used to construct the ensemble states at each time the observations are available. And then, the same method of updating the model states in the EAKF is applied for the ensemble states constructed by a static sampling method. The dynamic sampling method employs a similar method to construct the ensemble states, but the period of the simulated SWH is changing with time. Here, 7 days before and after the observation time is used as this period. To examine the performance of three schemes, EAKF, static, or dynamic sampling method, observations from satellite Jason-2 in 2014 are assimilated into a global wave model, and observations from satellite Saral are used for validation. The results indicate that the EAKF performs best, while the static sampling method is relatively worse. The dynamic sampling method improves an assimilation effect dramatically compared to the static sampling method, and its overall performance is closed to the EAKF. In low latitudes, the dynamic sampling method has a slight advantage over the EAKF. In the dynamic or static sampling methods, only one wave model is required to run and their computational cost is reduced sharply. According to the performance of these three methods, the dynamic sampling method can treated as an effective alternative of EnKF, which could reduce the computational cost and provide a good performance of data assimilation.     

The Local Ensemble Transform Kalman Filter with the Weather Research and Forecasting Model: Experiments with Real Observations

The local ensemble transform Kalman filter (LETKF) is implemented with the Weather Research and Forecasting (WRF) model, and real observations are assimilated to assess the newly-developed WRF-LETKF system. The WRF model is a widely-used mesoscale numerical weather prediction model, and the LETKF is an ensemble Kalman filter (EnKF) algorithm particularly efficient in parallel computer architecture. This study aims to provide the basis of future research on mesoscale data assimilation using the WRF-LETKF system, an additional testbed to the existing EnKF systems with the WRF model used in the previous studies. The particular LETKF system adopted in this study is based on the system initially developed in 2004 and has been continuously improved through theoretical studies and wide applications to many kinds of dynamical models including realistic geophysical models. Most recent and important improvements include an adaptive covariance inflation scheme which considers the spatial and temporal inhomogeneity of inflation parameters. Experiments show that the LETKF successfully assimilates real observations and that adaptive inflation is advantageous. Additional experiments with various ensemble sizes show that using more ensemble members improves the analyses consistently.     

Evaluating the utility of the ensemble transform Kalman filter for adaptive sampling when updating a hydrodynamic model

This paper compares two Monte Carlo sequential data assimilation methods based on the Kalman filter, for estimating the effect of measurements on simulations of state error variance made by a one-dimensional hydrodynamic model. The first method used an ensemble Kalman filter (EnKF) to update state estimates, which were then used as initial conditions for further simulations. The second method used an ensemble transform Kalman filter (ETKF) to quickly estimate the effect of measurement error covariance on forecast error covariance without the need to re-run the simulation model. The ETKF gave an unbiased estimate of EnKF analysed error variance, although differences in the treatment of measurement errors meant the results were not identical. Estimates of forecast error variance could also be made, but their accuracy deteriorated as the time from measurements increased due in part to model non-linearity and the decreasing signal variance. The motivation behind the study was to assess the ability of the ETKF to target possible measurements, as part of an adaptive sampling framework, before they are assimilated by an EnKF-based forecasting model on the River Crouch, Essex, UK. The ETKF was found to be a useful tool for quickly estimating the error covariance expected after assimilating measurements into the hydrodynamic model. It, thus, provided a means of quantifying the ‘usefulness’ (in terms of error variance) of possible sampling schemes.     

Handling non-linearity in radar data assimilation using the non-linear least squares enhanced POD-4DVar

The Proper Orthogonal Decomposition(POD)-based ensemble four-dimensional variational(4DVar) assimilation method(POD4DEnVar) was proposed to combine the strengths of EnKF(i.e.,the ensemble Kalman filter) and 4DVar assimilation methods.Recently,a POD4DEnVar-based radar data assimilation scheme(PRAS) was built and its effectiveness was demonstrated.POD4 DEnVar is based on the assumption of a linear relationship between the model perturbations(MPs)and the observation perturbations(OPs);however,this assumption is likely to be destroyed by the highly non-linear forecast model or observation operator.To address this issue,using the Gauss-Newton iterative method,the nonlinear least squares enhanced POD4 DEnVar algorithm(referred to as NLS-4DVar) was proposed.Naturally,the PRAS was upgraded to form the NLS-4DVar-based radar data assimilation scheme(NRAS).To evaluate the performance of NRAS against PRAS,observing system simulation experiments(OSSEs) were conducted to assimilate reflectivity and radial velocity individually,with one,two,and three iterations.The results demonstrated that the NRAS outperformed PRAS in improving the initial condition and the forecasting of model variables and rainfall.The NRAS,with a smaller number of iterations,can yield a convergent result.In contrast to the situation when assimilating radial velocity,the advantages of NRAS over PRAS were more obvious when assimilating reflectivity.     

Comparison of Ensemble Kalman Filter groundwater-data assimilation methods based on stochastic moment equations and Monte Carlo simulation

Traditional Ensemble Kalman Filter (EnKF) data assimilation requires computationally intensive Monte Carlo (MC) sampling, which suffers from filter inbreeding unless the number of simulations is large. Recently we proposed an alternative EnKF groundwater-data assimilation method that obviates the need for sampling and is free of inbreeding issues. In our new approach, theoretical ensemble moments are approximated directly by solving a system of corresponding stochastic groundwater flow equations. Like MC-based EnKF, our moment equations (ME) approach allows Bayesian updating of system states and parameters in real-time as new data become available. Here we compare the performances and accuracies of the two approaches on two-dimensional transient groundwater flow toward a well pumping water in a synthetic, randomly heterogeneous confined aquifer subject to prescribed head and flux boundary conditions.     

Particle Filter-based assimilation algorithms for improved estimation of root-zone soil moisture under dynamic vegetation conditions

In this study, we implement Particle Filter (PF)-based assimilation algorithms to improve root-zone soil moisture (RZSM) estimates from a coupled SVAT-vegetation model during a growing season of sweet corn in North Central Florida. The results from four different PF algorithms were compared with those from the Ensemble Kalman Filter (EnKF) when near-surface soil moisture was assimilated every 3 days using both synthetic and field observations. In the synthetic case, the PF algorithm with the best performance used residual resampling of the states and obtained resampled parameters from a uniform distribution and provided reductions of 76% in root mean square error (RMSE) over the openloop estimates. The EnKF provided the RZSM and parameter estimates that were closer to the truth than the PF with an 84% reduction in RMSE. When field observations were assimilated, the PF algorithm that maintained maximum parameter diversity offered the largest reduction of 16% in root mean square difference (RMSD) over the openloop estimates. Minimal differences were observed in the overall performance of the EnKF and PF using field observations since errors in model physics affected both the filters in a similar manner, with maximum reductions in RMSD compared to the openloop during the mid and reproductive stages.     

The Ensemble Kalman Filter: theoretical formulation and practical implementation

The purpose of this paper is to provide a comprehensive presentation and interpretation of the Ensemble Kalman Filter (EnKF) and its numerical implementation. The EnKF has a large user group, and numerous publications have discussed applications and theoretical aspects of it. This paper reviews the important results from these studies and also presents new ideas and alternative interpretations which further explain the success of the EnKF. In addition to providing the theoretical framework needed for using the EnKF, there is also a focus on the algorithmic formulation and optimal numerical implementation. A program listing is given for some of the key subroutines. The paper also touches upon specific issues such as the use of nonlinear measurements, in situ profiles of temperature and salinity, and data which are available with high frequency in time. An ensemble based optimal interpolation (EnOI) scheme is presented as a cost-effective approach which may serve as an alternative to the EnKF in some applications. A fairly extensive discussion is devoted to the use of time correlated model errors and the estimation of model bias.Responsible Editor: Jörg-Olaf Wolff     

Data assimilation for unsaturated flow models with restart adaptive probabilistic collocation based Kalman filter

The ensemble Kalman filter (EnKF) has gained popularity in hydrological data assimilation problems. As a Monte Carlo based method, a sufficiently large ensemble size is usually required to guarantee the accuracy. As an alternative approach, the probabilistic collocation based Kalman filter (PCKF) employs the polynomial chaos expansion (PCE) to represent and propagate the uncertainties in parameters and states. However, PCKF suffers from the so-called “curse of dimensionality”. Its computational cost increases drastically with the increasing number of parameters and system nonlinearity. Furthermore, PCKF may fail to provide accurate estimations due to the joint updating scheme for strongly nonlinear models. Motivated by recent developments in uncertainty quantification and EnKF, we propose a restart adaptive probabilistic collocation based Kalman filter (RAPCKF) for data assimilation in unsaturated flow problems. During the implementation of RAPCKF, the important parameters are identified and active PCE basis functions are adaptively selected at each assimilation step; the “restart” scheme is utilized to eliminate the inconsistency between updated model parameters and states variables. The performance of RAPCKF is systematically tested with numerical cases of unsaturated flow models. It is shown that the adaptive approach and restart scheme can significantly improve the performance of PCKF. Moreover, RAPCKF has been demonstrated to be more efficient than EnKF with the same computational cost.     

Real-time assimilation of streamflow observations into a hydrological routing model: effects of model structures and updating methods

This paper comparatively assesses the performance of five data assimilation techniques for three-parameter Muskingum routing with a spatially lumped or distributed model structure. The assimilation techniques used include direct insertion (DI), nudging scheme (NS), Kalman filter (KF), ensemble Kalman filter (EnKF) and asynchronous ensemble Kalman filter (AEnKF), which are applied to river reaches in Texas and Louisiana, USA. For both lumped and distributed routing, results from KF, EnKF and AEnKF are sensitive to the error specification. As expected, DI outperformed the other models in the case of lumped modelling, while in distributed routing, KF approaches, particularly AEnKF and EnKF, performed better than DI or nudging, reflecting the benefit of updating distributed states through error covariance modelling in KF approaches. The results of this work would be useful in setting up data assimilation systems that employ increasingly abundant real-time observations using distributed hydrological routing models.     

An approach to handling non-Gaussianity of parameters and state variables in ensemble Kalman filtering

The ensemble Kalman filter (EnKF) is a commonly used real-time data assimilation algorithm in various disciplines. Here, the EnKF is applied, in a hydrogeological context, to condition log-conductivity realizations on log-conductivity and transient piezometric head data. In this case, the state vector is made up of log-conductivities and piezometric heads over a discretized aquifer domain, the forecast model is a groundwater flow numerical model, and the transient piezometric head data are sequentially assimilated to update the state vector. It is well known that all Kalman filters perform optimally for linear forecast models and a multiGaussian-distributed state vector. Of the different Kalman filters, the EnKF provides a robust solution to address non-linearities; however, it does not handle well non-Gaussian state-vector distributions. In the standard EnKF, as time passes and more state observations are assimilated, the distributions become closer to Gaussian, even if the initial ones are clearly non-Gaussian. A new method is proposed that transforms the original state vector into a new vector that is univariate Gaussian at all times. Back transforming the vector after the filtering ensures that the initial non-Gaussian univariate distributions of the state-vector components are preserved throughout. The proposed method is based in normal-score transforming each variable for all locations and all time steps. This new method, termed the normal-score ensemble Kalman filter (NS-EnKF), is demonstrated in a synthetic bimodal aquifer resembling a fluvial deposit, and it is compared to the standard EnKF. The proposed method performs better than the standard EnKF in all aspects analyzed (log-conductivity characterization and flow and transport predictions).     

Calibration of a Land Subsidence Model Using InSAR Data via the Ensemble Kalman Filter

The application of interferometric synthetic aperture radar (InSAR) has been increasingly used to improve capabilities to model land subsidence in hydrogeologic studies. A number of investigations over the last decade show how spatially detailed time‐lapse images of ground displacements could be utilized to advance our understanding for better predictions. In this work, we use simulated land subsidences as observed measurements, mimicking InSAR data to inversely infer inelastic specific storage in a stochastic framework. The inelastic specific storage is assumed as a random variable and modeled using a geostatistical method such that the detailed variations in space could be represented and also that the uncertainties of both characterization of specific storage and prediction of land subsidence can be assessed. The ensemble Kalman filter (EnKF), a real‐time data assimilation algorithm, is used to inversely calibrate a land subsidence model by matching simulated subsidences with InSAR data. The performance of the EnKF is demonstrated in a synthetic example in which simulated surface deformations using a reference field are assumed as InSAR data for inverse modeling. The results indicate: (1) the EnKF can be used successfully to calibrate a land subsidence model with InSAR data; the estimation of inelastic specific storage is improved, and uncertainty of prediction is reduced, when all the data are accounted for; and (2) if the same ensemble is used to estimate Kalman gain, the analysis errors could cause filter divergence; thus, it is essential to include localization in the EnKF for InSAR data assimilation.     

An adaptive hybrid EnKF-OI scheme for efficient state-parameter estimation of reactive contaminant transport models

Reactive contaminant transport models are used by hydrologists to simulate and study the migration and fate of industrial waste in subsurface aquifers. Accurate transport modeling of such waste requires clear understanding of the system’s parameters, such as sorption and biodegradation. In this study, we present an efficient sequential data assimilation scheme that computes accurate estimates of aquifer contamination and spatially variable sorption coefficients. This assimilation scheme is based on a hybrid formulation of the ensemble Kalman filter (EnKF) and optimal interpolation (OI) in which solute concentration measurements are assimilated via a recursive dual estimation of sorption coefficients and contaminant state variables. This hybrid EnKF-OI scheme is used to mitigate background covariance limitations due to ensemble under-sampling and neglected model errors. Numerical experiments are conducted with a two-dimensional synthetic aquifer in which cobalt-60, a radioactive contaminant, is leached in a saturated heterogeneous clayey sandstone zone. Assimilation experiments are investigated under different settings and sources of model and observational errors. Simulation results demonstrate that the proposed hybrid EnKF-OI scheme successfully recovers both the contaminant and the sorption rate and reduces their uncertainties. Sensitivity analyses also suggest that the adaptive hybrid scheme remains effective with small ensembles, allowing to reduce the ensemble size by up to 80% with respect to the standard EnKF scheme.     

Data assimilation with the EnKF in a 1-D numerical model of a North Sea station

A series of numerical experiments for data assimilation with the Ensemble Kalman Filter (EnKF) in a shallow water model are reported. Temperature profiles measured at a North Sea location, 55°30ˊ North and 0°55ˊ East (referred to as the CS station of the NERC North Sea project), are assimilated in 1-D simulations. Comparison of simulations without assimilation to model results obtained when assimilating data with the EnKF allows us to assess the filter performance in reproducing features of the observations not accounted for by the model. The quality of the model error sampling is tested as well as the validity of the Gaussian hypothesis underlying the analysis scheme of the EnKF. The influence of the model error parameters and the frequency of the data assimilation are investigated and discussed. From these experiments, a set of optimal parameters for the model error sampling are deduced and used to test the behavior of the EnKF when propagating surface information into the water column.