Stochastic differential dynamic programming for multi-reservoir system control

: As with all dynamic programming formulations, differential dynamic programming (DDP) successfully exploits the sequential decision structure of multi-reservoir optimization problems, overcomes difficulties with the nonconvexity of energy production functions for hydropower systems, and provides optimal feedback release policies. DDP is particularly well suited to optimizing large-scale multi-reservoir systems due to its relative insensitivity to state-space dimensionality. This advantage of DDP encourages expansion of the state vector to include additional multi-lag hydrologic information and/or future inflow forecasts in developing optimal reservoir release policies. Unfortunately, attempts at extending DDP to the stochastic case have not been entirely successful. A modified stochastic DDP algorithm is presented which overcomes difficulties in previous formulations. Application of the algorithm to a four-reservoir hydropower system demonstrates its capabilities as an efficient approach to solving stochastic multi-reservoir optimization problems. The algorithm is also applied to a single reservoir problem with inclusion of multi-lag hydrologic information in the state vector. Results provide evidence of significant benefits in direct inclusion of expanded hydrologic state information in optimal feedback release policies.     

Single multipurpose reservoir design: a modified optimal control problem by chance-constrained programming

This paper presents a chance-constrained programming model for optimal control of a multipurpose reservoir and its modification to a model for single reservoir design. An algorithm is developed for solving complex stochastic problems of multipurpose reservoir planning and design. The complexity of the problem is resolved by a two-step algorithm: (1) transformation of chance constraints on the state and control variables is performed at the first step; and (2) the choice of optimum control or optimal reservoir storage is carried out in the second step. The method of iterative convolution is chosen for the first step, while linear programming is selected for the second step. The algorithm allows the use of random inflows and random demands together with other deterministic demands. The reservoir design problem is presented as a modified optimal control problem. The procedure is illustrated with an example of a hypothetical reservoir design problem with three different types of downstream releases (hydropower production, municipal water supply, and irrigation).     

A new formulation for the stochastic control of systems with bounded state variables: An application to a single reservoir system

A new formulation is presented for the analysis of reservoir systems synthesizing concepts from the traditional stochastic theory of reservoir storage, moments analysis and reliability programming. The analysis is based on the development of the first and second moments for the stochastic storage state variable. These expressions include terms for the failure probabilities (probabilities of spill or deficit) and consider the storage bounds explicitly. Using this analysis, expected values of the storage state, variances of storage, optimal release policies and failure probabilities — useful information in the context of reservoir operations and design, can be obtained from a nonlinear programming solution. The solutions developed from studies of single reservoir operations on both an annual and monthly basis, compare favorably with those obtained from simulation. The presentation herein is directed to both traditional reservoir storage theorists who are interested in the design of a reservoir and modern reservoir analysts who are interested in the long term operation of reservoirs.     

Development of stochastic dynamic Nash game model for reservoir operation II. The value of players’ information availability and cooperative behaviors

Stochastic dynamic game models can be applied to derive optimal reservoir operation policies by considering interactions among water users and reservoir operator, their preferences, their levels of information availability and cooperative behaviors. The stochastic dynamic game model with perfect information (PSDNG) has been developed by [Ganji A, Khalili D, Karamouz M. Development of stochastic dynamic Nash game model for reservoir operation. I. The symmetric stochastic model with perfect information. Adv Water Resour, this issue]. This paper develops four additional versions of stochastic dynamic game model of water users interactions based on the cooperative behavior and hydrologic information availability of beneficiary sectors of reservoir systems. It is shown that the proposed models are quite capable of providing appropriate reservoir operating policies when compared with alternative operating models, as indicated by several reservoir performance characteristics. Among the proposed models, the selected model by considering cooperative behavior and additional hydrologic information (about the randomness nature of reservoir operation parameters), as exercised by reservoir operator, provides the highest attained level of performance and efficiency. Furthermore, the selected model is more realistic since it also considers actual behavior of water users and reservoir operator in the analysis.     

The role of hydrologic information in reservoir operation – Learning from historical releases

Closing the gap between theoretical reservoir operation and the real-world implementation remains a challenge in contemporary reservoir operations. Past research has focused on optimization algorithms and establishing optimal policies for reservoir operations. In this study, we attempt to understand operators’ release decisions by investigating historical release data from 79 reservoirs in California and the Great Plains, using a data-mining approach. The 79 reservoirs are classified by hydrological regions, intra-annual seasons, average annual precipitation (climate), ratio of maximum reservoir capacity to average annual inflow (size ratio), hydrologic uncertainty associated with inflows, and reservoirs’ main usage. We use information theory – specifically, mutual information – to measure the quality of inference between a set of classic indicators and observed releases at the monthly and weekly timescales. Several general trends are found to explain which sources of hydrologic information dictate reservoir release decisions under different conditions. Current inflow is the most important indicator during wet seasons, while previous releases are more relevant during dry seasons and in weekly data (as compared with monthly data). Inflow forecasting is the least important indicator in release decision making, but its importance increases linearly with hydrologic uncertainty and decreases logarithmically with reservoir size. No single hydrologic indicator is dominant across all reservoirs in either of the two regions.     

Probabilistic reservoir operation using Bayesian stochastic model and support vector machine

Available water resources are often not sufficient or too polluted to satisfy the needs of all water users. Therefore, allocating water to meet water demands with better quality is a major challenge in reservoir operation. In this paper, a methodology to develop operating strategies for water release from a reservoir with acceptable quality and quantity is presented. The proposed model includes a genetic algorithm (GA)-based optimization model linked with a reservoir water quality simulation model. The objective function of the optimization model is based on the Nash bargaining theory to maximize the reliability of supplying the downstream demands with acceptable quality, maintaining a high reservoir storage level, and preventing quality degradation of the reservoir. In order to reduce the run time of the GA-based optimization model, the main optimization model is divided into a stochastic and a deterministic optimization model for reservoir operation considering water quality issues.The operating policies resulted from the reservoir operation model with the water quantity objective are used to determine the released water ranges (permissible lower and upper bounds of release policies) during the planning horizon. Then, certain values of release and the optimal releases from each reservoir outlet are determined utilizing the optimization model with water quality objectives. The support vector machine (SVM) model is used to generate the operating rules for the selective withdrawal from the reservoir for real-time operation. The results show that the SVM model can be effectively used in determining water release from the reservoir. Finally, the copula function was used to estimate the joint probability of supplying the water demand with desirable quality as an evaluation index of the system reliability. The proposed method was applied to the Satarkhan reservoir in the north-western part of Iran. The results of the proposed models are compared with the alternative models. The results show that the proposed models could be used as effective tools in reservoir operation.     

Application of the gravity search algorithm to multi-reservoir operation optimization

Complexities in river discharge, variable rainfall regime, and drought severity merit the use of advanced optimization tools in multi-reservoir operation. The gravity search algorithm (GSA) is an evolutionary optimization algorithm based on the law of gravity and mass interactions. This paper explores the GSA's efficacy for solving benchmark functions, single reservoir, and four-reservoir operation optimization problems. The GSA's solutions are compared with those of the well-known genetic algorithm (GA) in three optimization problems. The results show that the GSA's results are closer to the optimal solutions than the GA's results in minimizing the benchmark functions. The average values of the objective function equal 1.218 and 1.746 with the GSA and GA, respectively, in solving the single-reservoir hydropower operation problem. The global solution equals 1.213 for this same problem. The GSA converged to 99.97% of the global solution in its average-performing history, while the GA converged to 97% of the global solution of the four-reservoir problem. Requiring fewer parameters for algorithmic implementation and reaching the optimal solution in fewer number of functional evaluations are additional advantages of the GSA over the GA. The results of the three optimization problems demonstrate a superior performance of the GSA for optimizing general mathematical problems and the operation of reservoir systems.     

Real-time multipurpose reservoir operation: a case study

Abstract

A real-time operational methodology has been developed for multipurpose reservoir operation for irrigation and hydropower generation with application to the Bhadra reservoir system in the state of Karnataka, India. The methodology consists of three phases of computer modelling. In the first phase, the optimal release policy for a given initial storage and inflow is determined using a stochastic dynamic programming (SDP) model. Streamflow forecasting using an adaptive AutoRegressive Integrated Moving Average (ARIMA) model constitutes the second phase. A real-time simulation model is developed in the third phase using the forecast inflows of phase 2 and the operating policy of phase 1. A comparison of the optimal monthly real-time operation with the historical operation demonstrates the relevance, applicability and the relative advantage of the proposed methodology.     

Multi-reservoir design using Pontryagin principle

Several successful applications of optimal control theory (OCT) based on the Pontryagin's minimum principle have been recorded in literature. These applications were focused on optimizing the operating policy of multi-reservoir systems. In this study, the performance of OCT algorithm in designing multi-reservoir system is investigated. Three deterministic optimization models based on the OCT were developed to design the best storage strategies in a multi-reservoir system to supply water. Multi-objective programming methods were implemented in the three models in order to consider the two non-commensurate objectives of minimizing cost and water deficit. The applications of these models to a multi-reservoir system were compared to an existing dynamic programming model. The result of this study showed that in all cases, the developed OCT models presented sub-optimal solution in designing multi-reservoir systems.     

Building more realistic reservoir optimization models using data mining - A case study of Shelbyville Reservoir

In this paper, we promote a novel approach to develop reservoir operation routines by learning from historical hydrologic information and reservoir operations. The proposed framework involves a knowledge discovery step to learn the real drivers of reservoir decision making and to subsequently build a more realistic (enhanced) model formulation using stochastic dynamic programming (SDP). The enhanced SDP model is compared to two classic SDP formulations using Lake Shelbyville, a reservoir on the Kaskaskia River in Illinois, as a case study. From a data mining procedure with monthly data, the past month’s inflow (Q_t−1), current month’s inflow (Q_t), past month’s release (R_t−1), and past month’s Palmer drought severity index (PDSI_t−1) are identified as important state variables in the enhanced SDP model for Shelbyville Reservoir. When compared to a weekly enhanced SDP model of the same case study, a different set of state variables and constraints are extracted. Thus different time scales for the model require different information. We demonstrate that adding additional state variables improves the solution by shifting the Pareto front as expected while using new constraints and the correct objective function can significantly reduce the difference between derived policies and historical practices. The study indicates that the monthly enhanced SDP model resembles historical records more closely and yet provides lower expected average annual costs than either of the two classic formulations (25.4% and 4.5% reductions, respectively). The weekly enhanced SDP model is compared to the monthly enhanced SDP, and it shows that acquiring the correct temporal scale is crucial to model reservoir operation for particular objectives.     

Optimization and uncertainty analysis of operational policies for multipurpose reservoir system

This paper presents optimization and uncertainty analysis of operation policies for Hirakud reservoir system in Orissa state, India. The Hirakud reservoir project serves multiple purposes such as flood control, irrigation and power generation in that order of priority. A 10-daily reservoir operation model is formulated to maximize annual hydropower production subjected to satisfying flood control restrictions, irrigation requirements, and various other physical and technical constraints. The reservoir operational model is solved by using elitist-mutated particle swarm optimization (EMPSO) method, and the uncertainty in release decisions and end-storages are analyzed. On comparing the annual hydropower production obtained by EMPSO method with historical annual hydropower, it is found that there is a greater chance of improving the system performance by optimally operating the reservoir system. The analysis also reveals that the inflow into reservoir is highly uncertain variable, which significantly influences the operational decisions for reservoir system. Hence, in order to account uncertainty in inflow, the reservoir operation model is solved for different exceedance probabilities of inflows. The uncertainty in inflows is represented through probability distributions such as normal, lognormal, exponential and generalized extreme value distributions; and the best fit model is selected to obtain inflows for different exceedance probabilities. Then the reservoir operation model is solved using EMPSO method to arrive at suitable operational policies corresponding to various inflow scenarios. The results show that the amount of annual hydropower generated decreases as the value of inflow exceedance probability increases. The obtained operational polices provides confidence in release decisions, therefore these could be useful for reservoir operation.     

An application of Karmarkar's interior-point linear programming algorithm for multi-reservoir operations optimization

Optimization of multi-reservoir systems operations is typically a very large scale optimization problem. The following are the three types of optimization problems solved using linear programming (LP): (i) deterministic optimization for multiple periods involving fine stage intervals, for example, from an hour to a week (ii) implicit stochastic optimization using multiple years of inflow data, and (iii) explicit stochastic optimization using probability distributions of inflow data. Until recently, the revised simplex method has been the most efficient solution method available for solving large scale LP problems. In this paper, we show that an implementation of the Karmarkar's interior-point LP algorithm with a newly developed stopping criterion solves optimization problems of large multi-reservoir operations more efficiently than the simplex method. For example, using a Micro VAX II minicomputer, a 40 year, monthly stage, two-reservoir system optimization problem is solved 7.8 times faster than the advanced simplex code in MINOS 5.0. The advantage of this method is expected to be greater as the size of the problem grows from two reservoirs to multiples of reservoirs. This paper presents the details of the implementation and testing and in addition, some other features of the Karmarkar's algorithm which makes it a valuable optimization tool are illuminated.     

An application of Karmarkar's interior-point linear programming algorithm for multi-reservoir operations optimization

Optimization of multi-reservoir systems operations is typically a very large scale optimization problem. The following are the three types of optimization problems solved using linear programming (LP): (i) deterministic optimization for multiple periods involving fine stage intervals, for example, from an hour to a week (ii) implicit stochastic optimization using multiple years of inflow data, and (iii) explicit stochastic optimization using probability distributions of inflow data. Until recently, the revised simplex method has been the most efficient solution method available for solving large scale LP problems. In this paper, we show that an implementation of the Karmarkar's interior-point LP algorithm with a newly developed stopping criterion solves optimization problems of large multi-reservoir operations more efficiently than the simplex method. For example, using a Micro VAX II minicomputer, a 40 year, monthly stage, two-reservoir system optimization problem is solved 7.8 times faster than the advanced simplex code in MINOS 5.0. The advantage of this method is expected to be greater as the size of the problem grows from two reservoirs to multiples of reservoirs. This paper presents the details of the implementation and testing and in addition, some other features of the Karmarkar's algorithm which makes it a valuable optimization tool are illuminated.     

A stochastic dynamic programming model with fuzzy storage states for reservoir operations

Application of stochastic dynamic programming (SDP) models to reservoir optimization calls for state variables discretization. Reservoir storage volume is an important variable whose discretization has a pronounced effect on the computational efforts. The error caused by storage volume discretization is examined by considering it as a fuzzy state variable. In this approach, the point-to-point transitions between storage volumes at the beginning and end of each period are replaced by transitions between storage intervals. This is achieved by using fuzzy arithmetic operations with fuzzy numbers. In this approach, instead of aggregating single-valued crisp numbers, the membership functions of fuzzy numbers are combined. Running a simulation model with optimal release policies derived from fuzzy and non-fuzzy SDP models shows that a fuzzy SDP with a coarse discretization scheme performs as well as a classical SDP having much finer discretized space. It is believed that this advantage in the fuzzy SDP model is due to the smooth transitions between storage intervals which benefit from soft boundaries.     

Development of stochastic dynamic Nash game model for reservoir operation. I. The symmetric stochastic model with perfect information

Increasing water demands, higher standards of living, depletion of resources of acceptable quality and excessive water pollution due to agricultural and industrial expansions have caused intense social and political predicaments, and conflicting issues among water consumers. The available techniques commonly used in reservoir optimization/operation do not consider interaction, behavior and preferences of water users, reservoir operator and their associated modeling procedures, within the stochastic modeling framework. In this paper, game theory is used to present the associated conflicts among different consumers due to limited water. Considering the game theory fundamentals, the Stochastic Dynamic Nash Game with perfect information (PSDNG) model is developed, which assumes that the decision maker has sufficient (perfect) information regarding the associated randomness of reservoir operation parameters. The simulated annealing approach (SA) is applied as a part of the proposed stochastic framework, which makes the PSDNG solution conceivable. As a case study, the proposed model is applied to the Zayandeh-Rud river basin in Iran with conflicting demands. The results are compared with alternative reservoir operation models, i.e., Bayesian stochastic dynamic programming (BSDP), sequential genetic algorithm (SGA) and classical dynamic programming regression (DPR). Results show that the proposed model has the ability to generate reservoir operating policies, considering interactions of water users, reservoir operator and their preferences.     

Reservoir performance under uncertainty in hydrologic impacts of climate change

Relatively few studies have addressed water management and adaptation measures in the face of changing water balances due to climate change. The current work studies climate change impact on a multipurpose reservoir performance and derives adaptive policies for possible future scenarios. The method developed in this work is illustrated with a case study of Hirakud reservoir on the Mahanadi river in Orissa, India, which is a multipurpose reservoir serving flood control, irrigation and power generation. Climate change effects on annual hydropower generation and four performance indices (reliability with respect to three reservoir functions, viz. hydropower, irrigation and flood control, resiliency, vulnerability and deficit ratio with respect to hydropower) are studied. Outputs from three general circulation models (GCMs) for three scenarios each are downscaled to monsoon streamflow in the Mahanadi river for two future time slices, 2045–65 and 2075–95. Increased irrigation demands, rule curves dictated by increased need for flood storage and downscaled projections of streamflow from the ensemble of GCMs and scenarios are used for projecting future hydrologic scenarios. It is seen that hydropower generation and reliability with respect to hydropower and irrigation are likely to show a decrease in future in most scenarios, whereas the deficit ratio and vulnerability are likely to increase as a result of climate change if the standard operating policy (SOP) using current rule curves for flood protection is employed. An optimal monthly operating policy is then derived using stochastic dynamic programming (SDP) as an adaptive policy for mitigating impacts of climate change on reservoir operation. The objective of this policy is to maximize reliabilities with respect to multiple reservoir functions of hydropower, irrigation and flood control. In variations to this adaptive policy, increasingly more weightage is given to the purpose of maximizing reliability with respect to hydropower for two extreme scenarios. It is seen that by marginally sacrificing reliability with respect to irrigation and flood control, hydropower reliability and generation can be increased for future scenarios. This suggests that reservoir rules for flood control may have to be revised in basins where climate change projects an increasing probability of droughts. However, it is also seen that power generation is unable to be restored to current levels, due in part to the large projected increases in irrigation demand. This suggests that future water balance deficits may limit the success of adaptive policy options.     

Evaluation of stochastic reservoir operation optimization models

This paper investigates the performance of seven stochastic models used to define optimal reservoir operating policies. The models are based on implicit (ISO) and explicit stochastic optimization (ESO) as well as on the parameterization–simulation–optimization (PSO) approach. The ISO models include multiple regression, two-dimensional surface modeling and a neuro-fuzzy strategy. The ESO model is the well-known and widely used stochastic dynamic programming (SDP) technique. The PSO models comprise a variant of the standard operating policy (SOP), reservoir zoning, and a two-dimensional hedging rule. The models are applied to the operation of a single reservoir damming an intermittent river in northeastern Brazil. The standard operating policy is also included in the comparison and operational results provided by deterministic optimization based on perfect forecasts are used as a benchmark. In general, the ISO and PSO models performed better than SDP and the SOP. In addition, the proposed ISO-based surface modeling procedure and the PSO-based two-dimensional hedging rule showed superior overall performance as compared with the neuro-fuzzy approach.     

Deriving reservoir operational strategies considering water quantity and quality objectives by stochastic fuzzy neural networks

By taking advantage of the close relationship between quality and quantity of water, we investigated the potential improvements of the in-reservoir water quality through the optimization of reservoir operational strategies. However, the few available techniques for optimization of reservoir operational strategies present some limitations, such as restrictions on the number of state/decision variables, the impossibility considering stochastic characteristics and difficulties for considering simulation/prediction models. One technique which presents great potential for overcoming some of these limitations is applied here and investigated for the first time in such complex system. The method, named stochastic fuzzy neural network (SFNN), can be defined as a fuzzy neural network (FNN) model stochastically trained by a genetic algorithm (GA) based model to yield a quasi optimal solution. The term “stochastically trained” refers to the introduction of a new loop within the training process which accounts for the stochastic variable of the system and its probabilities of occurrence. The SFNN was successfully applied to the optimization of the monthly operational strategies considering maximum water utilization and improvements on water quality simultaneous. Results showed the potential improvements on the water quality through means of hydraulic control.     

A combined simulation–genetic algorithm optimization model for optimal rule curves of a reservoir: a case study of the Nam Oon Irrigation Project,Thailand

A combined simulation–genetic algorithm (GA) optimization model is developed to determine optimal reservoir operational rule curves of the Nam Oon Reservoir and Irrigation Project in Thailand. The GA and simulation models operate in parallel over time with interactions through their solution procedure. A GA is selected as an optimization model, instead of traditional techniques, owing to its powerful and robust performance and simplicity in combining with a simulation technique. A GA is different from conventional optimization techniques in the way that it uses objective function information and does not require its derivatives, whereas in real‐world optimization problems the search space may include discontinuities and may often include a number of sub‐optimum peaks. This may cause difficulties for calculus‐based and enumerative schemes, but not in a GA. The simulation model is run to determine the net system benefit associated with state and control variables. The combined simulation–GA model is applied to determine the optimal upper and lower rule curves on a monthly basis for the Nam Oon Reservoir, Thailand. The objective function is maximum net system benefit subject to given constraints for three scenarios of cultivated areas. The monthly release is calculated by the simulation model in accordance with the given release policy, which depends on water demand. The optimal upper and lower rule curves are compared with the results of the HEC‐3 model (Reservoir System Analysis for Conservation model) calculated by the Royal Irrigation Department, Thailand, and those obtained using the standard operating policy. It was found that the optimal rule curves yield the maximum benefit and minimum damages caused by floods and water shortages. The combined simulation–GA model shows an excellent performance in terms of its optimization results and efficient computation. Copyright © 2007 John Wiley & Sons, Ltd.     

Optimal operation of reservoirs for downstream water quality control using linked simulation optimization

A methodology is developed for optimal operation of reservoirs to control water quality requirements at downstream locations. The physicochemical processes involved are incorporated using a numerical simulation model. This simulation model is then linked externally with an optimization algorithm. This linked simulation–optimization‐based methodology is used to obtain optimal reservoir operation policy. An elitist genetic algorithm is used as the optimization algorithm. This elitist‐genetic‐algorithm‐based linked simulation–optimization model is capable of evolving short‐term optimal operation strategies for controlling water quality downstream of a reservoir. The performance of the methodology developed is evaluated for an illustrative example problem. Different plausible scenarios of management are considered. The operation policies obtained are tested by simulating the resulting pollutant concentrations downstream of the reservoir. These performance evaluations consider various scenarios of inflow, permissible concentration limits, and a number of management periods. These evaluations establish the potential applicability of the developed methodology for optimal control of water quality downstream of a reservoir. Copyright © 2007 John Wiley & Sons, Ltd.