A simulation study to examine the sensitivity of the Pettitt test to detect abrupt changes in mean

Abstract

The Pettitt test is a non-parametric test that has been used in a number of hydroclimatological studies to detect abrupt changes in the mean of the distribution of the variable of interest. This test is based on the Mann-Whitney two-sample test (rank-based test), and allows the detection of a single shift at an unknown point in time. This test is often used to detect shifts in extremes because of the lack of distributional assumptions. However, the downside of not specifying a distribution is that the Pettitt test may be inefficient in detecting breaks when dealing with extremes. Here we adopt a Monte Carlo approach to examine the sensitivity of the Pettitt test in detecting shifts in the mean under different conditions (location of the break within the series, magnitude of the shift, record length, level of variability in the data, extreme vs non-extreme records, and pre-assigned significance level). These simulation results show that the sensitivity of this test in detecting abrupt changes increases with the increase in the magnitude of the shift and record length. The number of detections is higher when the time series represents the central part of the distribution (e.g. changes in the time series of medians), while the skill decreases as we move toward either low or high extremes (e.g. changes in the time series of maxima). Furthermore, the number of detections decreases as the variability in the data increases. Finally, abrupt changes are more easily detected when they occur toward the center of the time series.

Editor D. Koutsoyiannis Associate editor K. Hamed     

Power of the Mann–Whitney test for detecting a shift in median or mean of hydro-meteorological data

 The non-parametric Mann–Whitney (MW) statistic test has been popularly used to assess the significance of a shift in median or mean of hydro-meteorological time series. It has been considered that the test is more suitable for non-normally distributed data and it may be not sensitive to the distribution type of sample data. However, no evidence has been provided to demonstrate these. This study investigates the power of the test in various circumstances by means of Monte Carlo simulation. Simulation results demonstrate that the power of the test is very sensitive to various properties of sample data. The power depends on the pre-assigned significance level, magnitude of a shift, sample size, and its occurrence position within a time series; and it is also strongly affected by the variation, skewness, and distribution type of a time series. The bigger the magnitude of a shift, the more powerful the test is; the larger the sample size, the more powerful the test is; and the bigger the variation within a time series, the less power the test has. The test has the highest power if a shift occurs at the midpoint of a time series. For the samples with different distribution types, the power of the test is dramatically different. The test has the highest power for time series with the extreme value type III (EV3) distribution while it indicates the lowest power for time series with the lognormal distribution.     

Bootstrap Pettitt test for detecting change points in hydroclimatological data: case study of Itaipu Hydroelectric Plant,Brazil

ABSTRACT

The Pettitt test is widely used in climate change and hydrological analyses. However, studies show difficulties of this test in detecting change points, especially in small samples. This study presents a bootstrap application of the Pettitt test, and compares it numerically with the classical Pettitt test by an extensive Monte Carlo simulation. The proposed test outperforms the classical test in all simulated scenarios. An application of the tests is conducted on the historical series of naturalized flows of the Itaipu Hydroelectric Plant in Brazil, for which several studies have shown a change point in the 1970s. When the series is split into shorter sub-series, to simulate actual situations of small samples, the proposed test is more powerful than the classical Pettitt test in detecting the change point. The proposed test can be an important tool for detecting abrupt changes in water availability, in support of hydroclimatological resources decision making.     

Changes in flood characteristics and the flood driving mechanism in the mountainous Haihe River Basin,China

ABSTRACT

Recently, the land surface in the Haihe River basin has changed, influencing the flood processes in the basin. To quantify this impact, seven typical sub-catchments were selected from different hydrological regions of the Haihe River basin for study. The non-parametric Mann-Kendall test was used to analyse for trends, and the non-parametric Pettitt test was adopted to detect any change point in the flood time series. Then, a hydrological model was established to simulate the effects of each potential driving factor on flood peak and volume. It was shown that flood peak and volume time series had decreased significantly, and the change point was around the year 1980. Groundwater depletion was not the main contribution to flood peak (FP) and volume (FV) decrease. In the Shifokou, Mubi and Lengkou sub-catchments, small hydraulic structures are the main driving factors for FP and FV decreasing. In the Xitaiyu, Daomaguan and Fuping sub-catchments, both land-use change and hydraulic structures are the main driving factors. The decreasing percentage decreases with the increase of the flood magnitude. The results provide valuable information for flood simulation and control in the Haihe River basin.     

Comparative analysis of nonparametric change-point detectors commonly used in hydrology

ABSTRACT

Several commonly-used nonparametric change-point detection methods are analysed in terms of power, ability and accuracy of the estimated change-point location. The analysis is performed with synthetic data for different sample sizes, two types of change and different magnitudes of change. The methods studied are the Pettitt method, a method based on the Cramér von Mises (CvM) two-sample test statistic and a variant of the CUSUM method. The methods differ considerably in behaviour. For all methods the spread of estimated change-point location increases significantly for points near one of the ends of the sample. Series of annual maximum runoff for four stations on the Yangtze River in China are used to examine the performance of the methods on real data. It was found that the CvM-based test gave the best results, but all three methods suffer from bias and low detection rates for change points near the ends of the series.     

The segmentation procedure as a tool for discrete modeling of hydrometeorological regimes

A possible cause of nonstationarity in time series is the existence of some abrupt modification of their statistical parameters, and especially of a sudden change of the mean. Series with such a change exhibit a strong temporal persistence, with high values of the Hurst coefficient, but with poor possibilities to fit any autoregressive model. Some classical tests (Pettitt, 1979; Buishand, 1982) enable to find a possible change point of the mean and then to split the original nonstationary series into two stationary sub-series. The Bayesian procedure defined by Lee and Heghinian (1977) supposes the “a-priori” existence of a change of the mean somewhere in the series and yields at each time step an “a-posteriori” probability of mean change. But these classical tests and procedures consider only one change point in the original series. To go further and to explore the theoretical multiple singularity models defined by Klemeš (1974) and Potter (1976), a segmentation procedure of time series has been designed. This procedure yields an optimal partition (from a least squares point of view) of the original series into as many subseries as possible, all differences between two contiguous means remaining simultaneously significant. This last requirement is ensured using the Scheffe test of contrasts. The main problem has been to master the combinatory explosion while exploring the tree of all possible segmentations of a series. Some applications of the procedure to hydrometeorological time series are reviewed and some possible improvements are presented.     

A framework for assessing flow regime alterations resulting from the effects of climate change and human disturbance

Flow regimes have been severely altered by climate change and human activities in recent decades, which has led to ecological degradation in rivers. This study proposes an analogy analysis-based framework, coupled with the Pettitt test, the indicators of hydrological alteration and the range of variation approach, which were used to distinguish the different effects. This framework was applied to the Sha River, a typical river in North China, to test its effectiveness. The results show that: (i) human disturbance had larger effects on pre-flood flow magnitude, the timing, frequency and duration of high and low pulse, and the flow change rate; (ii) climate change mainly influences the magnitude of flood and post-flood flows, and of extreme events; and (iii) the probability of high alteration from the target frequency increased by 69.7% due to the combined impacts. These results can provide valuable references for water resource and aquatic ecosystem management.     

Effect of autocorrelation on temporal trends in rainfall in a valley region at the foothills of Indian Himalayas

This study examines the effect of autocorrelation on step and monotonic trends in seasonal and annual rainfall. Initially, for step change, modified-Pettitt test is applied in two ways. First, using the corrected and unbiased trend-free-pre-whitening (TFPWcu) approach. Second, using a new approach in which time series is modelled by intervention analysis for modified Pettitt test. Subsequently, for monotonic trends, Mann–Kendall (MK) and six approaches of modified Mann–Kendall (MMK) test are applied to NCDC data for period 1901–2012 and its sub-periods. Approaches of MMK include pre-whitening (PW), trend-free-pre-whitening (TFPW), TFPWcu, two Variance Correction Approaches (VCAs) based on empirical formula (VCA:CF1) and Monte-Carlo-Simulations (VCA:CF2) and long term persistence (MK-LTP). A single change point is identified in 1970 for annual and monsoon rainfall from original and modified-Pettitt’s test using TFPWcu, while time series modelling approach has not exhibited any change point. Process shift in rainfall series is also studied using CUSUM and multiple change points are identified using Segment-Neighbourhood method. Outcomes of MMK show that TFPWcu is able to efficiently limit the effect of autocorrelation and may be preferred over PW and TFPW. The VCA:CF2 is not dependent on whole autocorrelation structure and corrects variance of all data series using lag-1 autocorrelation and may be preferred over VCA:CF1. MK-LTP considers long term persistence and it has exhibited presence of weaker trends than exhibited by other approaches. VCA:CF2 and MK-LTP are used to study trends of rainfall in Dehradun.     

基于SWAT模型的淮河上游流域设计洪水修订

变化环境下洪水序列的一致性遭到破坏,引发基于统计原理计算的设计洪水可靠性下降,亟需开展非一致性条件下的设计洪水修订研究.以淮河上游流域为研究区域,运用Pettitt检验法和滑动t检验法综合检测年最大洪峰流量序列突变点,在此基础上,采用SWAT分布式水文模型对变异前的洪峰与洪量序列进行还现,利用径流深的模拟结果修订设计洪...     

The influence of serial correlation on the Mann–Whitney test for detecting a shift in median

The non-parametric Mann–Whitney (MW) statistical test for assessing the significance of a shift in median or mean requires a tested series to be serially independent. However, hydrological time series such as water quality, streamflow, and others may frequently display serial correlation. In such cases, the existence of serial correlation might alter the ability of the test to detect a shift in mean. This study investigates this issue by means of the Monte Carlo simulation. Simulation results indicate that: (i) when there is no shift or a moderate shift in mean, the existence of positive serial correlation will increase the possibility to reject the null hypothesis of no shift while it might be true; and the existence of negative serial correlation will reduce the possibility to detect a shift; (ii) when a bigger shift occurs in a time series, for a series with smaller sample size, the influence of serial correlation on the test is similar to that in (i), but it is much less than that in (i); while for a series with larger sample size, the influence of serial correlation on the test is opposite to (i), i.e., positive serial correlation reduces the power of the test for detecting a shift while negative serial correlation slightly increases the power of the test for identifying a shift; and (iii) removal of serial correlation by pre-whitening can effectively remove the serial correlation and eliminate the influence of the serial correlation on the test.     

On robustness of large quantile estimates to largest elements of the observation series

The robustness of large quantile estimates of largest elements in a small sample by the methods of moments (MOM), L‐moments (LMM) and maximum likelihood (MLM) was evaluated and compared. Bias (B) and mean square error (MSE) were used to measure the estimation methods performance. Quantiles were estimated by eight two‐parameter probability distributions with the variation coefficient being the shape parameter. The effect of dropping largest elements of the series on large quantile values was assessed for various variation coefficient (C_V)/sample size (n) ‘combinations’ with n = 30 as the basic value. To that end, both the Monte Carlo sampling experiments and an asymptotic approach consisting in distribution truncation were applied. In general, both sampling and asymptotic approaches point to MLM as the most robust method of the three considered, with respect to bias of large quantiles. Comparing the performance of two other methods, the MOM estimates were found to be more robust for small and moderate hydrological samples drawn from distributions with zero lower‐bound than were the LMM estimates. Extending the evaluation to outliers, it was shown that all the above findings remain valid. However, using the MSE variation as a measure of performance, the LMM was found to be the best for most distribution/variation coefficient combinations, whereas MOM was found to be the worst. Moreover, removal of the largest sample element need not result in a loss of estimation efficiency. The gain in accuracy is observed for the heavy‐tailed and log‐normal distributions, being particularly distinctive for LMM. In practice, while dealing with a single sample deprived of its largest element, one should choose the estimation method giving the lowest MSE of large quantiles. For n = 30 and several distribution/variation coefficient combinations, the MLM outperformed the two other methods in this respect and its supremacy grew with sample size, while MOM was usually the worst. Copyright © 2006 John Wiley & Sons, Ltd.     

Quantitative assessment of the impact of climate variability and human activities on runoff changes for the upper reaches of Weihe River

In the wake of global and regional climate change and heightened human activities, runoff from some rivers in the world, especially in the arid and semi-arid regions, has significantly decreased. To reveal the varying characteristics leading to the change in runoff, detecting the influencing factors has been important in recent scientific discussions for water resources management in drainage basins. In this paper, an investigation into attributing the runoff response to climate change and human activities were conducted in two catchments (Wushan and Shetang), situated in the upper reaches of Weihe River in China. Prior to the identification of the factors that influenced runoff changes, the Mann–Kendall test was adopted to identify the trends in hydro-climate series. Also, change-points in the annual runoff were detected through Pettitt’s test and the precipitation–runoff double cumulative curve method. It is found that both catchments presented significant negative trend in annual runoff and the detected change-point in runoff occurs in 1993. Hence, the pre-change period and post-change period are defined before and after 1993, respectively. Then, runoff response to climate change and human activities was quantitatively evaluated on the basis of hydrologic sensitivity analysis and hydrologic model simulation. They provided similar estimates of the percentage change in mean annual runoff for the post-change period over the considered catchments. It is found that the decline in annual runoff over both catchments can be mainly attributed to the human activities, the reduction percentages due to human activities range from 59 to 77 %. The results of this study can provide a reference for the development, utilization and management of the regional water resources and ecological environment protection.     

Models of stream bed topography at the reach scale

A simple model of channel geometry is presented which focuses on the way that a single form variable may be autocorrelated with upstream values of that variable and which is amenable to analysis using time series methods. Stream bed topography at the local scale is the variable of interest. Bed height series were obtained for four stream lengths located at different positions along a mountain stream to test the hypothesis that, as control variables change spatially, the form of the model will also change. The Box-Jenkins models describing the series are dominated by autoregressive terms and indicate a non-random variation of bed height. An increase in the order and parameter magnitude of the models is associated with an increase in discharge and a decrease in bed material size, suggesting that the structure of the model reflects the adjustability of the stream bed and, specifically, the extent of riffle-pool development, although channel gradients may be too high for a well-defined riffle-pool sequence to be developed. Changes in the structure of models obtained for bed height and other form variables may indicate the variable ability of a stream to adjust its channel morphology.     

Quantifying the effects of climate variability and human activities on runoff from the Laohahe basin in northern China using three different methods

Much attention has recently been focused on the effects that climate variability and human activities have had on runoff. In this study, these effects are quantified using three methods, namely, multi‐regression, hydrologic sensitivity analysis, and hydrologic model simulation. A conceptual framework is defined to separate the effects. As an example, the change in annual runoff from the semiarid Laohahe basin (18 112 km²) in northern China was investigated. Non‐parametric Mann‐Kendall test, Pettitt test, and precipitation‐runoff double cumulative curve method were adopted to identify the trends and change‐points in the annual runoff from 1964 to 2008 by first dividing the long‐term runoff series into a natural period (1964–1979) and a human‐induced period (1980–2008). Then the three quantifying methods were calibrated and calculated, and they provided consistent estimates of the percentage change in mean annual runoff for the human‐induced period. In 1980–2008, human activities were the main factors that reduced runoff with contributions of 89–93%, while the reduction percentages due to changes in precipitation and potential evapotranspiration only ranged from 7 to 11%. For the various effects at different durations, human activities were the main reasons runoff decreased during the two drier periods of 1980–1989 and 2000–2008. Increased runoff during the wetter period of 1990–1999 is mainly attributed to climate variability. This study quantitatively separates the effects of climate variability and human activities on runoff, which can serve as a reference for regional water resources assessment and management. Copyright © 2011 John Wiley & Sons, Ltd.     

Streamflow estimation using optimal regional dependency function

The determination of spatial dependency of regionalized variable (ReV) is important in engineering studies. Regional dependency function that leads to calculation of weighting coefficients is required in order to make regional or point‐wise estimations. After obtaining this dependency function, it is possible to complete missing records in the time series and locate new measurement station. Also determination of regional dependency function is also useful to understand the regional variation of ReV. Point Cumulative Semi‐Variogram (PCSV) is another methodology to understand the regional dependency of ReV related to the magnitude and the location. However, this methodology is not useful to determine the weighting coefficient, which is required to make regional and point‐wise estimations. However, in Point Semi‐Variogram (PSV) proposed here, weighting coefficient depends on both magnitude and location. Although the regional dependency function has a fluctuating structure in PSV approach, this function gradually increases with distance in PCSV. The study area is selected in Mississippi river basin with 38 streamflow stations used for PCSV application before. It is aimed to compare two different geostatistical models for the same data set. PSV method has an ability to determine the value of variable along with optimum number of neighbour stations and influence radius. PSV and slope PSV approaches are compared with the PCSV. It was shown that slope slope point semi‐variogram (SPSV) approaches had relative error below 5%, and PSV and PCSV methods revealed relative errors below 10%. Copyright © 2009 John Wiley & Sons, Ltd.     

Block bootstrap for Mann–Kendall trend test of serially dependent data

Mann–Kendall (MK) test for trend detection must be modified when the data are serially correlated, to prevent the detection of false trends. Various approaches are developed for this purpose, such as prewhitening, trend‐free prewhitening, variance correction and block bootstrap. Each method has its own Type I and Type II errors. In this study, the errors of block bootstrapping MK test are estimated by a simulation study and compared with other methods. Optimal block length that minimizes the Type I error is determined as function of sample size and autocorrelation coefficient. It is shown that the power of block bootstrapping MK test is comparable with those of other modified MK tests. These tests are applied to some annual streamflow series with trend recorded in Turkish rivers, and their powers are compared. A modified form of the trend‐free prewhitening procedure is proposed that has a smaller Type I error. Copyright © 2011 John Wiley & Sons, Ltd.     

Spatiotemporal variations and abrupt changes of potential evapotranspiration and its sensitivity to key meteorological variables in the Wei River basin,China

Evapotranspiration is an important component of hydrological cycle and a key input to hydrological models. Therefore, analysis of the spatiotemporal variation of potential evapotranspiration (PET) will help a better understanding of climate change and its effect on hydrological cycle and water resources. In this study, the Penman–Monteith method was used to estimate PET in the Wei River basin (WRB) in China based on daily data at 21 meteorological stations during 1959–2008. Spatial distribution and temporal trends of annual and seasonal PET were analysed by using the Spline interpolation method and the Mann–Kendall test method. Abrupt changes were detected by using the Pettitt test method. In order to explore the contribution of key meteorological variables to the variation of PET, the sensitivity coefficients method was employed in this study. The results showed that: (1) mean annual and seasonal PET in the WRB was generally decreasing from northeast to southwest. Summer and spring made the major contributions to the annual values; (2) annual and seasonal PET series in most part of the WRB exhibited increasing trends; (3) abrupt changes appeared in 1993 for annual and spring PET series for the entire basin, while summer value series was detected in the late 1970s. (4) Relative humidity was the most sensitive variable for PET in general for the WRB, followed by wind speed, air temperature and solar radiation. In the headwater and outlet of the WRB, relative humidity and air temperature were the most sensitive variables to PET, while relative humidity and wind speed were more influential in most part of the middle‐lower region. Copyright © 2011 John Wiley & Sons, Ltd.     

Trend detection in hydrological time series by segment regression with application to Shiyang River Basin

Hydrological time series are generally subject to shift trends and abrupt changes. However, most of the methods used in the literature cannot detect both shift trends and abrupt changes simultaneously and have weak ability to detect multiple change points together. In this study, the segmented regression with constraints method, which can model both trend analysis and abrupt change detection, is introduced. The modified Akaike’s information criterion is used for model selection. As an application, the method is employed to analyse the mean annual temperature, precipitation, runoff and runoff coefficient time series in the Shiyang River Basin for the period from 1958 to 2003. The segmented regression model shows that the trends of the mean annual precipitation, temperature and runoff change over time, with different join (turning) points for different stations. The runoff pattern can potentially explained by the climate variables (precipitation and temperature). Runoff coefficients show slightly decreasing trends for Xiying, Huangyang, Gulang and Zamu catchments, slight increasing trends for Dongda and Dajing catchments and nearly no change for Xida catchment. No change points are found in runoff coefficient in all catchments.     

Effects of climate change and human activities on runoff and sediment inputs of the largest freshwater lake in China,Poyang Lake

The nonparametric Mann-Kendall test and the Pettitt test were employed to examine the change trends and shifts of runoff and sediment input to Poyang Lake between 1961 and 2013. Water balance and linear regression models were used to evaluate the impacts of climate variability and human activities on the runoff and sediment discharge changes. The results showed that runoff inputs to the lake had insignificant temporal trends and change points, while sediment inputs had significant decreasing trends, with an abrupt change in 1989. Quantitative assessment demonstrated that human activities led to a small decrease (5.5%) in runoff inputs to the lake, and a dramatic (121.4%) decrease in sediment inputs to the lake between the reference period (before the change point) and the human-influenced period (after the change point). This work provides a useful reference for future policy makers in water resource utilization and environmental safety of the Poyang Lake basin.     

基于高斯曲率模态相关系数的梁桥支座损伤识别研究

为方便地检测梁桥支座损伤,提出了利用运营桥梁实测模态位移结合其无损状态的模态位移判断支座损伤的高斯曲率模态相关系数法。通过简支梁桥的室内试验,验证了利用高斯曲率模态相关系数判定支座损伤的合理性以及该方法中无损状态下的模态位移可以通过模态试验和有限元模拟两种方法获得。利用该方法对实际简支梁桥和连续梁桥进行的支座损伤识别结果表明:高斯曲率模态相关系数法可准确识别出单支座和多支座损伤的支座损伤位置,具有较强的鲁棒性,可将此方法应用于实际工程中的支座损伤识别。