SIPVADE: A New Computer Programme with Seventeen Statistical Tests for Outlier Detection in Evaluation of International Geochemical Reference Materials and its Application to Whin Sill Dolerite WS-E from England and Soil-5 from Peru

A new computer programme was written in programming language TURBOC, which enables us to apply a procedure involving seventeen statistical tests (a total of sixty five single or multiple outlier versions of these tests) for outlier detection in univariate sample at a high confidence level of 99% (significance level α= 0.01). The outlying observations should be evaluated first for technical reasons and then rejected manually from the data base until no more outliers are detected and the final statistical parameters are computed from the remaining data. This programme has been used successfully to process two reference material data bases: WS-E from England and Soil-5 from Peru. The final mean values for WS-E are more reliable (characterized by smaller standard deviations and narrower confidence limits) than those obtained earlier using a different statistical approach. The application of a large number of statistical tests to Soil-5 also resulted in smaller standard deviation values for most elements than the method involving a limited number of such tests. For WS-E, some laboratories seem to have produced multiple data that were detected as statistical outliers. A close analysis of the distribution of outliers as a function of laboratory, country and analytical method leads to a technical justification for these outlying observations, probably in terms of inadequate QA/QC practices. Use of geochemical criteria indicates that the new mean values in WS-E might be closer to the "true" concentrations. This procedure of outlier detection and elimination is therefore recommended in the study of the existing RM.     

Some aspects of transformations of compositional data and the identification of outliers

The statistical analysis of compositional data is based on determining an appropriate transformation from the simplex to real space. Possible transfonnations and outliers strongly interact: parameters of transformations may be influenced particularly by outliers, and the result of goodness-of-fit tests will reflect their presence. Thus, the identification of outliers in compositional datasets and the selection of an appropriate transformation of the same data, are problems that cannot be separated. A robust method for outlier detection together with the likelihood of transformed data is presented as a first approach to solve those problems when the additive-logratio and multivariate Box-Cox transformations are used. Three examples illustrate the proposed methodology.     

Application of statistical geo-spatial information technology to soil stratification in the Seoul metropolitan area

Typical geotechnical testing results reflect the level of soil uncertainty, which requires statistical corrections of the data for an appropriate engineering decision. This study proposes frameworks to detect outlying data points using statistical analyses, the cross-validation-based method and the generalised extreme value distribution-based method. The borehole data regarding soil depth distribution in a central area of Seoul, South Korea are assessed to validate the aforementioned methods for comparison with the distribution-based method and the Moran scatterplot method. The results show that the proposed methods enable more reliable spatial distributions to be achieved with a quantitative evaluation of local reliability.     

Comparing the robustness of ordinary kriging and lognormal kriging: Outlier resistance

Ordinary kriging is well-known to be optimal when the data have a multivariate normal distribution (and if the variogram is known), whereas lognormal kriging presupposes the multivariate lognormality of the data. But in practice, real data never entirely satisfy these assumptions. In this article, the sensitivity of these two kriging estimators to departures from these assumptions and in particular, their resistance to outliers is considered. An outlier effect index designed to assess the effect of a single outlier on both estimators is proposed, which can be extended to other types of estimators. Although lognormal kriging is sensitive to slight variations in the sill of the variogram of the logs (i.e., their variance), it is not influenced by the estimate of the mean of the logs.This paper was presented at MGUS 87 Conference, Redwood City, California, 14 April 1987.     

Confidence level-based robust design of cantilever retaining walls in sand

Reliability-based geotechnical design entails accurate sample statistics (i.e., mean and standard deviation or coefficient of variation, denoted herein as cov) of soil parameters. However, the cov values of soil parameters are difficult to determine with confidence due to the limited availability of high-quality data and inherent spatial variability. As a result, estimated cov values of soil parameters can vary within a wide range, which can result in overdesign or underdesign. In this paper, a confidence level (CL)-based method is proposed to address the problem of geotechnical design in the face of uncertainty. Here, CL is a measure of confidence that the target reliability index will be satisfied in the face of uncertainty in the estimated cov. The proposed method is demonstrated through the design of a cantilever retaining wall in sand. To ensure the practicality of the proposed method, a simplified approach was developed, which requires little extra effort over that required for traditional reliability-based design. To develop the CL-based method further, a metric called the “true reliability index” is proposed, which is the actual reliability index in the face of the uncertainty in the estimated parameter statistics (mainly cov).     

Bayesian approach for calibrating transformation model from spatially varied CPT data to regular geotechnical parameter

Cone Penetration Test (CPT) is widely utilized to gain regular geotechnical parameters such as compression modulus, cohesion coefficient and internal friction angle by transformation model in the site investigation. However, it is challenging to obtain simultaneously the unknown coefficients and error of a transformation model, given the intrinsic uncertainty (i.e., spatial variability) of geomaterial and the epistemic uncertainty of geotechnical investigation. A Bayesian approach is therefore proposed calibrating the transformation model based on spatial random field theory. The approach consists of three key elements: (1) three-dimensional anisotropic spatial random field theory; (2) classifications of measurement and error, and the uncertainty propagation diagram of geotechnical investigation; and (3) the unknown coefficients and error calibration of the transformation model given Bayesian inverse modeling method. The massive penetration resistance data from CPT, which is denoted as a spatial random field variable to account for the spatial variability of soil, are classified as type A data. Meanwhile, a few laboratory test data such as the compression modulus are defined as type B data. Based on the above two types of data, the unknown coefficients and error of the transformation model are inversely calibrated with consideration of intrinsic uncertainty of geomaterial, epistemic uncertainties such as measurement errors, prior knowledge uncertainty of transformation model itself, and computing uncertainties of statistical parameters as well as Bayesian method. Baseline studying indicates the proposed approach is applicable to calibrate the transformation model between CPT data and regular geotechnical parameter within spatial random field theory. Next, the calibrated transformation model was compared with classical linear regression in cross-validation, and then it was implemented at three-dimensional site characterization of the background project.     

Probabilistic and fuzzy reliability analysis of a sample slope near Aliano

Slope stability assessment is a geotechnical problem characterized by many sources of uncertainty. Some of them, e.g., are connected to the variability of soil parameters involved in the analysis. Beginning from a correct geotechnical characterization of the examined site, only a complete approach to uncertainty matter can lead to a significant result. The purpose of this paper is to demonstrate how to model data uncertainty in order to perform slope stability analysis with a good degree of significance.

Once the input data have been determined, a probabilistic stability assessment (first-order second moment and Monte Carlo analysis) is performed to obtain the variation of failure probability vs. correlation coefficient between soil parameters. A first result is the demonstration of the stability of first-order second moment (FOSM) (both with normal and lognormal distribution assumption) and Monte Carlo (MC) solutions, coming from a correct uncertainty modelling. The paper presents a simple algorithm (Fuzzy First Order Second Moment, FFOSM), which uses a fuzzy-based analysis applied to data processing.     

Bayesian assessment of site‐specific performance of geotechnical design charts with unknown model uncertainty

Soil properties are indispensable input parameters in geotechnical design and analysis. In engineering practice, particularly for projects with relatively small or medium sizes, soil properties are often not measured directly, but estimated from geotechnical design charts using results of some commonly used laboratory or in situ tests. For example, effective friction angle ?′ of soil is frequently estimated using standard penetration test (SPT) N values and design charts relating SPT N values to ?′. Note that directly measured ?′ data are generally not available when (and probably why) the use of design charts is needed. Because design charts are usually developed from past observation data, on either empirical or semi‐theoretical basis, uncertainty is unavoidably involved in the design charts. This situation leads to two important questions in engineering practice: 1 how good or reliable are the soil properties estimated in a specific site when using the design charts? (or how to measure the performance of the design charts in a specific site?); and 2 how to incorporate rationally the model uncertainty when estimating soil properties using the design charts? This paper aims to address these two questions by developing a Bayesian statistical approach. In this paper, the second question is firstly addressed (i.e., soil properties are probabilistically characterized by rationally incorporating the model uncertainty in the design chart). Then, based on the characterization results obtained, an index is proposed to evaluate the site‐specific performance of design charts (i.e., to address the first question). Equations are derived for the proposed approach, and the proposed approach is illustrated using both real and simulated SPT data. Copyright © 2016 John Wiley & Sons, Ltd.     

Relative Efficiency of Single-Outlier Discordancy Tests for Processing Geochemical Data on Reference Materials and Application to Instrumental Calibrations by a Weighted Least-Squares Linear Regression Model

Numerous studies report geochemical data on reference materials (RMs) processed by outlier-based methods that use univariate discordancy tests. However, the relative efficiency of the discordancy tests is not precisely known. We used an extensive geochemical database for thirty-five RMs from four countries (Canada, Japan, South Africa and USA) to empirically evaluate the performance of nine single-outlier tests with thirteen test variants. It appears that the kurtosis test (N15) is the most powerful test for detecting discordant outliers in such geochemical RM databases and is closely followed by the Grubbs type tests (N1 and N4) and the skewness test (N14). The Dixon-type tests (N7, N8, N9 and N10) as well as the Grubbs type test (N2) depicted smaller global relative efficiency criterion values for the detection of outlying observations in this extensive database. Upper discordant outliers were more common than the lower discordant outliers, implying that positively skewed inter-laboratory geochemical datasets are more frequent than negatively skewed ones and that the median, a robust central tendency indicator, is likely to be biased especially for small-sized samples. Our outlier-based procedure should be useful for objectively identifying discordant outliers in many fields of science and engineering and for interpreting them accordingly. After processing these databases by single-outlier discordancy tests and obtaining reliable estimates of central tendency and dispersion parameters of the geochemical data for the RMs in our database, we used these statistical data to apply a weighted least-squares linear regression (WLR) model for the major element determinations by X-ray fluorescence spectrometry and compared the WLR results with an ordinary least-squares linear regression model. An advantage in using our outlier procedure and the new concentration values and uncertainty estimates for these RMs was clearly established.     

Method and analysis for the upscaling of structural data

3D geological models are created to integrate a set of input measurements into a single geological model. There are many problems with this approach, as there is uncertainty in all stages of the modelling process, from initial data collection to the approach used in the modelling scheme itself to calculate the geological model. This study looks at the uncertainty inherent in geological models due to data density and introduces a novel method to upscale geological data that optimises the information in the initial dataset. This method also provides the ability for the dominant trend of a geological dataset to be determined at different scales. By using self-organizing maps (SOM's) to examine the different metrics used to quantify a geological model, we allow for a larger range of metrics to be used compared to traditional statistical methods, due to the SOM's ability to deal with incomplete datasets. The classification of the models into clusters based on the geological metrics using k-means clustering provides a useful insight into the models that are most similar and models that are statistical outliers. Our approach is guided and can be calculated on any input dataset of this type to determine the effect that data density will have on a resultant model. These models are all statistical derivations that represent simplifications and different scales of the initial dataset and can be used to interrogate the scale of observations.     

常量金标准物质定值中离群值的统计识别

离群值的剔除常用数理统计的方法,如格拉布斯检验法和迪克逊检验法等,但是这些统计方法用于常量金标准物质分析结果的统计检验,都存在着对离群值剔除明显不够的问题.本文建立了以常量金重复分析相对偏差允许限为依据的离群值统计识别方法,包括统计计算待定值样品中金的算术平均值x和相对偏差允许限YG,确定合格的测定结果的数据区间,从而识别出离群值并予以剔除;一次剔除后,按照新的统计量确定下一轮的离群值剔除范围,直到无离群值后,给出金的平均值及其波动范围.以15个人工组合的常量金标准物质为例,模拟金标准物质定值分析,以密码形式分派给不同单位和分析者,共收集10套独立分析结果,采用本法剔除离群值后,所得金算术平均值与金标准参考值更加接近,其相对偏差的质量分数为0.35,达到优秀;而格拉布斯法(或迪克逊法)和中位值法的质量分数分别为0.42和0.40,只能达到良好.应用本文建立的离群值统计识别方法,质量分数等级有了明显提高,增强了数据统计分析的有效性.     

Global sensitivity analysis through polynomial chaos expansion of a basin-scale geochemical compaction model

We present a model-driven uncertainty quantification methodology based on sparse grid sampling techniques in the context of a generalized polynomial chaos expansion (GPCE) approximation of a basin-scale geochemical evolution scenario. The approach is illustrated through a one-dimensional example involving the process of quartz cementation in sandstones and the resulting effects on the dynamics of the vertical distribution of porosity, pressure, and temperature. The proposed theoretical framework and computational tools allow performing an efficient and accurate global sensitivity analysis (GSA) of the system states (i.e., porosity, temperature, pressure, and fluxes) in the presence of uncertain key mechanical and geochemical model parameters as well as boundary conditions. GSA is grounded on the use of the variance-based Sobol indices. These allow discriminating the relative weights of uncertain quantities on the global model variance and can be computed through the GPCE of the model response. Evaluation of the GPCE of the model response is performed through the implementation of a sparse grid approximation technique in the space of the selected uncertain quantities. GPCE is then be employed as a surrogate model of the system states to quantify uncertainty propagation through the model in terms of the probability distribution (and its statistical moments) of target system states.     

Probabilistic analysis of tunnel displacements based on correlative recognition of rock mass parameters

Displacement is vital in the evaluations of tunnel excavation processes,as well as in determining the postexcavation stability of surrounding rock masses.The prediction of tunnel displacement is a complex problem because of the uncertainties of rock mass properties.Meanwhile,the variation and the correlation relationship of geotechnical material properties have been gradually recognized by researchers in recent years.In this paper,a novel probabilistic method is proposed to estimate the uncertainties of rock mass properties and tunnel displacement,which integrated multivariate distribution function and a relevance vector machine(RVM).The multivariate distribution function is used to establish the probability model of related random variables.RVM is coupled with the numerical simulation methods to construct the nonlinear relationship between tunnel displacements and rock mass parameters,which avoided a large number of numerical simulations.Also,the residual rock mass parameters are taken into account to reflect the brittleness of deeply buried rock mass.Then,based on the proposed method,the uncertainty of displacement in a deep tunnel of CJPL-II laboratory are analyzed and compared with the in-situ measurements.It is found that the predicted tunnel displacements by the RVM model closely match with the measured ones.The correlations of parameters have significant impacts on the uncertainty results.The uncertainty of tunnel displacement decreases while the reliability of the tunnel increases with the increases of the negative correlations among rock mass parameters.When compared to the deterministic method,the proposed approach is more rational and scientific,and also conformed to rock engineering practices.     

岩土工程最小勘探数据量确定方法

准确地确定岩土设计参数统计特征值诸如均值、标准差是岩土工程可靠度分析与设计的重要前提。在满足岩土设计参数统计特征值计算精度条件下,文中提出了岩土工程最小勘探数据量的确定方法,定义了相对误差和相对变异性指标衡量岩土设计参数统计特征值计算准确性。系统地分析了静力触探试验数据量对砂土有效内摩擦角统计特征值计算精度的影响,并且根据相对误差和相对变异性指标确定了静力触探最小勘探数据量。研究结果表明,由静力触探试验间接估计砂土有效内摩擦角时均值相对误差较低,砂土有效内摩擦角相对变异性指标随静力触探试验数据量的增加而降低,即由认知不足引起的不确定性占总变异性的比值随静力触探试验数据量的增加而减小;当砂土有效内摩擦角容许相对变异性指标小于0.2时砂土有效内摩擦角在最大变异（COV=20%）与最小变异性（COV=5%）范围内,满足预定要求所需的最小静力触探试验数据量为10～100;若容许相对变异性指标小于0.3,所需的最小静力触探试验数据量为5～43。此外,间接估计岩土设计参数时经验回归模型不确定性对最小勘探数据量有显著影响。静力触探试验最小勘探数据量随经验回归模型不确定性的增大而增加,在确定岩土设计参数统计特征值时应尽量广泛收集勘探数据并选择精度较高的计算模型。     

Multivariate Spatial Outlier Detection Using Robust Geographically Weighted Methods

Outlier detection is often a key task in a statistical analysis and helps guard against poor decision-making based on results that have been influenced by anomalous observations. For multivariate data sets, large Mahalanobis distances in raw data space or large Mahalanobis distances in principal components analysis, transformed data space, are routinely used to detect outliers. Detection in principal components analysis space can also utilise goodness of fit distances. For spatial applications, however, these global forms can only detect outliers in a non-spatial manner. This can result in false positive detections, such as when an observation’s spatial neighbours are similar, or false negative detections such as when its spatial neighbours are dissimilar. To avoid mis-classifications, we demonstrate that a local adaptation of various global methods can be used to detect multivariate spatial outliers. In particular, we account for local spatial effects via the use of geographically weighted data with either Mahalanobis distances or principal components analysis. Detection performance is assessed using simulated data as well as freshwater chemistry data collected over all of Great Britain. Results clearly show value in both geographically weighted methods to outlier detection.     

Quasi-Stochastic Analysis of Uncertainty for Modelling Structurally Controlled Failures

In one approach to predicting the behaviour of rock masses, effort is being devoted to the use of probabilistic methods to model structures interior to a rock mass (sometimes referred to as ‘inferred’ or ‘stochastic’ structures). The physical properties of these structures (e.g. position, orientation, size) are modelled as random parameters, the statistical properties of which are derived from the measurements of a sample of the population (sometimes referred to as ‘deterministic’ structures). Relatively little attention has been devoted to the uncertainty associated with the deterministic structures. Typical geotechnical analyses rely on either an entirely stochastic analysis, or deterministic analyses representing the structures with a fixed shape (i.e. disc), position, size, and orientation. The simplifications assumed for this model introduce both epistemic and stochastic uncertainties. In this paper, it is shown that these uncertainties should be quantified and propagated to the predictions of behaviour derived from subsequent analyses. We demonstrate a methodology which we have termed quasi-stochastic analysis to perform this propagation. It is shown that relatively small levels of uncertainty can have large influence on the uncertainties associated with geotechnical analyses, such as predictions of block size and block stability, and therefore this methodology can provide the practitioner with a method for better interpretation of these results.     

马氏距离法在东昆仑东段多元异常圈定中的对比试验

马氏距离是一种多元异常识别方法,目前已有多种基于马氏距离的异常识别方法。笔者选择青海省东昆仑东段1∶50万水系沉积物测量数据,对比常规马氏距离、基于最小方差行列式(FMCD)的稳健马氏距离、基于校正的最小方差行列式的稳健马氏距离(Adaptive)和基于协中值的稳健马氏距离(Comedian)4种方法在识别Cu、Co、Cr、Ni、V、Fe,Cd、Cu、Mo、Pb、Zn、Ag和Au、As、Sb三种组合异常中的应用效果。结果显示,基于Comedian方法识别的异常效果最好,而常规方法识别的异常效果最差,因此Comedian方法是该区最有效的多元异常识别方法。     

先验概率分布及似然函数模型的选择对边坡可靠度评价影响的定量评估

受工程勘察成本及试验场地限制,可获得的试验数据通常有限,基于有限的试验数据难以准确估计岩土参数统计特征和边坡可靠度。贝叶斯方法可以融合有限的场地信息降低对岩土参数不确定性的估计进而提高边坡可靠度水平。但是,目前的贝叶斯更新研究大多假定参数先验概率分布为正态、对数正态和均匀分布,似然函数为多维正态分布,这种做法的合理性有待进一步验证。总结了岩土工程贝叶斯分析常用的参数先验概率分布及似然函数模型,以一个不排水黏土边坡为例,采用自适应贝叶斯更新方法系统探讨了参数先验概率分布和似然函数对空间变异边坡参数后验概率分布推断及可靠度更新的影响。计算结果表明：参数先验概率分布对空间变异边坡参数后验概率分布推断及可靠度更新均有一定的影响,选用对数正态和极值I型分布作为先验概率分布推断的参数后验概率分布离散性较小。选用Beta分布和极值I型分布获得的边坡可靠度计算结果分别偏于保守和危险,选用对数正态分布获得的边坡可靠度计算结果居中。相比之下,似然函数的影响更加显著。与其他类型似然函数相比,由多维联合正态分布构建的似然函数可在降低对岩土参数不确定性估计的同时,获得与场地信息更为吻合的计算结果。另外,构建似然函数时不同位置处测量误差之间的自相关性对边坡后验失效概率也具有一定的影响。     

Sixteen Statistical Tests for Outlier Detection and Rejection in Evaluation of International Geochemical Reference Materials: Example of Microgabbro PM-S

A totally objective procedure involving sixteen statistical tests (a total of thirty four single or multiple outlier versions of these tests) for outlier detection and rejection in a univariate sample is applied to a data base of sixty four elements in a recently issued international geochemical reference material (RM), a microgabbro PM-S from Scotland. This example illustrates the relative importance and usefulness of these tests in processing modern geochemical data for possible outliers and obtaining mean concentration and other statistical parameters from a final normal sample of univariate data. The final mean values are more reliable (characterized by smaller standard deviations and narrower confidence limits) than those obtained earlier using an accommodation approach (robust techniques) applied to this data base. Very high quality (certified value equivalent, cve) mean data are now obtained for eleven elements as well as high quality recommended values (rv) for thirty three elements in PM-S. Earlier work using the accommodation approach failed to establish even one cve value for any of the sixty four elements compiled here. The present procedure of outlier detection and elimination is therefore recommended in the study of RMs