Expert judgment and climate forecasting: A methodological critique of “climate change to the year 2000”

The National Defense University's study of climate change to the year 2000 was based largely on the judgments of the members of two expert panels. Although the study has been widely distributed and apparently read by policy makers in the U.S. and abroad, the method of eliciting and analyzing expert judgment has not been critically reviewed. This paper uses the literature on judgment and subjective probability to evaluate the expert judgment methods used in the study.The National Center for Atmospheric Research is sponsored by the National Science Foundation.     

Unleashing expert judgment in assessment

Assessment evaluates accumulated knowledge and its limits. It informs and ideally empowers decisions and actions on complex, contested issues with persistent uncertainties. Applying rigorous expert judgment is an important dimension of assessment. Here we evaluate advances and challenges in approaches to expert judgment in the Intergovernmental Panel on Climate Change's Fifth Assessment Report (IPCC AR5). We find that revised guidance for author teams improved the development of balanced judgments on scientific evidence across disciplines. In particular, expert judgments underpinning conclusions are more extensively, transparently, and consistently communicated: degree-of-certainty terms are more abundant in AR5 policymaker summaries; wider ranges of possible outcomes are presented with greater inclusion of lower-certainty, decision-relevant findings; and expert judgments supporting conclusions are more comparable across working groups. But challenges in developing and communicating assessment conclusions persist, especially for findings with substantial uncertainties and for subjective aspects of judgments. Based on our evaluations and AR5 lessons learned, we propose a simpler, more rigorous framework for developing and communicating expert judgments in environmental assessment. We also describe practices for reducing expert-judgment biases, for advancing integration of evidence and expert judgment, and for addressing subjective dimensions of expert opinion directly and proactively.     

Uncertainty analysis comes to integrated assessment models for climate change…and conversely

This article traces the development of uncertainty analysis through three generations punctuated by large methodology investments in the nuclear sector. Driven by a very high perceived legitimation burden, these investments aimed at strengthening the scientific basis of uncertainty quantification. The first generation building off the Reactor Safety Study introduced structured expert judgment in uncertainty propagation and distinguished variability and uncertainty. The second generation emerged in modeling the physical processes inside the reactor containment building after breach of the reactor vessel. Operational definitions and expert judgment for uncertainty quantification were elaborated. The third generation developed in modeling the consequences of release of radioactivity and transport through the biosphere. Expert performance assessment, dependence elicitation and probabilistic inversion are among the hallmarks. Third generation methods may be profitably employed in current Integrated Assessment Models (IAMs) of climate change. Possible applications of dependence modeling and probabilistic inversion are sketched. It is unlikely that these methods will be fully adequate for quantitative uncertainty analyses of the impacts of climate change, and a penultimate section looks ahead to fourth generation methods.     

Bene-Eia: A Bayesian Approach to Expert Judgment Elicitation with Case Studies on Climate Change Impacts on Surface Waters

Climatic change impact studies are among the most complicated environmental assessments scientists have ever faced. The questions that policy makers face are enormous. There is plenty of experience and systematization in the environmental impact assessment (EIA) practice, especially at project level studies, but it has not been fully utilized in climatic change studies, we argue. Screening and scoping in EIA are typical examples. Beset by uncertainty and interdisciplinary divisions, climatic change impact analyses and policy assessments have been dominated by very detailed studies without the prior cross-sectorial, integrative phases that would aid in focusing the issues. Here, we present a probabilistic, Bayesian impact matrix approach (BeNe-EIA) for expert judgment elicitation, using belief networks from artificial intelligence. One or more experts are used to define a Bayesian prior distribution to each of the selected attributes, and the interattribute links, of the system under study. Posterior probabilities are calculated interactively, indicating consistency of the assessment and allowing iterative analysis of the system. Illustration is given by 2 impact studies of surface waters. In addition to climatic change studies, the approach has been designed to be applicable to conventional EIA. Insufficient attention has thus far been devoted to the probabilistic nature of the assessment and potential inconsistencies in expert judgment.     

利用层次分析法开展建筑物雷评的实例分析

本文在综合考虑建筑物雷电风险评价的一级指标和二级指标基础上,以某办公楼防雷装置为实例,采用AHP分析法建立雷电风险评价层次模型,通过对层次模型一级指标层和二级指标层指标建立判断矩阵,利用和法归一化和一致性检验法后得出各层指标相对权向量。最后采用Delphi专家调查法给予定量表示,使用数学方法计算反映出该办公楼雷电风险评价的综合得分。其结果分析表明,该办公楼雷电风险评价等级为临界安全型,须对其采取高等级的防护措施。AHP分析法将建筑物雷电风险的定性和定量评价有效地结合起来,使评价结论更加科学、客观、合理,对于深化建筑物雷电风险评价方法研究具有一定的借鉴意义。     

气候变化适应措施的选择与偏好分析——基于青藏高原生态功能保护区的调查

以青藏高原生态功能保护区为案例,基于专家判断,采用层次分析法（AHP）,对气候变化适应措施进行了优先性排序,结果表明生态功能保护气候变化专项资金、载畜量控制配套工程和产业结构调整规划是相对最重要的适应措施;决策与执行者、决策支持者和科学研究者对不同适应措施存在一定偏好差异,这些差异与不同利益相关者的职能、适应措施执行时所牵涉的利益关系等有一定联系。将AHP应用于适应措施择优的可行性探索对国家气候变化适应战略具有一定方法论意义。     

在Delphi中控制Excel电子表格技术的研究

探讨了在Delphi中控制Excel电子表格的方法和技术，以及Excel表导入数据库、数据库内容导出为Excel电子表格的方法。     

农业气象服务效益评估方法研究

科学、准确的对我国农业气象服务效益进行评估和分析,对政府和企业正确决策,提高农业气象防灾减灾能力,具有重要的作用和意义。结合农业气象服务的特点,采用剂量-反应分析和德尔菲法相结合的研究方法,研究提出了农业气象服务效益评估模式。该评估模式以典型单位的气象服务贡献率的实际测定值为参考值,运用专家评估法,由农业专家估测出气象服务在该农业生产中的贡献率,以此为基础分析评估出农业气象服务效益值。应用这一评估模式,对河南省农业气象服务效益进行了实证分析评估。结果表明,种植业气象服务贡献率为6.90 %,而后依次是牧业4.20 %、林业3.93 %、农林牧渔服务业3.36 %和渔业2.90 %。同时,分析研究了2007年以来,河南省农业气象服务效益的动态变化,研究表明,气象服务效益值随着农业产值的增加而不断提高,从2007年128.27亿元攀升至2010年191.65亿元,年均增速达到14.32 %。河南省农业气象服务平均贡献率保持在5.60 %-5.90 %之间,这一比率远高于全国,由此也说明了农业气象服务对河南省农业生产的重要作用。     

Objective probabilities about future climate are a matter of opinion

In this paper, the unfeasibility of producing “objective” probabilistic climate change scenarios is discussed. Realizing that the knowledge of “true” probabilities of the different scenarios and temperature changes is unachievable, the objective must be to find the probabilities that are the most consistent with what our state of knowledge and expert judgment are. Therefore, subjective information plays, and should play, a crucial role. A new methodology, based on the Principle of Maximum Entropy, is proposed for constructing probabilistic climate change scenarios when only partial information is available. The objective is to produce relevant information for decision-making according to different agents’ judgment and subjective beliefs. These estimates have desirable properties such as: they are the least biased estimate possible on the available information; maximize the uncertainty (entropy) subject to the partial information that is given; The maximum entropy distribution assigns a positive probability to every event that is not excluded by the given information; no possibility is ignored. The probabilities obtained in this manner are the best predictions possible with the state of knowledge and subjective information that is available. This methodology allows distinguishing between reckless and cautious positions regarding the climate change threat.     

对流天气临近预报技术的发展与研究进展

目前,临近预报技术主要包括雷暴识别追踪和外推预报技术、数值预报技术以及以分析观测资料为主的概念模型预报技术等。其中,识别追踪和外推预报技术主要以雷达资料为基础,在这方面,交叉相关外推和回波特征追踪识别外推是比较成熟的技术,已经用于许多的临近预报业务系统中,其缺陷是预报时效较短,准确率也不是很高。随着精细数值天气预报技术和计算机技术的发展,利用多普勒天气雷达资料和其它中小尺度观测资料进行数值模式初始化,来预报雷暴的发生、发展和消亡已经成为一个研究的热点,该技术发展很快但还不成熟。概念模型预报技术主要是通过综合分析多种中小尺度观测资料,包括雷达和气象卫星资料等,在此基础上建立雷暴发生、发展和消亡的概念模型,特别是边界层辐合线和强对流的密切关系等,再结合数值模式分析预报和其它外推技术的结果,然后建立雷暴临近预报的专家系统,其不但可以获取雷暴和对流降水移动、发展的信息,还可以预报它们的生成和消亡。检验和定性评估也表明,将多种资料和技术集于一体的概念模型专家系统,其临近预报的准确率最高,时效也最长,是临近预报技术未来发展的主要趋势之一。NCAR的Auto Nowcaster系统是雷暴临近预报概念模型专家系统的一个典型代表。     

Making decisions to conserve species under climate change

Severe impacts on biodiversity are predicted to arise from climate change. These impacts may not be adequately addressed by conventional approaches to conservation. As a result, additional management actions are now being considered. However, there is currently limited guidance to help decision makers choose which set of actions (and in what order) is most appropriate for species that are considered to be vulnerable. Here, we provide a decision framework for the full complement of actions aimed at conserving species under climate change from ongoing conservation in existing refugia through various forms of mobility enhancement to ex situ conservation outside the natural environment. We explicitly recognize that allocation of conservation resources toward particular actions may be governed by factors such as the likelihood of success, cost and likely co-benefits to non-target species in addition to perceived vulnerability of individual species. As such, we use expert judgment of probable tradeoffs in resource allocation to inform the sequential evaluation of proposed management interventions.     

德尔斐法在江苏省风电行业气象服务效益评估中的应用

利用德尔斐法和对比分析法,选取江苏龙源风力发电有限公司为典型调查单位,对江苏风电行业气象服务效益进行实际评估,结果表明,德尔斐法简单易行、便于操作;在风电运营的各环节中,风电生产环节对气象服务需求较高,尤以生产维护、防大风、雷暴等灾害性天气最为明显.     

A survey of census bureau population projection methods

Population projections methods of the U.S. Census Bureau draw upon several different traditions of forecasting: demographic accounting, judgmental, time series, deterministic, and explanatory. This paper reviews each of the forecasting traditions in population projections, describes the U.S. Census Bureau's current methods for national and state population projections, and proposes new hybrid approaches such as demographic-time series methods for national fertility projections and economic-demographic methods for state migration projections. Throughout the article, possible parallels with forecasting in other disciplines are noted.     

冬小麦冬季干旱时段灌溉气象适宜指数研究

对1955-2010年宿州市冬季降水量和气温变化的数理统计分析结果表明：宿州市冬小麦冬季仍处于缓慢生长状态,特别是在干旱和暖冬的双重影响下,冬季麦田一般需要灌溉一次。在此基础上,采用德尔菲法（Delphi technique）,确定了影响冬季麦田实施灌溉的降水、日最低气温和风速3个气象要素的各自的量化等级和影响灌溉适宜与否的权重系数,给出了冬季干旱时段灌溉气象适宜指数概念和计算模型,再结合天气预报产品的释用方法,得到了冬小麦冬季干旱时段灌溉农用天气预报产品。经实例应用,可适应对外发布冬季麦田灌溉农用天气产品的需要。     

利用专家评估法作未来10年（2000～2010年）广州市气象灾害影响趋势展望

将气象灾害（主要是风灾、涝灾和旱灾）对广州市的影响程度分成轻微、偏轻、中等、偏重和严重五个级别，在时间上将未来10年分成三个时期：近期（2000～2002年）、中期（2003～2006年）、远期（2007～2010年），通过对广州市57名长期从事灾害研究的专家的调查，再运用专家评估法，结果表明：在三种灾害中，对广州市影响最大的是风灾，未来10年几乎每年都有热带气旋影响广州，造成较大程度的损失，其损失程度约比中等年份偏多10～15%;涝灾对广州市的影响在近期比台风小，中期与台风的影响接近，至远期的影响超过台风，整个预测期内比中等年份偏多10%左右。对广州市影响最小的是旱灾（特别在近期和远期），比中等年份偏少5%左右。     

许昌市区域性暴雨集成预报系统

以新的预报流程和预报技术路线为指导，使用省台指导预报和数值预报产品等资料，通过专家系统、MOS、完全预报等预报方法，分别建立许昌市区域性暴雨预报诊断模型，在此基础上，使用VB6．0语言编程，建立许昌市区域性暴雨集成预报系统。     

Response to commentary by J. L. Bamber,W. P. Aspinall and R. M. Cooke (2016)

In a commentary paper, Bamber et al. (Nat Clim Change 3:424–427, 2016) respond to our recent assessment (De Vries and Van de Wal Clim Change 1–14, 2015) of their expert judgment based study on projections of future sea level rise due to the melting of the large ice sheets (Bamber and Aspinall Nat Clim Chang 3:424–427, 2013). In this response we comment on their remarks.     

用层次分析法确定气象服务评价指标权重

在气象服务评价体系中,以气象服务总体评价为目标层,分层次确立一级和二级评价指标,利用2010年全国公众气象服务满意度调查中评价指标数据,应用层次分析法 (AHP) 中的9分位标度法和0.618标度法来构建判断矩阵,分析了各级评价指标的权重系数确定方法。分析发现,当评价指标少于5个时,可采用9分位标度法构建判断矩阵,以确定各评价指标的权重值;而当评价指标达到5个时,需采用0.618标度法来构建判断矩阵,计算出相应的权向量即为权重系数。将上述方法应用于2011年公众气象服务满意度调查数据中,前后两年主要评价指标对比显示,气象服务信息内容的权重在增加,而气象知识宣传普及的权重有所减小,即公众对于气象服务信息内容更为重视,同时气象部门对于公众的气象知识宣传普及工作也取得了成效。     

Differentiating theory from evidence in determining confidence in an assessment finding

The Guidance Notes for Lead Authors of the IPCC Fifth Assessment Report on Consistent Treatment of Uncertainties describes a process for consistently evaluating and communicating levels of certainty in findings. The process begins with an assessment of the scientific evidence and agreement supporting a finding, where evidence is defined as including mechanistic understanding, theory, data, models, and expert judgment. The appropriateness of categorizing theory as one line of evidence varies by scientific discipline; for the natural and social sciences, developing theory and collecting data are different steps in the scientific method. Further, decision-makers often find it valuable for scientists to differentiate situations where a theory is generally agreed but for which supporting data are limited, from situations where empirical data lack an explanatory theory. The paper describes the approach used by the International Agency for Research on Cancer (IARC) for assessing the relative robustness of a theory separately from the strength and quality of its supporting evidence, and then developing consensus statements of whether an agent is a human carcinogenic. Although the IARC and IPCC processes are very similar, the IARC process also differs by combining theory, evidence, and agreement as equal partners in a limited set of standardized categories of confidence. Incorporating aspects of the IARC approach into the IPCC guidance could improve the evaluation and communication of theory, evidence, and agreement in future versions of the uncertainty guidance.     

Climate change and road freight safety: a multidisciplinary exploration

The freight and logistics sector is of significant importance as an enabler and driver of the global economy, but it is also inherently vulnerable to hazardous weather. Despite this, there is currently no quantitative assessment of how climate change may affect the sector. This paper applies multidisciplinary climate change impact assessment tools and conceptual frameworks to the road freight sector of Great Britain in order to identify potential future weather-related safety issues. Relationships between weather and freight accidents are determined using road accident data and meteorological observations, which are then used with climate change scenarios to arrive at projections of possible impacts across the regions of Great Britain. Included in the study are industry perceptions of future trends within the sector and wider economy which many affect freight’s exposure and sensitivity to weather. These are elicited through interviews and an iterative expert Delphi study. Hence, unlike many other climate change impact assessments, this innovative study takes into account the potentially significant impact of socio-economic change (including institutional and operational). The results show that summer precipitation and winter ice-related accidents are likely to decrease across most of the country, whereas winter rain-related accidents are projected to increase. However, it is postulated that some of the impacts of climate change will be modified by reflexive behavioural change on the part of the driver and either institutional adaptation or complacency on the part of the road authorities. The paper concludes by framing the study in a range of future scenarios outlining how the socio-economic environment could influence the road transport network and how it is used, modifying the impact of climate change.