A spatiotemporal indexing approach for efficient processing of big array-based climate data with MapReduce

Climate observations and model simulations are producing vast amounts of array-based spatiotemporal data. Efficient processing of these data is essential for assessing global challenges such as climate change, natural disasters, and diseases. This is challenging not only because of the large data volume, but also because of the intrinsic high-dimensional nature of geoscience data. To tackle this challenge, we propose a spatiotemporal indexing approach to efficiently manage and process big climate data with MapReduce in a highly scalable environment. Using this approach, big climate data are directly stored in a Hadoop Distributed File System in its original, native file format. A spatiotemporal index is built to bridge the logical array-based data model and the physical data layout, which enables fast data retrieval when performing spatiotemporal queries. Based on the index, a data-partitioning algorithm is applied to enable MapReduce to achieve high data locality, as well as balancing the workload. The proposed indexing approach is evaluated using the National Aeronautics and Space Administration (NASA) Modern-Era Retrospective Analysis for Research and Applications (MERRA) climate reanalysis dataset. The experimental results show that the index can significantly accelerate querying and processing (~10× speedup compared to the baseline test using the same computing cluster), while keeping the index-to-data ratio small (0.0328%). The applicability of the indexing approach is demonstrated by a climate anomaly detection deployed on a NASA Hadoop cluster. This approach is also able to support efficient processing of general array-based spatiotemporal data in various geoscience domains without special configuration on a Hadoop cluster.     

Estimating the Great Lakes net radiation using satellite remote sensing and MERRA reanalysis

ABSTRACT

The net all-wave radiation of the Great Lakes (GL) is a key to understanding the effects of climate change on the GL. There is a high possibility of underestimating the net all-wave radiation of the GL when using existing methodologies with inputs from near-shore and land-based meteorological data. This study provides the first technique to estimate net all-wave radiation over the GL from July 2001 to December 2014 using a combination of data from satellite remote sensing, reanalysis data sets, and direct measurements. The components of the surface radiation budget estimated from the proposed method showed good statistical agreement. The instantaneous net radiation estimated by our methods was compared with the in situ measurements from June 2008 to April 2012 (Stannard Rock Lighthouse: SR) and September 2009–April 2011 (Spectacle Reef Lighthouse: SP). The comparisons from SR and SP also showed strong statistic agreement (R²?=?0.74 and 0.7; RMSE?=?9.26 and 10.60?W?m^?2 respectively). Monthly spatial variations of net shortwave radiation varied with cloud cover and surface albedo while net longwave radiation varied with the temperature difference between the water surface and the atmosphere.     

南亚高压在NCEP-2和MERRA再分析资料中的对比分析

通过双线性插值、相关分析、Morlet小波分析等分析方法,围绕新高分辨率再分析资料MERRA的可用性,对比分析了NASA高分辨率再分析资料MERRA和NCEP-DOE（NCEP-2）再分析资料中100hPa南亚高压的活动特征.结果表明：100hPa上,NCEP-2和MERRA资料南亚高压的特征线数值不同,NCEP-2资料为1680dgpm,比MERRA资料大4dgpm,但MERRA资料南亚高压的范围明显大于NCEP-2资料.除东伸指数外,NCEP-2和MERRA资料反映的南亚高压同一特征指数年际变化趋势基本一致,特别是两种资料南亚高压脊线指数的年际变化曲线基本重合.以20世纪90年代初为界,之前,NCEP-2资料南亚高压东伸指数、面积指数和强度指数正异常,MERRA资料南亚高压东伸指数、面积指数和强度指数负异常,NCEP-2中的指数值大于MERRA中对应的指数值;之后,反之.NCEP-2和MERRA资料南亚高压面积指数、强度指数的气候均值间存在显著差异.NCEP-2和MERRA资料南亚高压强度指数的方差间存在显著差异.两种资料反映的夏季南亚高压同一特征指数的显著周期在1979～2009年有很好的一致性：都具有相同的显著周期,并且位相也基本吻合,但两种资料在反映南亚高压主周期特征上存在一定差异：南亚高压面积指数、强度指数在MERRA资料中以准4年周期为主,在NCEP-2资料中则同时表现为准4年和8～9年两个周期.     

新疆夏季降水时空分布的适用性评估

新疆气象站点稀疏且分布不均,高精度时空气象数据缺乏。基于数据同化的再分析资料,可成为解决这一问题的有效途径。利用美国国家环境预报中心再分析数据(CFSR)、欧洲中期数值预报中心再分析数据(ERA-Interim)和美国国家航空航天局再分析数据(MERRA)中的降水数据,分别与1979-2007年新疆气象观测数据和日本气象厅高分辨率亚洲陆地降水数据(APHRO)进行数理统计分析,评估了这3套再分析数据在新疆的适用性。3套再分析数据可有效表征新疆大部分地区年内降水的时空分布特征,夏季降水偏差小于100%;但未能捕捉到夏季降水的长期趋势。夏季降水的偏差与高程具有显著的相关性,这可为订正3套再分析数据、提高降水数据的精度提供技术支撑。     

ERA5-Land、MERRA2气温、降水数据在呼伦贝尔地区的适用性研究

利用1981—2019年呼伦贝尔地区16个气象站观测数据，结合相关系数、均方根误差、散点斜率等方法，对ERA5-Land和MERRA2再分析月气温、降水数据的区域适用性进行评估。研究表明：ERA5-Land、MERRA2两套再分析气温、降水数据与观测数据对比，整体相关性较高、误差较低、本地适用性较高，但也均存在低观测值时再分析数据的高估和高观测值时再分析数据的低估现象。ERA5-Land的月气温数据略优于MERRA2，而月降水数据差于MERRA2。ERA5-Land的年降水量在2005年之后出现明显的低估，降水量被低估的月份主要为6—8月，建议在再分析数据使用前对其进行系统偏差订正，以提高数据的适用性。