A hierarchical indexing strategy for optimizing Apache Spark with HDFS to efficiently query big geospatial raster data

ABSTRACT

Earth observations and model simulations are generating big multidimensional array-based raster data. However, it is difficult to efficiently query these big raster data due to the inconsistency among the geospatial raster data model, distributed physical data storage model, and the data pipeline in distributed computing frameworks. To efficiently process big geospatial data, this paper proposes a three-layer hierarchical indexing strategy to optimize Apache Spark with Hadoop Distributed File System (HDFS) from the following aspects: (1) improve I/O efficiency by adopting the chunking data structure; (2) keep the workload balance and high data locality by building the global index (k-d tree); (3) enable Spark and HDFS to natively support geospatial raster data formats (e.g., HDF4, NetCDF4, GeoTiff) by building the local index (hash table); (4) index the in-memory data to further improve geospatial data queries; (5) develop a data repartition strategy to tune the query parallelism while keeping high data locality. The above strategies are implemented by developing the customized RDDs, and evaluated by comparing the performance with that of Spark SQL and SciSpark. The proposed indexing strategy can be applied to other distributed frameworks or cloud-based computing systems to natively support big geospatial data query with high efficiency.     

SMAP卫星海表盐度产品的检验与校正

In this study, sea surface salinity(SSS) Level 3(L3) daily product derived from soil moisture active passive(SMAP)during the year 2016, was validated and compared with SSS daily products derived from soil Moisture and ocean salinity(SMOS) and in-situ measurements. Generally, the root mean square error(RMSE) of the daily SSS products is larger along the coastal areas and at high latitudes and is smaller in the tropical regions and open oceans. Comparisons between the two types of daily satellite SSS product revealed that the RMSE was higher in the daily SMOS product than in the SMAP, whereas the bias of the daily SMOS was observed to be less than that of the SMAP when compared with Argo floats data. In addition, the latitude-dependent bias and RMSE of the SMAP SSS were found to be primarily influenced by the precipitation and the sea surface temperature(SST). Then, a regression analysis method which has adopted the precipitation and SST data was used to correct the larger bias of the daily SMAP product. It was confirmed that the corrected daily SMAP product could be used for assimilation in high-resolution forecast models, due to the fact that it was demonstrated to be unbiased and much closer to the in-situ measurements than the original uncorrected SMAP product.     

2017年春夏季西北太平洋浮游植物的区域差异及其与不同水团的关系

The West Pacific Ocean is considered as the provenance center of global marine life and has the highest species diversity of numerous marine taxa. The phytoplankton, as the primary producer at the base of the food chain,effects on climate change, fish resources as well as the entire ecosystem. However, there are few large-scale surveys covering several currents with different hydrographic characteristics. This study aimed to explore the relationships between the spatio-temporal variation in phytoplankton community structure and different water masses. A total of 630 water samples and 90 net samples of phytoplankton were collected at 45 stations in the Northwest Pacific Ocean(21.0°–42.0°N, 118.0°–156.0°E) during spring and summer 2017. A total of 281 phytoplankton taxa(5 μm) belonging to 61 genera were identified in the study area. The distribution pattern of the phytoplankton community differed significantly both spatially and temporally. The average abundances of phytoplankton in spring and summer were 797.07×10~2 cells/L and 84.94×10~2 cells/L, respectively. Whether in spring or summer, the maximum abundance always appeared in the northern transition region affected by the Oyashio Current, where nutrients were abundant and diatoms dominated the phytoplankton community;whereas the phytoplankton abundance was very low in the oligotrophic Kuroshio region, and the proportion of dinoflagellates in total abundance increased significantly. The horizontal distribution of phytoplankton abundance increased from low to high latitudes, which was consistent with the trend of nutrient distributions, but contrary to that of water temperature and salinity. In the northern area affected by the Oyashio Current, the phytoplankton abundance was mainly concentrated in the upper 30 m of water column, while the maximum abundance often occurred at depths of 50–75 m in the south-central area affected by the Kuroshio Current.Pearson correlation and redundancy analysis(RDA) showed that phytoplankton abundance was significant negatively correlated with temperature and salinity, but positively correlated with nutrient concentration. The phytoplankton community structure was mainly determined by nutrient availability, especially the N:P ratio.     

2018?2019年舟山近海浮游动物群落结构春季年际变化及其与水团的关系

根据2018?2019年春季两个航次在舟山近海进行的浮游生物调查结果,对舟山近海的浮游动物群落结构(类群组成、优势种数量)年际变化进行了研究,利用典范对应分析(Canonical Correspondence Analysis, CCA)研究了两年春季浮游动物类群组成差异、优势种变化的原因,初步探讨了春季浮游动物群落结构动态变化的机制。结果表明:根据表层温度(Sea Surface Temperature,SST)、表层盐度(Sea Surface Salinity,SSS)的聚类分析,将该区域分为3个水团:杭州湾内水团(I区)、舟山本岛上升流水团II区)、舟山近海水团(III区)。不同水团对浮游动物类群组成影响显著,引起2018年和2019年春季3个水团区差异的主要贡献种(贡献率>10%)均为中华哲水蚤,同一水团两年间年际差异的贡献种如下:I区为捷氏歪水蚤(56.91%)和真刺唇角水蚤(12.34%);II区为中华哲水蚤(72.64%)、五角水母(13.35%);III区为中华哲水蚤(41.93%)、夜光虫(22.94%)。CCA分析表明,第1 CCA轴(CCA1)和第2 CCA轴(CCA2)共解释了两年春季浮游动物优势种累计方差的46.14%和物种?环境累计方差的97.82%。CCA1主要反映了空间(近海水团和湾内水团)的差异。CCA2主要反映了2018年和2019年站位的年际差异。盐度是影响春季浮游动物群落结构空间差异的主要因素,而温度、叶绿素a浓度是春季浮游动物群落结构年际差异的主要因素。     

发展广东省空间天气轨道的思考

根据中国气象发展战略研究成果中8条业务轨道对空间天气轨道的要求,通过对空间天气和空间天气学的描述,分析广东省气象部门空间天气发展的可行性与必要性,结合广东省气象局业务体制改革,对发展广东省空间天气业务提出一些建议。     

海洋自然资本评估中生物能值转换率的研究

自然资本评估是研究生态系统服务与社会经济发展的重要方式。由于海洋具有特殊性和复杂性、开放性、流动性和多层次耦合性等特点,海洋自然资本评估方法的研究进展缓慢。能值分析理论通过能值转换率将生态系统内流动和储存的各种不同类别的能量和物质转换为同一标准的能值,可衡量和比较不同等级的能量价值,系统完整地反映自然资本价值,这一特点是其他评估方法所缺乏的。为精确量化海洋生物资本价值,本研究引入海洋食物网信息能流图和生态系统能量传递规律,提出了生物能值转换率的经验公式,即Tn=T1·E^1-nL(T1为初级生产者的太阳能值转换率,n为营养级,EL为林德曼效率)。为验证经验公式的可行性,选用相关案例进行了研究,得到海洋食物网中不同食性生物所处营养级及其相应的能值转换率。经验公式有一定误差,具体应用时应通过调研文献和相关数据,确定关键参数的取值范围以提高准确性。本研究得出的经验公式简化了海洋生物能值转换率的计算方法,促进了能值分析理论在海洋自然资本评估中的进一步应用。     

Terrestrial heat flow and lithospheric thermal structure in the Chagan Depression of the Yingen-Ejinaqi Basin,north central China

The Chagan Depression in the Yingen-Ejinaqi Basin, located at the intersection of the Paleo-Asian Ocean and the Tethys Ocean domains is an important region to gain insights on terrestrial heat flow, lithospheric thermal structure and deep geodynamic processes. Here, we compute terrestrial heat flow values in the Chagan Depression using a large set of system steady-state temperature data from four representative wells and rock thermal conductivity. We also estimate the “thermal” lithospheric thickness, mantle heat flow, ratio of mantle heat flow to surface heat flow and Moho temperature to evaluate the regional tectonic framework and deep dynamics. The results show that the heat flow in the Chagan Depression ranges from 66.5 to 69.8 mW/m², with an average value of 68.3 ± 1.2 mW/m². The Chagan Depression is characterized by a thin “thermal” lithosphere, high mantle heat flow, and high Moho temperature, corresponding to the lithospheric thermal structure of “cold mantle and hot crust” type. We correlate the formation of the Yingen-Ejinaqi Basin to the Early Cretaceous and Cenozoic subduction of the western Pacific Plate and the Cenozoic multiple extrusions. Our results provide new insights into the thermal structure and dynamics of the lithospheric evolution in central China.     

长江三角洲全新世沉积物光释光测年研究

长江三角洲是研究古海岸沉积环境演化的理想区域。选取长江三角洲平原一支长岩芯钻孔上部55 m层位的沉积物进行光释光(optically stimulated luminescence,OSL)测年研究。根据沉积物粒度特性,在实验前处理过程中,提取100~200 μm或63~100 μm粗颗粒石英矿物进行OSL测年。测年样品的预热坪与剂量恢复实验表明选择180℃作为预热条件较为合适,石英OSL信号衰退曲线以快组分为主。条件实验结果、等效剂量分布以及各测片的循环比、热转移等方法学层面的实验结果表明,OSL测年技术对该孔的沉积物测年具有适应性和可靠性。通过年代—深度关系模型,建立该段地层中全新世以来的年龄框架。依据该孔的年代地层序列并结合前人工作,探讨全新世以来长江三角洲的沉积演化。     

Population,urbanization and economic scenarios over the Belt and Road region under the Shared Socioeconomic Pathways

The countries throughout the Belt and Road region account for more than 60% of the world's population and half of the global economy. Future changes in this area will have significant influences on the global economic growth, industrial structure and resource allocation. In this study, the proportion of the urban population to the total population and the gross domestic product were used to represent the levels of urbanization and economic development, respectively. The population, urbanization and economic levels of the Belt and Road countries for 2020–2050 were projected under the framework of the IPCC's shared socioeconomic pathways(SSPs), and the following conclusions are drawn.(1) The population, urbanization and economic levels in the Belt and Road region will likely increase under all five pathways. The population will increase by 2%–8%/10 a during 2020–2050 and reach 5.0–6.0 billion in 2050. Meanwhile, the urbanization rate will increase by 1.4%–7.5%/10 a and reach 49%–75%. The GDP will increase by 17%–34%/10 a and reach 134–243 trillion USD.(2) Large differences will appear under different scenarios. The SSP1 and SSP5 pathways demonstrate relatively high urbanization and economic levels, but the population size is comparatively smaller; SSP3 shows the opposite trend. Meanwhile, the economy develops slowly under SSP4, but it has a relatively high urbanization level, while SSP2 exhibits an intermediate trend.(3) In 2050, the population will increase relative to 2016 in most countries, and population size in the fastest growing country in Central Asia and the Middle East countries will be more than double. Urbanization will develop rapidly in South Asia, West Asia and Central Asia, and will increase by more than 150% in the fastest growing countries. The economy will grow fastest in South Asia, Southeast Asia and West Asia, and increase by more than 10 times in some counties with rapid economic development.     

哈达贺休盐湖卤水氢氧同位素特征及成因分析

内蒙古哈达贺休盐湖蕴藏着较为丰富的地下卤水资源,但人们对其成因和演化机制尚缺乏充分的认识。本文采用稳定同位素方法,研究了哈达贺休盐湖地下卤水及其周边水体的氢氧同位素组成特征,并对卤水的成因进行了分析。结果表明：哈达贺休盐湖地下卤水的?D和?18O值平均值分别为-0.53 ‰和4.01 ‰,黑河河水的?D和?18O值平均值分别为-36.73 ‰和-5.51 ‰,居延海湖水的?D和?18O值平均值分别为1.26 ‰和2.73 ‰,当地大气降水的?D和?18O值平均值分别为-5.30 ‰和-1.20 ‰。研究区水体的蒸发趋势线方程为δD=5.32δ18O-20.08,该蒸发线偏离全球大气降水线。黑河河水的氘盈余值（d）最大,湖水和地下卤水的最小,而且湖水和卤水的d值与TDS呈负相关关系。偏正的?D和?18O值以及较小的d值,表明研究区卤水经历了强烈的蒸发,同时还存在与含氧类矿物的同位素交换反应。卤水和居延海湖水氢氧同位素值分布比较集中并且接近,二者都由黑河河水演化而来。