A hierarchical indexing strategy for optimizing Apache Spark with HDFS to efficiently query big geospatial raster data

ABSTRACT

Earth observations and model simulations are generating big multidimensional array-based raster data. However, it is difficult to efficiently query these big raster data due to the inconsistency among the geospatial raster data model, distributed physical data storage model, and the data pipeline in distributed computing frameworks. To efficiently process big geospatial data, this paper proposes a three-layer hierarchical indexing strategy to optimize Apache Spark with Hadoop Distributed File System (HDFS) from the following aspects: (1) improve I/O efficiency by adopting the chunking data structure; (2) keep the workload balance and high data locality by building the global index (k-d tree); (3) enable Spark and HDFS to natively support geospatial raster data formats (e.g., HDF4, NetCDF4, GeoTiff) by building the local index (hash table); (4) index the in-memory data to further improve geospatial data queries; (5) develop a data repartition strategy to tune the query parallelism while keeping high data locality. The above strategies are implemented by developing the customized RDDs, and evaluated by comparing the performance with that of Spark SQL and SciSpark. The proposed indexing strategy can be applied to other distributed frameworks or cloud-based computing systems to natively support big geospatial data query with high efficiency.     

SMAP卫星海表盐度产品的检验与校正

In this study, sea surface salinity(SSS) Level 3(L3) daily product derived from soil moisture active passive(SMAP)during the year 2016, was validated and compared with SSS daily products derived from soil Moisture and ocean salinity(SMOS) and in-situ measurements. Generally, the root mean square error(RMSE) of the daily SSS products is larger along the coastal areas and at high latitudes and is smaller in the tropical regions and open oceans. Comparisons between the two types of daily satellite SSS product revealed that the RMSE was higher in the daily SMOS product than in the SMAP, whereas the bias of the daily SMOS was observed to be less than that of the SMAP when compared with Argo floats data. In addition, the latitude-dependent bias and RMSE of the SMAP SSS were found to be primarily influenced by the precipitation and the sea surface temperature(SST). Then, a regression analysis method which has adopted the precipitation and SST data was used to correct the larger bias of the daily SMAP product. It was confirmed that the corrected daily SMAP product could be used for assimilation in high-resolution forecast models, due to the fact that it was demonstrated to be unbiased and much closer to the in-situ measurements than the original uncorrected SMAP product.     

全水深多波束测深系统Seabeam3012在西太平洋马里亚纳海山区地形测量中的应用

中国科学院海洋研究所在开展西太平洋马里亚纳海山区多学科综合科学考察的过程中,利用“科学”轮船载的全水深多波束测深系统Seabeam3012对多个海山进行了地形测量工作。针对作业过程中遇到的恶劣海况导致采集数据质量差、多波束系统易检测错误海底信息、测线布设难度大等问题,提出了基于船体姿态对数据质量影响分析的多波束测线方向优化、基于地形变化并参考浅地层剖面资料的作业参数优化和基于实时采集情况的多波束采集测线布设优化等一系列措施,有效地提高了海山区多波束数据采集质量,并提高了作业效率。获得的高品质地形数据,为多学科协同研究奠定了基础,为ROV等设备的现场作业提供了安全保障。     

南海开放共享航次关键科学问题的思考——从多尺度海洋动力学角度出发

南海是西太平洋最大的边缘海, 通过一系列的海峡与西太平洋和印度洋相联通, 其不同时空尺度的海洋环流动力过程及其生态环境效应是南海区域海洋学研究的重要内容。自20世纪50年代末全国第一次海洋普查开始, 我国对海洋调查的支持力度不断加大, 以科学考察船为代表的海洋科学观测平台建设不断加强; 进入新千年以来, 国内海洋科考船依托的各主要研究所和院校本着开放的理念, 先后组织多单位联合进行海上观测。尤其是最近10年, 国家自然科学基金委员会支持实施了船时共享航次计划, 进一步促进了国内海洋界的交流和合作, 南海区域海洋学的相关研究取得了很多重要的成果。从多尺度环流动力学的角度出发, 本文简要回顾了南海海洋观测的发展历程, 并初步总结了近些年来南海关键科学问题的研究进展, 包括南海和西太平洋的水体交换过程、南海中小尺度过程、多尺度相互作用及其生态环境效应等; 并且在现有的研究基础上, 对未来南海的观测和科学问题提出若干思考与展望。     

东印度洋海域最优深度基准面模型构建

利用东印度洋海域周边长期验潮站实测数据、TOPEX/Poseidon等系列卫星测高反演结果,评估了DTU10,EOT11a,FES2014,GOT4.8,OSU12和TPXO8六种全球潮汐模型精度,根据卫星测高结果给出了浅水分潮改正量和长周期分潮改正量的经验模型,又在此基础上分析并构建了研究区域精度最优的深度基准面模型。考虑到全球潮汐模型在近岸的影响因素及验潮站位置,将13个验潮站分成开阔海域与近海海域两类,与潮汐模型的对比,结果表明,DTU10和FES2014模型分别在开阔海域和近海海域精度最优。根据潮汐模型在不同分潮处的精度,如EOT11a模型在O1和K1分潮处精度较高,DTU10在N2,M2,S2和K2分潮处精度较高等,分别构建了开阔海域与近海海域的组合深度基准面模型,计算得知误差分别为11.33和20.95 cm,其精度显著提高。     

面向遥感大数据的地学知识图谱构想

由于地球表面的时空异质性与复杂性,传统从遥感影像具有的信息特征出发,构建智能解译算法解决遥感地学认知的思路在应对面向全球的海量遥感大数据分析时,其精度和地学实用性已触及瓶颈.为此,本文从地学知识为核心的角度出发,结合当前知识图谱理论的发展,提出一种新的面向遥感大数据分析的地学思维构想——地学知识图谱.本构想将地学知识的...     

基于Stacking集成学习的恒星/星系分类研究

机器学习在当今诸多领域已经取得了巨大的成功,但是机器学习的预测效果往往依赖于具体问题.集成学习通过综合多个基分类器来预测结果,因此,其适应各种场景的能力较强,分类准确率较高.基于斯隆数字巡天(Sloan Digital Sky Survey,SDSS)计划恒星/星系中最暗源星等集分类正确率低的问题,提出一种基于Stacking集成学习的恒星/星系分类算法.从SDSS-DR7(SDSS Data Release 7)中获取完整的测光数据集,并根据星等值划分为亮源星等集、暗源星等集和最暗源星等集.仅针对分类较为复杂且困难的最暗源星等集展开分类研究.首先,对最暗源星等集使用10折嵌套交叉验证,然后使用支持向量机(Support Vector Machine,SVM)、随机森林(Random Forest,RF)、XGBoost(eXtreme Gradient Boosting)等算法建立基分类器模型;使用梯度提升树(Gradient Boosting Decision Tree,GBDT)作为元分类器模型.最后,使用基于星系的分类正确率等指标,与功能树(Function Tree,FT)、SVM、RF、GBDT、XGBoost、堆叠降噪自编码(Stacked Denoising AutoEncoders,SDAE)、深度置信网络(Deep Belief Network,DBN)、深度感知决策树(Deep Perception Decision Tree,DPDT)等模型进行分类结果对比分析.实验结果表明,Stacking集成学习模型在最暗源星等集分类中要比FT算法的星系分类正确率提高了将近10%.同其他传统的机器学习算法、较强的提升算法、深度学习算法相比,Stacking集成学习模型也有较大的提升.     

An Automatic Method for Detecting Transients and Variable Sources in AST3 Survey Based on Image Subtraction and Random Forest

AST3-2 (the second Antarctic Survey Telescope) is located in Antarctic Dome A, the loftiest ice dome on the Antarctic Plateau. It produces a huge amount of observational data which require a more efficient data reduction program to be developed. Also the data transmission in Antarctica is much difficult, thus it is necessary to perform data reduction and detect variable and transient sources remotely and automatically in Antarctica, but this attempt is restricted by the unsatisfactory performance of the low power consumption computer in Antarctica. For realizing this purpose, to develop a new method based on the existing image subtraction method and random forest algorithm, taking the AST3-2 2016 dataset as the test sample, becomes an alternative choice. This method performs image subtraction on the dataset, then applies the principle component analysis to extract the features of residual images. Random forest is used as a machine learning classifier, and in the test a recall rate of 97% is resulted for the positive sample. Our work has verified the feasibility and accuracy of this method, and finally found out a batch of candidates for variable stars in the AST3-2 2016 dataset.     

Formation of Carbon-Enhanced Metal-Poor RR Lyrae Stars

Carbon-enhanced metal-poor (CEMP) stars are considered to be related to the first generation of stars, and responsible for the chemical evolution of the early Galaxy. More than half of them are in binaries, and could be explained by the binary evolution, but the formation channel of them is still not fully understood. Among the hundreds of CEMP stars, there are nine CEMP RR Lyrae stars identified, and at least seven of which are very likely not binaries. The usual binary star evolution channel is difficult to produce such a single star, particularly that of carbon enrichment. One way in which such a single star might be produced is the merger of a helium white dwarf with a Hertzsprung gap (HG) star. We use a stellar evolution program to calculate the models of the merger remnants, and find that the models can reproduce the observed distribution of these CEMP single RR Lyrae stars in terms of surface temperature, gravity, and carbon abundance. Hence, it is extremely possible that the helium white dwarf and HG star merger model is one of the formation channels of the metal-poor carbon-rich RR Lyrae stars.     

遗传算法优化的BP神经网络卫星钟差预报

针对BP (Back Propagation)神经网络模型预测卫星钟差中权值和阈值的最优化问题, 提出了基于遗传算法优化的BP神经网络卫星钟差短期预报模型, 给出了遗传算法优化BP神经网络的基本思想、具体方法和实施步骤. 为验证该优化模型的有效性和可行性, 利用北斗卫星导航系统(BeiDou navigation satellite system, BDS)卫星钟差数据进行钟差预报精度分析, 并将其与灰色模型(GM(1,1))和BP神经网络模型预报的结果比较分析. 结果表明: 该模型在短期钟差预报中具有较好的精度, 优于GM(1,1)模型和BP神经网络模型.