| 海量GNSS小文件云存储优化方法研究 |
| |
| 引用本文： | 李林阳,吕志平,崔阳,王宇谱,周海涛.海量GNSS小文件云存储优化方法研究[J].武汉大学学报(信息科学版),2017,42(8):1068-1074. |
| |
| 作者姓名： | 李林阳 吕志平 崔阳 王宇谱 周海涛 |
| |
| 作者单位： | 1.地理信息工程国家重点实验室, 陕西 西安, 710054 |
| |
| 基金项目： | 国家自然科学基金41674019国家重点研发计划2016YFB0501701地理信息工程国家重点实验室开放基金SKLGIE2016-M-1-2 |
| |
| 摘 要： | GNSS数据量呈指数级趋势增长，Hadoop分布式文件系统（HDFS）解决了海量GNSS数据存储瓶颈的难题，却面临内存占用多、文件相关性差和缺乏优化机制的问题。针对HDFS处理海量GNSS小文件效率不高的问题，结合GNSS数据类型、特点以及存储过程，提出了一种新的GNSS小文件云存储方法，优化了GNSS小文件的写入、读取、添加和删除策略。该方法分别按观测文件和解算成果的类型进行合并，对合并后的文件构建压缩Trie树索引，索引切分后，根据匹配算法分布式地存储索引块。实验采用国际GNSS服务（IGS）28 d的数据和产品进行云存储优化。结果表明，该方法降低了各节点内存消耗，提高了海量GNSS小文件写入、读取和删除的效率，实现了对海量GNSS小文件的高效云存储。
|
| 关 键 词： | 全球卫星导航系统 小文件 Hadoop分布式文件系统 云存储 压缩Trie树索引 |
| 收稿时间： | 2015-09-14 |
The Optimized Cloud Storage Method of Massive GNSS Small Files |
| |
| Affiliation: | 1.State Key Laboratory of Geo-Information Engineering, Xi'an 710054, China2.School of Geographical Spatial Information, Information Engineering University, Zhengzhou 450052, China |
| |
| Abstract: | The data volume of GNSS is increasing exponentially, while HDFS is capable of handling the problem of the storage bottleneck of massive GNSS data, it is faced with much time consumption, poor file correlation and lack of optimization mechanisms. According to the matter of the low processing efficiency of massive GNSS small files faced by HDFS, a new cloud storage method is provided based on the types, characteristics and storage flow of GNSS data, the writing, adding, reading and deleting strategies are optimized. First the observation files and solution files are respectively merged, and the compressed Trie index is established on the merged files; and after splitting the existed index, the index blocks are distributed stored in each mode based on the matching algorithm. Data and products of 28 days from IGS are applied in the experiment, and the result shows that the memory consumption of each node can be decreased greatly, and the efficiency of writing, direct reading, reading after adding files, concurrent reading and deleting can be improved significantly, effective cloud storage of massive GNSS small files is hereafter realized. |
| |
| Keywords: | |
| 本文献已被 CNKI 等数据库收录！ |
| 点击此处可从《武汉大学学报(信息科学版)》浏览原始摘要信息 |
|
点击此处可从《武汉大学学报(信息科学版)》下载全文 |
|
