Hadoop在气象数据密集型处理领域中的应用

气象资料的统计分析计算属于数据密集型计算，目前的处理方式多为单机处理，对大量数据的处理比较慢，难以应对日益增长的数据，对气象资料的研究形成一定的制约。针对数据密集型气象数据的处理，尝试应用Hadoop的MapReduce思想提高计算效率；对Hadoop在处理大量小文件组成的气象数据时的低效率，提出对原始文件进行预处理，将多个小文件整合成能直接用于计算的大文件。试验证明，该方法解决了Hadoop处理大量小文件时的低效率问题，通过与Oracle入库检索的比较，应用Hadoop处理数据密集型气象资料具有实际意义。

Application of Hadoop in Data-Intensive Processing of Meteorological Data

The statistical analysis of meteorological data processing is data intensive and always conducts on a single machine. The speed is too slow when the data set is large, which restrains researches of meteorological data. Hadoop and MapReduce are used to speed up the data intensive processing of meteorological data. In allusion to the low efficiency of processing enormous and small files by using Hadoop, a preprocess is conducted to integrate the enormous and small files to a large one. The experiment proved that this method can solve the low efficiency problem when using Hadoop to process enormous and small files. Comparing with Oracle, it is more useful to use Hadoop to process data intensive Meteorological computing.