Storm流式技术在地面气象数据处理中的应用

地面自动站气象数据的量级、观测传输频次日益增长,传输方式从传统的文本方式发展到消息传输,对数据处理时效有了更高的要求。该文提出采用Storm流式技术进行地面自动站气象数据的解码和入库,使用Spout节点连接外部数据源,将数据转化为Tuple,传递给解码Bolt、入库Bolt和DI/EI消息Bolt,分别进行实时的入库和监控,且达到灵活调整系统资源的目的。该应用与传统的简约流程对比,稳定性、处理效率、拓展性都有了显著提高。

Application of Storm Flow Technology in Surface Meteorological Data Processing

The quantity and observational frequency of meteorological data by surficial automatic station are increasing. The transmission mode has developed from the traditional text mode to the message transmission mode. These changes make higher requirement for data processing timeliness. In this paper, we used Storm Flow Technology on decoding and warehouse of surficial automatic station data. It used Spout to connect external data sources, and convert data into Tuple, and then transmit to Bolt-decode, Bolt-warehouse and Bolt-DIEI. They can be used to real-time warehouse, monitoring, and quickly adjust system resource. Compared with the traditional simplified process, the stability, processing efficiency and expansibility of the application have been greatly improved.