基于流程控制的国际气象通信数据收集管理系统的设计与实现

近年来,天气和气候业务对国外数据种类和时效要求不断提高,为解决大量数据的收集分发压力以及管理问题,设计并实现了基于流程控制的国际气象通信数据收集管理系统。系统应用数据收集器Logstash、Inotify监听机制与分析平台Kibana等技术,支持多种通信协议,扩展Scrapy网络爬虫功能,通过流程控制、任务调度等方式实现大量数据快速高效收集和集成管理,有效提高数据并行处理能力和可扩展性,且设计了全流程监视,保障数据时效。系统已经上线提供服务,为业务/科研用户提供全球观测数据、国外卫星数据和数值预报产品等300余种,日收集数据量约2 TB。

Design and Implementation of International Meteorological Communication Data Collection and Management System

In recent years, meteorological and climate services have increased their requirements for foreign data types and timeliness. An international meteorological communication data collection and management system based on process control is designed and implemented to solve the collection and distribution pressure and management problems of a large amount of data. The system applies the technology of Logstash, Inotify and Kibana, supports a variety of communication protocols, expands the network crawler function of Scrapy, and achieves the fast and efficient collection and integration management of a large amount of data through process control and task scheduling, effectively improving the data parallel processing ability and scalability. And the whole process of monitoring is designed to ensure data timeliness. The system has been online to provide services, providing operational/scientific research users with more than 300 kinds of global observation data, foreign satellite data and numerical prediction products; the daily data collection is about 2 TB.