The data zoo in Astro-WISE

In this paper we describe the way the Astro-WISE information system (or simply Astro-WISE) supports the data from a wide range of instruments and combines multiple surveys and their catalogues. Astro-WISE allows ingesting of data from any optical instrument, survey or catalogue, processing of this data to create new catalogues and bringing in data from different surveys into a single catalogue, keeping all dependencies back to the original data. Full data lineage is kept on each step of compiling a new catalogue with an ability to add a new data source recursively. With these features, Astro-WISE allows not only combining and retrieving data from multiple surveys, but performing scientific data reduction and data mining down to the rawest data in the data processing chain within a single environment.     

Astro-WISE and Grid

The paper reviews the Astro-WISE infrastructure and demonstrates that the Astro-WISE Information System provides a Grid itself. We describe the integration of Astro-WISE with an external Grid infrastructure (BiGGrid). The integration is performed on all infrastructural layers (data storage, metadata and processing layers) with Astro-WISE as a “master” infrastructure. We report the use of the integrated infrastructure for the processing of Astro-WISE hosted data and for the future development of Astro-WISE and Target projects.     

The Astro-WISE datacentric information system

In this paper we present the various concepts behind the Astro-WISE Information System. The concepts form a blueprint for general scientific information systems (WISE) which can satisfy a wide and challenging range of requirements for the data dissemination, storage and processing for various fields in science. We review the main features of the information system and its practical implementation.     

Scaling Astro-WISE to LOFAR long term archive

The Astro-WISE information system was developed to handle data processing for the KIDS survey. In this paper we describe the adaptation of the WISE concept to allow scaling to support archives containing tens of petabytes of stored data and the changes we introduced to accommodate the system for the LOFAR Long Term Archive. With this we provide an example of how Astro-WISE technology can be adapted to support a wider range and scale of data.     

Astro-WISE interfaces

From a simple text interface to a graphical user interfaces—Astro-WISE provides the user with a wide range of possibilities to interact with the information system according to the user’s tasks and use cases. We describe a general approach to the interfacing of a scientific information system. We use this approach to create a number of services, which allows the user to browse the data stored in the system, to process the data and to exchange the newly created images and catalogs with the users within the system and wider astronomical community. Reusability of interfaces and services is another important feature of our approach. It reduces the time and resources spent to interface other information systems created from Astro-WISE.     

MDia and POTS

We describe the Munich Difference Imaging Analysis pipeline that we developed and implemented in the framework of the Astro-WISE¹ package to automatically measure high precision light curves of a large number of stellar objects using the difference imaging approach. Combined with programs to detect time variability, this software can be used to search for planetary systems or binary stars with the transit method and for variable stars of different kinds. As a first scientific application, we discuss the data reduction and analysis performed with Astro-WISE on the pre-OmegaTranS data set, that we collected during a monitoring campaign of a dense stellar field with the Wide Field Imager at the ESO 2.2 m telescope.     

Monitoring the photometric behavior of OmegaCAM with Astro-WISE

The OmegaCAM wide-field optical imager is the sole instrument on the VLT Survey Telescope at ESO’s Paranal Observatory. The instrument, as well as the telescope, have been designed for surveys with very good, natural seeing-limited image quality over a 1 square degree field. OmegaCAM was commissioned in 2011 and has been observing three ESO Public Surveys in parallel since October 15, 2011. We use the Astro-WISE information system to monitor the calibration of the observatory and to produce the Kilo Degree Survey (KiDS). Here we describe the photometric monitoring procedures in Astro-WISE and give a first impression of OmegaCAM’s photometric behavior as a function of time. The long-term monitoring of the observatory goes hand in hand with the KiDS survey production in Astro-WISE. KiDS is observed under partially non-photometric conditions. Based on the first year of OmegaCAM operations it is expected that a ～ 1–2 % photometric homogeneity will be achieved for KiDS.     

基于SQL/NoSQL的空间目标光学特性数据混合存储策略

为解决海量空间目标光学特性数据入库与检索效率低、非结构化数据存储与管理困难、数据服务扩展性差和无法满足多用户获取实时请求数据的问题,提出一种空间目标光学特性数据混合存储策略.首先,利用光学特性数据中光源与探测角度的规律性,对数据进行处理并构建检索策略;其次,结合空间目标光学特性数据应用领域在数据存储、处理和访问方面的需求特点,构建空间目标光学特性数据混合存储策略,并设计了数据检索体系结构;最后,选取空间目标模拟光学特性数据入库和检索两个场景进行验证.实验结果表明,存储数值数据情况下,混合存储策略数据入库效率较传统关系型数据库提高超过17倍;存储数值数据和图像文件情况下,提高超过34倍;在多用户请求下混合存储策略数据检索效率较传统存储策略有较大提升.提出的混合存储策略能够有效地满足空间目标光学特性相关的仿真、测试与实验对数据的实时请求需求.     

On Dimensionality Reduction for Indexing and Retrieval of Large-Scale Solar Image Data

This work investigates the applicability of several dimensionality reduction techniques for large-scale solar data analysis. Using a solar benchmark dataset that contains images of multiple types of phenomena, we investigate linear and nonlinear dimensionality reduction methods in order to reduce our storage and processing costs and maintain a good representation of our data in a new vector space. We present a comparative analysis of several dimensionality reduction methods and different numbers of target dimensions by utilizing different classifiers in order to determine the degree of data dimensionality reduction that can be achieved with these methods, and to discover the method that is the most effective for solar images. After determining the optimal number of dimensions, we then present preliminary results on indexing and retrieval of the dimensionally reduced data.     

Information systems playground—the target infrastructure

The Target infrastructure has been specially built as a storage and compute infrastructure for the information systems derived from Astro-WISE. This infrastructure will be used by several applications that collaborate in the area of information systems within the Target project. It currently consists of 10 PB of storage and thousands of computational cores. The infrastructure has been constructed based on the requirements of the applications. The storage is controlled by the Global Parallel File System of IBM. This file system takes care of the required flexibility by combining storage hardware with different characteristics into a single file system. It is also very scalable, which allows the system to be extended into the future, while replacing old hardware with new technology.     

The Astro-WISE approach to quality control for astronomical data

We present a novel approach to quality control during the processing of astronomical data. Quality control in the Astro-WISE Information System is integral to all aspects of data handing and provides transparent access to quality estimators for all stages of data reduction from the raw image to the final catalog. The implementation of quality control mechanisms relies on the core features in this Astro-WISE Environment (AWE): an object-oriented framework, full data lineage, and both forward and backward chaining. Quality control information can be accessed via the command-line awe-prompt and the web-based Quality-WISE service. The quality control system is described and qualified using archive data from the 8-CCD Wide Field Imager (WFI) instrument (http://www.eso.org/lasilla/instruments/wfi/) on the 2.2-m MPG/ESO telescope at La Silla and (pre-)survey data from the 32-CCD OmegaCAM instrument (http://www.astro-wise.org/~omegacam/) on the VST telescope at Paranal.     

INES: Astronomy Data Distribution for the Future

We describe here the structure under which the IUE Project will leave its archive at the completion of the distribution system and final data processing, within the INES system.The INES system is a total system, which comprises both the data in their final bulk processing mode for direct application to scientific analysis, as well as the software driving the distributed service for data retrieval directly by the end user.As a consequence of the expected long-term usage and support needs, it has been designed to require minimum maintenance costs and will not suffer single point failures because of the distributed nature. Integration of the INES system in a more general multi-wavelength archive for specialized analysis is anticipated tobe relatively easy, so further evolution of the data availabilitywill be driven fully by the users community and will evolve as thedata needs of the community develop. The long term responsibilitywill be transferred from ESA to the astronomical community throughthe establishment of the INES Principal Centre at the LAEFF Instituteof INTA in Spain.     

基于虚拟天文台的HXMT卫星数据检索发布系统设计与实现

目前,国际上诸多天文项目均遵循虚拟天文台（Virtual Observatory,VO）标准协议开发各天文数据检索发布系统,对外公开发布数据,并对数据资源进行VO注册,从而使用户通过虚拟天文台门户网站即可访问获取不同天文项目的数据集.硬X射线调制望远镜（HXMT）卫星项目也将虚拟天文台技术引入HXMT卫星数据检索发布系统的设计与实现过程中,既满足HXMT卫星数据发布需求,又将HXMT卫星数据融入虚拟天文台环境,实现国际天文数据的共享共用.系统提出了符合虚拟天文台规范的体系架构,并选取SCS锥形检索、VOTable数据格式等虚拟天文台标准协议加以实现,采用MVC模式、SSH框架以及各种J2EE技术进行软件研发,提供检索访问、浏览下载和可视化功能.实践和应用结果表明,系统在解决天文数据资源互操作、共享发布、检索访问及异构应用集成方面均具有可操作性,对我国空间天文卫星数据检索发布系统的研制具有参考意义.     

Sub-image data processing in Astro-WISE

Most often, astronomers are interested in a source (e.g., moving, variable, or extreme in some colour index) that lies on a few pixels of an image. However, the classical approach in astronomical data processing is the processing of the entire image or set of images even when the sole source of interest may exist on only a few pixels of one or a few images. This is because pipelines have been written and designed for instruments with fixed detector properties (e.g., image size, calibration frames, overscan regions, etc.). Furthermore, all metadata and processing parameters are based on an instrument or a detector. Accordingly, out of many thousands of images for a survey, this can lead to unnecessary processing of data that is both time-consuming and wasteful. We describe the architecture and an implementation of sub-image processing in Astro-WISE. The architecture enables a user to select, retrieve and process only the relevant pixels in an image where the source exists. We show that lineage data collected during the processing and analysis of datasets can be reused to perform selective reprocessing (at sub-image level) on datasets while the remainder of the dataset is untouched, a difficult process to automate without lineage.     

AstroGrid-D: Grid technology for astronomical science

We present status and results of AstroGrid-D, a joint effort of astrophysicists and computer scientists to employ grid technology for scientific applications. AstroGrid-D provides access to a network of distributed machines with a set of commands as well as software interfaces. It allows simple use of computer and storage facilities and to schedule or monitor compute tasks and data management. It is based on the Globus Toolkit middleware (GT4).Chapter 1 describes the context which led to the demand for advanced software solutions in Astrophysics, and we state the goals of the project.We then present characteristic astrophysical applications that have been implemented on AstroGrid-D in chapter 2. We describe simulations of different complexity, compute-intensive calculations running on multiple sites (Section 2.1), and advanced applications for specific scientific purposes (Section 2.2), such as a connection to robotic telescopes (Section 2.2.3). We can show from these examples how grid execution improves e.g. the scientific workflow.Chapter 3 explains the software tools and services that we adapted or newly developed. Section 3.1 is focused on the administrative aspects of the infrastructure, to manage users and monitor activity. Section 3.2 characterises the central components of our architecture: The AstroGrid-D information service to collect and store metadata, a file management system, the data management system, and a job manager for automatic submission of compute tasks.We summarise the successfully established infrastructure in chapter 4, concluding with our future plans to establish AstroGrid-D as a platform of modern e-Astronomy.     

基于Mark5B＋GPU脉冲星观测系统

云南天文台40m射电望远镜进行的脉冲星观测数据量巨大,必须实现数据的实时处理,否则将会产生海量的数据积压.为实现这一目标,采用图形处理器架构,对Mark5B数据进行解码、消色散、折叠等处理.实验结果表明,对以1s8MB的实时采样,可以在0.51s内处理完成,从而实现了实时处理的要求.首先介绍这一观测系统各部分的图形处理器实现,然后相对于传统中央处理器构架,对各部分的运算速度进行了详细的对比.针对时间开销最大的消色散部分,分析了单次傅里叶变换的数据量大小对执行效率的影响.从系统最终的输出轮廓和柱状图上可以看到实时处理的结果符合要求.最后对存在的问题和未来的工作进行了讨论.     

Integrating VOEvent into the ground‐based O/IR system: A case study at NOAO

In this article we describe a case study of how NOAO is considering improving its management of Target‐of‐Opportunity (ToO) observations by integrating VOEvent into the flow of activities. We believe that using VOEvent to help document and track the use of ToO time will improve the user experience of ToOs at NOAO. It will also greatly aid in the management of the process and of the resulting data, allowing us to better track the ownership and provenance of the data and any resulting data products. Finally, it will provide an important method of archival access to the data and data “collections,” which might include not only processed data from a single VOEvent triggered observation but could also include multiple observations traceable to a single (or set of related) VOEvents. (© 2008 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)     

The Science Operations Centre for Spectrum-UV

The Science Operations Centre (SOC) for the Spectrum-UV project is expected to provide a number of services for the scientific community: support to observers, long-term correlation and trend analysis, analysis of the calibration observations, standard data processing, archiving, data retrieval and distribution. This paper provides an overall conceptual and architectural design of the Spectrum-UV SOC.