Astro-WISE and Grid

The paper reviews the Astro-WISE infrastructure and demonstrates that the Astro-WISE Information System provides a Grid itself. We describe the integration of Astro-WISE with an external Grid infrastructure (BiGGrid). The integration is performed on all infrastructural layers (data storage, metadata and processing layers) with Astro-WISE as a “master” infrastructure. We report the use of the integrated infrastructure for the processing of Astro-WISE hosted data and for the future development of Astro-WISE and Target projects.     

Tracing and using data lineage for pipeline processing in Astro-WISE

Most workflow systems that support data provenance primarily focus on tracing lineage of data. Data provenance by data lineage provides the derivation history of data including information about services and input data that contributed to the creation of a data product. We show that tracing lineage by means of full backward chaining not only enables users to share, discover and reuse the data, but also supports scientific data processing through storage, retrieval and (re)processing of digitized scientific data. In this paper, we present Astro-WISE, a distributed system for processing, analyzing and disseminating wide field imaging astronomical data. We show how Astro-WISE traces lineage of data and how it facilitates data processing, retrieval, storage and archiving. Particularly we show how it solves issues related to the changing data items typical for the scientific environment, such as physical changes in calibrations, our insight in these changes and improved methods for deriving results.     

Monitoring the photometric behavior of OmegaCAM with Astro-WISE

The OmegaCAM wide-field optical imager is the sole instrument on the VLT Survey Telescope at ESO’s Paranal Observatory. The instrument, as well as the telescope, have been designed for surveys with very good, natural seeing-limited image quality over a 1 square degree field. OmegaCAM was commissioned in 2011 and has been observing three ESO Public Surveys in parallel since October 15, 2011. We use the Astro-WISE information system to monitor the calibration of the observatory and to produce the Kilo Degree Survey (KiDS). Here we describe the photometric monitoring procedures in Astro-WISE and give a first impression of OmegaCAM’s photometric behavior as a function of time. The long-term monitoring of the observatory goes hand in hand with the KiDS survey production in Astro-WISE. KiDS is observed under partially non-photometric conditions. Based on the first year of OmegaCAM operations it is expected that a ～ 1–2 % photometric homogeneity will be achieved for KiDS.     

The data zoo in Astro-WISE

In this paper we describe the way the Astro-WISE information system (or simply Astro-WISE) supports the data from a wide range of instruments and combines multiple surveys and their catalogues. Astro-WISE allows ingesting of data from any optical instrument, survey or catalogue, processing of this data to create new catalogues and bringing in data from different surveys into a single catalogue, keeping all dependencies back to the original data. Full data lineage is kept on each step of compiling a new catalogue with an ability to add a new data source recursively. With these features, Astro-WISE allows not only combining and retrieving data from multiple surveys, but performing scientific data reduction and data mining down to the rawest data in the data processing chain within a single environment.     

Scaling Astro-WISE to LOFAR long term archive

The Astro-WISE information system was developed to handle data processing for the KIDS survey. In this paper we describe the adaptation of the WISE concept to allow scaling to support archives containing tens of petabytes of stored data and the changes we introduced to accommodate the system for the LOFAR Long Term Archive. With this we provide an example of how Astro-WISE technology can be adapted to support a wider range and scale of data.     

The Astro-WISE datacentric information system

In this paper we present the various concepts behind the Astro-WISE Information System. The concepts form a blueprint for general scientific information systems (WISE) which can satisfy a wide and challenging range of requirements for the data dissemination, storage and processing for various fields in science. We review the main features of the information system and its practical implementation.     

The Virtual Solar Observatory—A Resource for International Heliophysics Research

The Virtual Solar Observatory (VSO) has been developed to allow researchers, educators, and the general public to access data and images from the major sources of on-line solar data. The VSO substantially reduces the effort required to locate disparate data sets, and removes the need for the user to locate the data and learn multiple interfaces. The VSO provides a single interface to about 60 geographically distributed data sets including space- and ground-based sources. These data sets incorporate several physical variables including magnetic field, intensity, Doppler velocity, etc., and all wavelengths from X-ray to radio. All layers of the sun, from the interior to the corona, are included. In this paper we describe the system and present the interface that the user will encounter. We also discuss future enhancements planned for the system.     

Development of Control Softwares for the SRAO 6-Meter Telescope Based on PCs Running Linux

We have developed a control system for the Seoul Radio Astronomy Observatory (SRAO) 6-meter telescope operating in the 85–115 Ghz frequency range. Four PCs running the Linux operating system control source tracking, take data, execute observations and handle user interface. The control system utilizes a modular and multiprocessing structure to facilitate easy upgrading and troubleshooting. Communication between the processes relies on the interprocess communication (IPC) resources on Linux such as shared memory, message queues, and TCP/IP sockets. Communication between PCs is made via an Ethernet link. We also use digital I/O lines for some status signals which require a short delay. The control system supports scheduling observations, updates observation logs automatically and also supports graphical user interfaces. These all makes the operation easy. By using a commercially available motion control card with an embedded microcomputer for antenna control, we achieved a tracking accuracy to better than 1 arcsec. This revised version was published online in July 2006 with corrections to the Cover Date.     

MDia and POTS

We describe the Munich Difference Imaging Analysis pipeline that we developed and implemented in the framework of the Astro-WISE¹ package to automatically measure high precision light curves of a large number of stellar objects using the difference imaging approach. Combined with programs to detect time variability, this software can be used to search for planetary systems or binary stars with the transit method and for variable stars of different kinds. As a first scientific application, we discuss the data reduction and analysis performed with Astro-WISE on the pre-OmegaTranS data set, that we collected during a monitoring campaign of a dense stellar field with the Wide Field Imager at the ESO 2.2 m telescope.     

基于虚拟天文台的HXMT卫星数据检索发布系统设计与实现

目前,国际上诸多天文项目均遵循虚拟天文台（Virtual Observatory,VO）标准协议开发各天文数据检索发布系统,对外公开发布数据,并对数据资源进行VO注册,从而使用户通过虚拟天文台门户网站即可访问获取不同天文项目的数据集.硬X射线调制望远镜（HXMT）卫星项目也将虚拟天文台技术引入HXMT卫星数据检索发布系统的设计与实现过程中,既满足HXMT卫星数据发布需求,又将HXMT卫星数据融入虚拟天文台环境,实现国际天文数据的共享共用.系统提出了符合虚拟天文台规范的体系架构,并选取SCS锥形检索、VOTable数据格式等虚拟天文台标准协议加以实现,采用MVC模式、SSH框架以及各种J2EE技术进行软件研发,提供检索访问、浏览下载和可视化功能.实践和应用结果表明,系统在解决天文数据资源互操作、共享发布、检索访问及异构应用集成方面均具有可操作性,对我国空间天文卫星数据检索发布系统的研制具有参考意义.     

Apex I and Apex II software packages for the reduction of astronomical CCD observations

We describe in detail the Apex I and Apex II software packages created for astrometric and photometric reductions of astronomical observations with CCD cameras. The Apex I software package has been created for the semiautomatic reduction of astronomical observations and has a convenient user interface. The Apex II software package allows for the completely automatic reduction of astrometric and photometric observations to be performed.     

Scalable desktop visualisation of very large radio astronomy data cubes

Observation data from radio telescopes is typically stored in three (or higher) dimensional data cubes, the resolution, coverage and size of which continues to grow as ever larger radio telescopes come online. The Square Kilometre Array, tabled to be the largest radio telescope in the world, will generate multi-terabyte data cubes – several orders of magnitude larger than the current norm. Despite this imminent data deluge, scalable approaches to file access in Astronomical visualisation software are rare: most current software packages cannot read astronomical data cubes that do not fit into computer system memory, or else provide access only at a serious performance cost. In addition, there is little support for interactive exploration of 3D data.We describe a scalable, hierarchical approach to 3D visualisation of very large spectral data cubes to enable rapid visualisation of large data files on standard desktop hardware. Our hierarchical approach, embodied in the AstroVis prototype, aims to provide a means of viewing large datasets that do not fit into system memory. The focus is on rapid initial response: our system initially rapidly presents a reduced, coarse-grained 3D view of the data cube selected, which is gradually refined. The user may select sub-regions of the cube to be explored in more detail, or extracted for use in applications that do not support large files. We thus shift the focus from data analysis informed by narrow slices of detailed information, to analysis informed by overview information, with details on demand. Our hierarchical solution to the rendering of large data cubes reduces the overall time to complete file reading, provides user feedback during file processing and is memory efficient. This solution does not require high performance computing hardware and can be implemented on any platform supporting the OpenGL rendering library.     

Information systems playground—the target infrastructure

The Target infrastructure has been specially built as a storage and compute infrastructure for the information systems derived from Astro-WISE. This infrastructure will be used by several applications that collaborate in the area of information systems within the Target project. It currently consists of 10 PB of storage and thousands of computational cores. The infrastructure has been constructed based on the requirements of the applications. The storage is controlled by the Global Parallel File System of IBM. This file system takes care of the required flexibility by combining storage hardware with different characteristics into a single file system. It is also very scalable, which allows the system to be extended into the future, while replacing old hardware with new technology.     

The Astro-WISE approach to quality control for astronomical data

We present a novel approach to quality control during the processing of astronomical data. Quality control in the Astro-WISE Information System is integral to all aspects of data handing and provides transparent access to quality estimators for all stages of data reduction from the raw image to the final catalog. The implementation of quality control mechanisms relies on the core features in this Astro-WISE Environment (AWE): an object-oriented framework, full data lineage, and both forward and backward chaining. Quality control information can be accessed via the command-line awe-prompt and the web-based Quality-WISE service. The quality control system is described and qualified using archive data from the 8-CCD Wide Field Imager (WFI) instrument (http://www.eso.org/lasilla/instruments/wfi/) on the 2.2-m MPG/ESO telescope at La Silla and (pre-)survey data from the 32-CCD OmegaCAM instrument (http://www.astro-wise.org/~omegacam/) on the VST telescope at Paranal.     

RATAN-600 radio telescope in the 24th solar activity cycle. IV. Information system for RATAN-600 solar observations

The development of observational equipment and software for processing and efficient representation of spectral and polarization solar microwave observations on the RATAN-600 contributes to obtaining new information about the parameters of plasma at the chromospheric and coronal levels. Current status of information system for RATAN-600 solar observations is described, which is devoted to automatically capturing, storing, transmitting and processing the data and near-real-time publishing them on the Internet. The user web interface for interactive search, visualization, and on-line analysis of the data is available at .     

AstroGrid-D: Grid technology for astronomical science

We present status and results of AstroGrid-D, a joint effort of astrophysicists and computer scientists to employ grid technology for scientific applications. AstroGrid-D provides access to a network of distributed machines with a set of commands as well as software interfaces. It allows simple use of computer and storage facilities and to schedule or monitor compute tasks and data management. It is based on the Globus Toolkit middleware (GT4).Chapter 1 describes the context which led to the demand for advanced software solutions in Astrophysics, and we state the goals of the project.We then present characteristic astrophysical applications that have been implemented on AstroGrid-D in chapter 2. We describe simulations of different complexity, compute-intensive calculations running on multiple sites (Section 2.1), and advanced applications for specific scientific purposes (Section 2.2), such as a connection to robotic telescopes (Section 2.2.3). We can show from these examples how grid execution improves e.g. the scientific workflow.Chapter 3 explains the software tools and services that we adapted or newly developed. Section 3.1 is focused on the administrative aspects of the infrastructure, to manage users and monitor activity. Section 3.2 characterises the central components of our architecture: The AstroGrid-D information service to collect and store metadata, a file management system, the data management system, and a job manager for automatic submission of compute tasks.We summarise the successfully established infrastructure in chapter 4, concluding with our future plans to establish AstroGrid-D as a platform of modern e-Astronomy.     

The Role of Data Centres in the Era of Electronic Publishing|How CDS weaves links towards astronomical databases and archives

The astronomy data centres, and in particular the Centre de Données astronomiques de Strasbourg (CDS), have been building electronic information services for many years. References of publications, observational data related to objects, data tables, nomenclature, have been homogenized and organized into information retrieval systems. This undertaking implied an effort of collaboration between data centres, data providers, agencies, journal editors, etc. Evolution in recent years has brought the data centres closer from the publishing process. General standards for electronic tables, tabular data, and catalogues have been proposed and implemented. With the emergence of fully electronic publication, new digital library services are being organized, and pave the way to innovative new services, linking publications to information from other sources, and making use of new methods for textual information retrieval. The data centres expect to play a key rôle in these new developments, taking advantage of their expertise in the development of value-added services, and of their long-term involvement towards a fully linked astronomy information system.     

Implementation of PhotoZ under Astro-WISE

We describe the implementation of the PhotoZ code in the framework of the Astro-WISE package and as part of the Photometric Classification Server of the PanSTARRS pipeline. Both systems allow the automatic measurement of photometric redshifts for the millions of objects being observed in the PanSTARRS project or expected to be observed by future surveys like KIDS, DES or EUCLID.