The Data Flow, from Observations to the Archive: Simulating the Case of TNG

The telescopes of the new generation allow an archive to be built as a section of data management; nevertheless, careful planning is needed and data handling needs to be designed together with the control system of the telescope itself, both for space-borne and for ground-based facilities. Simulations are essential to understand how observations will be archived, and to build and test an archiving system capable of dealing efficiently with the expected data flow.The TNG (Telescopio Nazionale Galileo) will be one of the first ground-based observing facilities where archiving of both technical and scientific data will be made directly at the telescope as a natural extension of the data handling chain. The results obtained testing the prototype implementation of the archive system at the TNG with a simulated data flow will be shown.     

The SUMER Data in the SOHO Archive

We have released an archive of all observational data of the VUV spectrometer Solar Ultraviolet Measurements of Emitted Radiation (SUMER) on SOHO that have been acquired until now. The operational phase started with ‘first light’ observations on 27 January 1996 and will end in 2014. Future data will be added to the archive when they become available. The archive consists of a set of raw data (Level 0) and a set of data that are processed and calibrated to the best knowledge we have today (Level 1). This communication describes step by step the data acquisition and processing that has been applied in an automated manner to build the archive. It summarizes the expertise and insights into the scientific use of SUMER spectra that has accumulated over the years. It also indicates possibilities for further enhancement of the data quality. With this article we intend to convey our own understanding of the instrument performance to the scientific community and to introduce the new, standard FITS-format database.     

Wide‐Field Plate Archive of the University Observatory Jena

We present the archive of the wide‐field plate observations obtained at the University Observatory Jena, which is stored at the Astrophysical Institute of the Friedrich Schiller University Jena. The archive contains plates taken in the period February 1963 to December 1982 with the 60/90/180‐cm Schmidt telescope of the university observatory. A computer‐readable version of the plate metadata catalogue (for 1257 plates), the logbooks, as well as the digitized Schmidt plates in low and high resolution are now accessible to the astronomical community.This paper describes the properties of the archive, as well as the processing procedure of all plates in detail. (© 2014 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)     

AVES: A high performance computer cluster array for the INTEGRAL satellite scientific data analysis

In this paper we describe a new computing system array, designed, built and now used at the Space Astrophysics and Planetary Institute (IAPS) in Rome, Italy, for the INTEGRAL Space Observatory scientific data analysis. This new system has become necessary in order to reduce the processing time of the INTEGRAL data accumulated during the more than 9 years of in-orbit operation. In order to fulfill the scientific data analysis requirements with a moderately limited investment the starting approach has been to use a ??cluster?? array of commercial quad-CPU computers, featuring the extremely large scientific and calibration data archive on line.     

The GALEX public archive at MAST

The Galaxy Evolution Explorer (GALEX) was successfully launched on April 28, 2003. Over the last 7 years, the GALEX mission has expanded our understanding of the process of star formation and galaxy evolution. GALEX data, consisting of far-UV (1344-1786A) and near-UV (1771-2831A) imaging over most of the sky, and grism spectroscopy, are available to the entire astronomical community and to the general public via the Multi-mission Archive at Space Telescope (MAST). The main characteristics and access tools to the large data archive (16TB) are presented. Groups within the GALEX Science Team, and the MAST archive teams, have undertaken complementary projects to conduct all-sky searches for time-variable and transient sources within GALEX data (e.g. Welsh et al. in Astron. J. 130:823, 2005; Wheatley et al. in Astron. J. 136:259, 2008). Results and tools for variability searches developed by MAST during this process, will be made available to the general community.     

The Kilo-Degree Survey

The Kilo Degree Survey (KiDS) is a 1500 square degree optical imaging survey with the recently commissioned OmegaCAM wide-field imager on the VLT Survey Telescope (VST). A suite of data products will be delivered to the European Southern Observatory (ESO) and the community by the KiDS survey team. Spread over Europe, the KiDS team uses Astro-WISE as its main tool to collaborate efficiently and pool hardware resources. In Astro-WISE the team shares, calibrates and archives all survey data. The data-centric architectural design realizes a dynamic ‘live archive’ in which new KiDS survey products of improved quality can be shared with the team and eventually the full astronomical community in a flexible and controllable manner.     

The Astro-WISE optical image pipeline

We have designed and implemented a novel way to process wide-field astronomical data within a distributed environment of hardware resources and humanpower. The system is characterized by integration of archiving, calibration, and post-calibration analysis of data from raw, through intermediate, to final data products. It is a true integration thanks to complete linking of data lineage from the final catalogs back to the raw data. This paper describes the pipeline processing of optical wide-field astronomical data from the WFI (http://www.eso.org/lasilla/instruments/wfi/) and OmegaCAM (http://www.astro-wise.org/~omegacam/) instruments using the Astro-WISE information system (the Astro-WISE Environment or simply AWE). This information system is an environment of hardware resources and humanpower distributed over Europe. AWE is characterized by integration of archiving, data calibration, post-calibration analysis, and archiving of raw, intermediate, and final data products. The true integration enables a complete data processing cycle from the raw data up to the publication of science-ready catalogs. The advantages of this system for very large datasets are in the areas of: survey operations management, quality control, calibration analyses, and massive processing.     

EURONEAR: Data mining of asteroids and Near Earth Asteroids

Besides new observations, mining old photographic plates and CCD image archives represents an opportunity to recover and secure newly discovered asteroids, also to improve the orbits of Near Earth Asteroids (NEAs), Potentially Hazardous Asteroids (PHAs) and Virtual Impactors (VIs). These are the main research aims of the EURONEAR network. As stated by the IAU, the vast collection of image archives stored worldwide is still insufficiently explored, and could be mined for known NEAs and other asteroids appearing occasionally in their fields. This data mining could be eased using a server to search and classify findings based on the asteroid class and the discovery date as “precoveries” or “recoveries”. We built PRECOVERY, a public facility which uses the Virtual Observatory SkyBoT webservice of IMCCE to search for all known Solar System objects in a given observation. To datamine an entire archive, PRECOVERY requires the observing log in a standard format and outputs a database listing the sorted encounters of NEAs, PHAs, numbered and un‐numbered asteroids classified as precoveries or recoveries based on the daily updated IAU MPC database. As a first application, we considered an archive including about 13 000 photographic plates exposed between 1930 and 2005 at the Astronomical Observatory in Bucharest, Romania. Firstly, we updated the database, homogenizing dates and pointings to a common format using the JD dating system and J2000 epoch. All the asteroids observed in planned mode were recovered, proving the accuracy of PRECOVERY. Despite the large field of the plates imaging mostly 2.27° × 2.27° fields, no NEA or PHA could be encountered occasionally in the archive due to the small aperture of the 0.38m refractor insufficiently to detect objects fainter than V ∼ 15. PRECOVERY can be applied to other archives, being intended as a public facility offered to the community by the EURONEAR project. This is the first of a series of papers aimed to improve orbits of PHAs and NEAs using precovered data derived from archives of images to be data mined in collaboration with students and amateurs. In the next paper we will search the CFHT Legacy Survey, while data mining of other archives is planned for the near future (© 2009 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)     

A New Analysis of Fireball Data from the Meteorite Observation and Recovery Project (MORP)

Sixty fireball cameras operated in Western Canada from 1971 to 1985. Over one thousand (1016) fireballs were recorded at more than one station, but only 367 were reduced, of which 285 have been published, including that of the Innisfree meteorite. Digitization of all the data is underway, and procedures are being developed which will allow the automatic reduction of events not previously examined. The results of the analysis of 80 fireballs reduced but not previously published are presented. When the new analysis is complete, the MORP archive will be a valuable source of information on meteoroid orbits.     

HIDDRA: a highly independent data distribution and retrieval architecture for space observation missions

Institutions such as NASA, ESA or JAXA find solutions to distribute data from their missions to the scientific community, and their long term archives. This is a complex problem, as it includes a vast amount of data, several geographically distributed archives, heterogeneous architectures with heterogeneous networks, and users spread around the world. We propose a novel architecture that solves this problem aiming to fulfill the requirements of the final user. Our architecture is a modular system that provides a highly efficient parallel multiprotocol download engine, using a publisher/subscriber policy which helps the final user to obtain data of interest transparently. We have evaluated a first prototype, in collaboration with the ESAC centre in Villafranca del Castillo (Spain) that shows a high scalability and performance, opening a wide spectrum of opportunities.     

Data Archive of the Hinode Mission

All of the Hinode telemetry data are to be reformatted and archived in the DARTS system at ISAS and mirrored to data centers around the word. The archived data are distributed to users through the Internet. This paper gives an overview of the files in the archive, including the file formats. All formats are portable and have heritage from the previous missions. From the reformatted files, index information is created for faster data search. Users can perform queries based on information contained in the index. This allows for searches to return observations that conform to particular observing conditions.     

A systematic analysis of the <Emphasis Type="Italic">XMM-Newton</Emphasis> background: II. Properties of the in-Field-Of-View excess component

We present an accurate characterization of the particle background behaviour on XMM-Newton based on the entire EPIC archive. This corresponds to the largest EPIC data set ever examined. Our results have been obtained thanks to the collaboration between the FP7 European program EXTraS and the ESA R&D ATHENA activity AREMBES. We used as a diagnostic an improved version of the diagnostic which compares the data collected in unexposed region of the detector with the region of the field of view in the EPIC-MOS. We will show that the in Field-of-View excess background is made up of two different components, one associated to flares produced by soft protons and the other one to a low-intensity background. Its origin needs to be further investigated.     

Retrieval algorithm for atmospheric dust properties from Mars Global Surveyor Thermal Emission Spectrometer data during global dust storm 2001A

We present a new parameter retrieval algorithm for Mars Global Surveyor Thermal Emission Spectrometer data. The algorithm uses Newtonian first-order sensitivity functions of the infrared spectrum in response to variations in physical parameters to fit a model spectrum to the data at 499, 1099, and 1301 cm⁻¹. The algorithm iteratively fits the model spectrum to data to simultaneously retrieve dust extinction optical depth, effective radius, and surface temperature. There are several sources of uncertainty in the results. The assumed dust vertical distribution can introduce errors in retrieved optical depth of a few tens of percent. The assumed dust optical constants can introduce errors in both optical depth and effective radius, although the systematic nature of these errors will not affect retrieval of trends in these parameters. The algorithm does not include the spectral signature of water ice, and hence data needs to be filtered against this parameter before the algorithm is applied. The algorithm also needs sufficient dust spectral signature, and hence surface-to-atmosphere temperature contrast, to successfully retrieve the parameters. After the application of data filters the algorithm is both relatively accurate and very fast, successfully retrieving parameters, as well as meaningful parameter variability and trends from tens of thousands of individual spectra on a global scale (Elteto, A., Toon, O.B. [2010]. Icarus, this issue). Our results for optical depth compare well with TES archive values when corrected by the single scattering albedo. Our results are on average 1–4 K higher in surface temperatures from the TES archive values, with greater differences at higher optical depths. Our retrieval of dust effective radii compare well with the retrievals of Wolff and Clancy (Wolff, M.J., Clancy, R.T. [2003]. J. Geophys. Res. 108 (E9), 5097) for the corresponding data selections from the same orbits.     

Astronomical relativistic reference systems with multipolar expansion: the global one

With the rapid development of techniques for astronomical observations,the precision of measurements has been significantly increasing. Theories describing astronomical relativistic reference systems, which are the foundation for processing and interpreting these data now and in the future, may require extensions to satisfy the needs of these trends. Besides building a framework compatible with alternative theories of gravity and the pursuit of higher order post-Newtonian approximation, it will also be necessary to make the first order post-Newtonian multipole moments of celestial bodies be explicitly expressed in the astronomical relativistic reference systems.This will bring some convenience into modeling the observations and experiments and make it easier to distinguish different contributions in measurements. As a first step,the global solar system reference system is expressed as a multipolar expansion and the post-Newtonian mass and spin moments are shown explicitly in the metric which describes the coordinates of the system. The full expression of the global metric is given.     

Antenna data storage concept for phased array radio astronomical instruments

Low frequency Radio Astronomy instruments like LOFAR and SKA-LOW use arrays of dipole antennas for the collection of radio signals from the sky. Due to the large number of antennas involved, the total data rate produced by all the antennas is enormous. Storage of the antenna data is both economically and technologically infeasible using the current state of the art storage technology. Therefore, real-time processing of the antenna voltage data using beam forming and correlation is applied to achieve a data reduction throughout the signal chain. However, most science could equally well be performed using an archive of raw antenna voltage data coming straight from the A/D converters instead of capturing and processing the antenna data in real time over and over again. Trends on storage and computing technology make such an approach feasible on a time scale of approximately 10 years. The benefits of such a system approach are more science output and a higher flexibility with respect to the science operations. In this paper we present a radically new system concept for a radio telescope based on storage of raw antenna data. LOFAR is used as an example for such a future instrument.     

The Chandra X-ray Observatory: An Astronomical Facility Available to the World

The Chandra X-ray observatory, one of NASA’s “Great Observatories,” provides high angular and spectral resolution X-ray data which is freely available to all. In this review I describe the instruments on Chandra along with their current calibration, as well as the Chandra proposal system, the freely-available Chandra analysis software package CIAO, and the Chandra archive. As Chandra is in its 6th year of operation, the archive already contains calibrated observations of a large range of X-ray sources. The Chandra X-ray Center is committed to assisting astronomers from any country who wish to use data from the archive or propose for observations.     

基于分布式执行框架的低频射电干涉阵列成像管线优化

平方公里阵列(Square Kilometre Array,SKA)项目是建设全球最大射电望远镜的国际合作项目,其灵敏度和测量速度将比当前所有的射电望远镜都要高出一个数量级.连续谱巡天是SKA的主要观测模式之一,基于连续谱成像建立巡天区域的标准星图,将能为后续天文科学研究奠定重要基础.银河系与河外星系全天默奇森宽场阵列拓展巡天(GaLactic and Extragalactic All-sky Murchison Widefield Array survey eXtended,GLEAM-X)是2018—2020年利用SKA先导望远镜默奇森宽场阵列(Murchison Wide-field Array,MWA)二期拓展阵列开展的新的射电连续谱巡天项目,观测期间积累了大量的低频巡天观测数据.海量观测数据的自动化、大批量处理是SKA望远镜项目所面临的的最大挑战和难题之一,基于分布式执行框架的成像管线优化经验将有助于解决海量数据处理问题.详细介绍了GLEAM-X成像管线并对其进行整合和改进,在中国SKA区域中心原型机(China SKA Regional Centre Prototype,...