The Astro-WISE optical image pipeline

We have designed and implemented a novel way to process wide-field astronomical data within a distributed environment of hardware resources and humanpower. The system is characterized by integration of archiving, calibration, and post-calibration analysis of data from raw, through intermediate, to final data products. It is a true integration thanks to complete linking of data lineage from the final catalogs back to the raw data. This paper describes the pipeline processing of optical wide-field astronomical data from the WFI (http://www.eso.org/lasilla/instruments/wfi/) and OmegaCAM (http://www.astro-wise.org/~omegacam/) instruments using the Astro-WISE information system (the Astro-WISE Environment or simply AWE). This information system is an environment of hardware resources and humanpower distributed over Europe. AWE is characterized by integration of archiving, data calibration, post-calibration analysis, and archiving of raw, intermediate, and final data products. The true integration enables a complete data processing cycle from the raw data up to the publication of science-ready catalogs. The advantages of this system for very large datasets are in the areas of: survey operations management, quality control, calibration analyses, and massive processing.     

The Astro-WISE approach to quality control for astronomical data

We present a novel approach to quality control during the processing of astronomical data. Quality control in the Astro-WISE Information System is integral to all aspects of data handing and provides transparent access to quality estimators for all stages of data reduction from the raw image to the final catalog. The implementation of quality control mechanisms relies on the core features in this Astro-WISE Environment (AWE): an object-oriented framework, full data lineage, and both forward and backward chaining. Quality control information can be accessed via the command-line awe-prompt and the web-based Quality-WISE service. The quality control system is described and qualified using archive data from the 8-CCD Wide Field Imager (WFI) instrument (http://www.eso.org/lasilla/instruments/wfi/) on the 2.2-m MPG/ESO telescope at La Silla and (pre-)survey data from the 32-CCD OmegaCAM instrument (http://www.astro-wise.org/~omegacam/) on the VST telescope at Paranal.     

The Helioseismic and Magnetic Imager (HMI) Vector Magnetic Field Pipeline: SHARPs – Space-Weather HMI Active Region Patches

A new data product from the Helioseismic and Magnetic Imager (HMI) onboard the Solar Dynamics Observatory (SDO) called Space-weather HMI Active Region Patches (SHARPs) is now available. SDO/HMI is the first space-based instrument to map the full-disk photospheric vector magnetic field with high cadence and continuity. The SHARP data series provide maps in patches that encompass automatically tracked magnetic concentrations for their entire lifetime; map quantities include the photospheric vector magnetic field and its uncertainty, along with Doppler velocity, continuum intensity, and line-of-sight magnetic field. Furthermore, keywords in the SHARP data series provide several parameters that concisely characterize the magnetic-field distribution and its deviation from a potential-field configuration. These indices may be useful for active-region event forecasting and for identifying regions of interest. The indices are calculated per patch and are available on a twelve-minute cadence. Quick-look data are available within approximately three hours of observation; definitive science products are produced approximately five weeks later. SHARP data are available at jsoc.stanford.edu and maps are available in either of two different coordinate systems. This article describes the SHARP data products and presents examples of SHARP data and parameters.     

Leveraging data lineage to infer logical relationships between astronomical catalogs

A novel method to infer logical relationships between sets is presented. These sets can be any collection of elements, for example astronomical catalogs of celestial objects. The method does not require the contents of the sets to be known explicitly. It combines incomplete knowledge about the relationships between sets to infer a priori unknown relationships. Relationships between sets are represented by sets of Boolean hypercubes. This leads to deductive reasoning by application of logical operators to these sets of hypercubes. A pseudo code for an efficient implementation is described. The method is used in the Astro-WISE information system to infer relationships between catalogs of astronomical objects. These catalogs can be very large and, more importantly, their contents do not have to be available at all times. Science products are stored in Astro-WISE with references to other science products from which they are derived, or their dependencies. This creates a full data lineage that links every science product all the way back to the raw data. Catalogs are created in a way that maximizes knowledge about their relationship with their dependencies. The presented algorithm is used to determine which objects a catalog represents by leveraging this information.     

EIDOSCOPE: particle acceleration at plasma boundaries

We describe the mission concept of how ESA can make a major contribution to the Japanese Canadian multi-spacecraft mission SCOPE by adding one cost-effective spacecraft EIDO (Electron and Ion Dynamics Observatory), which has a comprehensive and optimized plasma payload to address the physics of particle acceleration. The combined mission EIDOSCOPE will distinguish amongst and quantify the governing processes of particle acceleration at several important plasma boundaries and their associated boundary layers: collisionless shocks, plasma jet fronts, thin current sheets and turbulent boundary layers. Particle acceleration and associated cross-scale coupling is one of the key outstanding topics to be addressed in the Plasma Universe. The very important science questions that only the combined EIDOSCOPE mission will be able to tackle are: 1) Quantitatively, what are the processes and efficiencies with which both electrons and ions are selectively injected and subsequently accelerated by collisionless shocks? 2) How does small-scale electron and ion acceleration at jet fronts due to kinetic processes couple simultaneously to large scale acceleration due to fluid (MHD) mechanisms? 3) How does multi-scale coupling govern acceleration mechanisms at electron, ion and fluid scales in thin current sheets? 4) How do particle acceleration processes inside turbulent boundary layers depend on turbulence properties at ion/electron scales? EIDO particle instruments are capable of resolving full 3D particle distribution functions in both thermal and suprathermal regimes and at high enough temporal resolution to resolve the relevant scales even in very dynamic plasma processes. The EIDO spin axis is designed to be sun-pointing, allowing EIDO to carry out the most sensitive electric field measurements ever accomplished in the outer magnetosphere. Combined with a nearby SCOPE Far Daughter satellite, EIDO will form a second pair (in addition to SCOPE Mother-Near Daughter) of closely separated satellites that provides the unique capability to measure the 3D electric field with high accuracy and sensitivity. All EIDO instrumentation are state-of-the-art technology with heritage from many recent missions. The EIDOSCOPE orbit will be close to equatorial with apogee 25-30 RE and perigee 8-10 RE. In the course of one year the orbit will cross all the major plasma boundaries in the outer magnetosphere; bow shock, magnetopause and magnetotail current sheets, jet fronts and turbulent boundary layers. EIDO offers excellent cost/benefits for ESA, as for only a fraction of an M-class mission cost ESA can become an integral part of a major multi-agency L-class level mission that addresses outstanding science questions for the benefit of the European science community.     

Query driven visualization of astronomical catalogs

Interactive visualization of astronomical catalogs requires novel techniques due to the huge volumes and complex structure of the data produced by existing and upcoming astronomical surveys. The creation as well as the disclosure of the catalogs can be handled by data pulling mechanisms (Buddelmeijer et al. 2011). These prevent unnecessary processing and facilitate data sharing by having users request the desired end products. In this work we present query driven visualization as a logical continuation of data pulling. Scientists can request catalogs in a declarative way and set process parameters directly from within the visualization. This results in profound interoperation between software with a high level of abstraction. New messages for the Simple Application Messaging Protocol are proposed to achieve this abstraction. Support for these messages are implemented in the Astro-WISE information system and in a set of demonstrational applications.     

Solar Flare Prediction Using Advanced Feature Extraction, Machine Learning, and Feature Selection

Novel machine-learning and feature-selection algorithms have been developed to study: i) the flare-prediction-capability of magnetic feature (MF) properties generated by the recently developed Solar Monitor Active Region Tracker (SMART); ii) SMART’s MF properties that are most significantly related to flare occurrence. Spatiotemporal association algorithms are developed to associate MFs with flares from April 1996 to December 2010 in order to differentiate flaring and non-flaring MFs and enable the application of machine-learning and feature-selection algorithms. A machine-learning algorithm is applied to the associated datasets to determine the flare-prediction-capability of all 21 SMART MF properties. The prediction performance is assessed using standard forecast-verification measures and compared with the prediction measures of one of the standard technologies for flare-prediction that is also based on machine-learning: Automated Solar Activity Prediction (ASAP). The comparison shows that the combination of SMART MFs with machine-learning has the potential to achieve more accurate flare-prediction than ASAP. Feature-selection algorithms are then applied to determine the MF properties that are most related to flare occurrence. It is found that a reduced set of six MF properties can achieve a similar degree of prediction accuracy as the full set of 21 SMART MF properties.     

Evaluation of a Revised Interplanetary Shock Prediction Model: 1D CESE-HD-2 Solar-Wind Model

We modified the one-dimensional conservation element and solution element (CESE) hydrodynamic (HD) model into a new version [1D CESE-HD-2], by considering the direction of the shock propagation. The real-time performance of the 1D CESE-HD-2 model during Solar Cycle 23 (February 1997?–?December 2006) is investigated and compared with those of the Shock Time of Arrival Model (STOA), the Interplanetary-Shock-Propagation Model (ISPM), and the Hakamada–Akasofu–Fry version 2 (HAFv.2). Of the total of 584 flare events, 173 occurred during the rising phase, 166 events during the maximum phase, and 245 events during the declining phase. The statistical results show that the success rates of the predictions by the 1D CESE-HD-2 model for the rising, maximum, declining, and composite periods are 64 %, 62 %, 57 %, and 61 %, respectively, with a hit window of ±?24 hours. The results demonstrate that the 1D CESE-HD-2 model shows the highest success rates when the background solar-wind speed is relatively fast. Thus, when the background solar-wind speed at the time of shock initiation is enhanced, the forecasts will provide potential values to the customers. A high value (27.08) of χ ² and low p-value (<?0.0001) for the 1D CESE-HD-2 model give considerable confidence for real-time forecasts by using this new model. Furthermore, the effects of various shock characteristics (initial speed, shock duration, background solar wind, longitude, etc.) and background solar wind on the forecast are also investigated statistically.     

A Multi-wavelength Analysis of Active Regions and Sunspots by Comparison of Automatic Detection Algorithms

Since the Solar Dynamics Observatory (SDO) began recording ≈?1 TB of data per day, there has been an increased need to automatically extract features and events for further analysis. Here we compare the overall detection performance, correlations between extracted properties, and usability for feature tracking of four solar feature-detection algorithms: the Solar Monitor Active Region Tracker (SMART) detects active regions in line-of-sight magnetograms; the Automated Solar Activity Prediction code (ASAP) detects sunspots and pores in white-light continuum images; the Sunspot Tracking And Recognition Algorithm (STARA) detects sunspots in white-light continuum images; the Spatial Possibilistic Clustering Algorithm (SPoCA) automatically segments solar EUV images into active regions (AR), coronal holes (CH), and quiet Sun (QS). One month of data from the Solar and Heliospheric Observatory (SOHO)/Michelson Doppler Imager (MDI) and SOHO/Extreme Ultraviolet Imaging Telescope (EIT) instruments during 12 May?–?23 June 2003 is analysed. The overall detection performance of each algorithm is benchmarked against National Oceanic and Atmospheric Administration (NOAA) and Solar Influences Data Analysis Center (SIDC) catalogues using various feature properties such as total sunspot area, which shows good agreement, and the number of features detected, which shows poor agreement. Principal Component Analysis indicates a clear distinction between photospheric properties, which are highly correlated to the first component and account for 52.86% of variability in the data set, and coronal properties, which are moderately correlated to both the first and second principal components. Finally, case studies of NOAA 10377 and 10365 are conducted to determine algorithm stability for tracking the evolution of individual features. We find that magnetic flux and total sunspot area are the best indicators of active-region emergence. Additionally, for NOAA 10365, it is shown that the onset of flaring occurs during both periods of magnetic-flux emergence and complexity development.     

Interpreting the Helioseismic and Magnetic Imager (HMI) Multi-Height Velocity Measurements

The Solar Dynamics Observatory/Helioseismic and Magnetic Imager (SDO/HMI) filtergrams, taken at six wavelengths around the Fe i 6173.3 Å line, contain information about the line-of-sight velocity over a range of heights in the solar atmosphere. Multi-height velocity inferences from these observations can be exploited to study wave motions and energy transport in the atmosphere. Using realistic convection-simulation datasets provided by the STAGGER and MURaM codes, we generate synthetic filtergrams and explore several methods for estimating Dopplergrams. We investigate at which height each synthetic Dopplergram correlates most strongly with the vertical velocity in the model atmospheres. On the basis of the investigation, we propose two Dopplergrams other than the standard HMI-algorithm Dopplergram produced from HMI filtergrams: a line-center Dopplergram and an average-wing Dopplergram. These two Dopplergrams correlate most strongly with vertical velocities at the heights of 30?–?40 km above (line center) and 30?–?40 km below (average wing) the effective height of the HMI-algorithm Dopplergram. Therefore, we can obtain velocity information from two layers separated by about a half of a scale height in the atmosphere, at best. The phase shifts between these multi-height Dopplergrams from observational data as well as those from the simulated data are also consistent with the height-difference estimates in the frequency range above the photospheric acoustic-cutoff frequency.     

TomograPy: A Fast, Instrument-Independent, Solar Tomography Software

Solar tomography has progressed rapidly in recent years thanks to the development of robust algorithms and the availability of more powerful computers. It can today provide crucial insights in solving issues related to the line-of-sight integration present in the data of solar imagers and coronagraphs. However, there remain challenges such as the increase of the available volume of data, the handling of the temporal evolution of the observed structures, and the heterogeneity of the data in multi-spacecraft studies. We present a generic software package that can perform fast tomographic inversions that scales linearly with the number of measurements, linearly with the length of the reconstruction cube (and not the number of voxels), and linearly with the number of cores and can use data from different sources and with a variety of physical models: TomograPy ( http://nbarbey.github.com/TomograPy/ ), an open-source software freely available on the Python Package Index. For performance, TomograPy uses a parallelized-projection algorithm. It relies on the World Coordinate System standard to manage various data sources. A variety of inversion algorithms are provided to perform the tomographic-map estimation. A test suite is provided along with the code to ensure software quality. Since it makes use of the Siddon algorithm it is restricted to rectangular parallelepiped voxels but the spherical geometry of the corona can be handled through proper use of priors. We describe the main features of the code and show three practical examples of multi-spacecraft tomographic inversions using STEREO/EUVI and STEREO/COR1 data. Static and smoothly varying temporal evolution models are presented.     

CorPITA: An Automated Algorithm for the Identification and Analysis of Coronal “EIT Waves”

The continuous stream of data available from the Atmospheric Imaging Assembly (AIA) telescopes onboard the Solar Dynamics Observatory (SDO) spacecraft has allowed a deeper understanding of the Sun. However, the sheer volume of data has necessitated the development of automated techniques to identify and analyse various phenomena. In this article, we describe the Coronal Pulse Identification and Tracking Algorithm (CorPITA) for the identification and analysis of coronal “EIT waves”. CorPITA uses an intensity-profile technique to identify the propagating pulse, tracking it throughout its evolution before returning estimates of its kinematics. The algorithm is applied here to a data set from February 2011, allowing its capabilities to be examined and critiqued. This algorithm forms part of the SDO Feature Finding Team initiative and will be implemented as part of the Heliophysics Event Knowledgebase (HEK). This is the first fully automated algorithm to identify and track the propagating “EIT wave” rather than any associated phenomenon and will allow a deeper understanding of this controversial phenomenon.     

Optical spectral decomposition of NGC 4051

Considering the host galaxy contribution, a spectral decomposition method is used to reanalyzed the archive data of optical spectra for a narrow line Seyfert 1 galaxy, NGC 4051. The light curves of the continuum f _λ (5100 Å), and Hβ, He ii, Fe ii emission lines are given. We find strong flux correlations between line emissions of Hβ, He ii, Fe ii and the continuum f _λ (5100 Å). These low-ionization lines (Hβ, Fe ii, He ii) have “inverse” intrinsic Baldwin effects. Using the methods of the cross-correlation function and the Monte Carlo simulation, we find the time delays, with respect to the continuum, are $3.45^{+12.0}_{-0.5}~\mbox{days}$ with the probability of 34 % for the intermediate component of Hβ, $6.45^{+13.0}_{-1.0}~\mbox{days}$ with the probability of 65 % for the intermediate component of He ii. From these intermediate components of Hβ and He ii, the calculated central black hole masses are $0.86^{+4.35}_{-0.33}\times 10^{6}$ and $0.82^{+3.12}_{-0.45}\times 10^{6}~M_{\odot }$ . We also find that the time delays for Fe ii are $9.7^{+3.0}_{-5.0}~\mbox{days}$ with the probability of 36 %, $8.45^{+1.0}_{-2.0}~\mbox{days}$ with the probability of 18 % for the total epochs and “subset 1” data, respectively. It seems that the Fe ii emission region is outside of the Hβ emission region.     

Spectral Line Selection for HMI: A Comparison of Fe I 6173 Å and Ni I 6768 Å

We present a study of two spectral lines, Fe I 6173 Å and Ni I 6768 Å, that were candidates to be used in the Helioseismic and Magnetic Imager (HMI) for observing Doppler velocity and the vector magnetic field. The line profiles were studied using the Mt. Wilson Observatory, the Advanced Stokes Polarimeter and the Kitt Peak-McMath Pierce telescope and one-meter Fourier transform spectrometer atlas. Both Fe I and Ni I profiles have clean continua and no blends that threaten instrument performance. The Fe I line is 2% deeper, 15% narrower, and has a 6% smaller equivalent width than the Ni I line. The potential of each spectral line to recover pre-assigned solar conditions is tested using a least-squares minimization technique to fit Milne-Eddington models to tens of thousands of line profiles that have been sampled at five spectral positions across the line. Overall, the Fe I line has a better performance than the Ni I line for vector-magnetic-field retrieval. Specifically, the Fe I line is able to determine field strength, longitudinal and transverse flux four times more accurately than the Ni I line in active regions. Inclination and azimuthal angles can be recovered to ≈2^° above 600 Mx cm^?2 for Fe I and above 1000 Mx cm^?2 for Ni I. Therefore, the Fe I line better determines the magnetic-field orientation in plage, whereas both lines provide good orientation determination in penumbrae and umbrae. We selected the Fe I spectral line for use in HMI due to its better performance for magnetic diagnostics while not sacrificing velocity information. The one exception to the better performance of the Fe I line arises when high field strengths combine with high velocities to move the spectral line beyond the effective sampling range. The higher g _eff of Fe I means that its useful range of velocity values in regions of strong magnetic field is smaller than Ni I.     

Transient Artifacts in a Flare Observed by the Helioseismic and Magnetic Imager on the Solar Dynamics Observatory

The Helioseismic and Magnetic Imager (HMI) onboard the Solar Dynamics Observatory (SDO) provides a new tool for the systematic observation of white-light flares, including Doppler and magnetic information as well as continuum. In our initial analysis of the highly impulsive $\mathrm{\gamma}$ -ray flare SOL2010-06-12T00:57 (Martínez Oliveros et al., Solar Phys. 269, 269, 2011), we reported the signature of a strong blueshift in the two footpoint sources. Concerned that this might be an artifact due to aliasing peculiar to the HMI instrument, we undertook a comparative analysis of Global Oscillation Network Group (GONG++) observations of the same flare, using the PArametric Smearing Correction ALgorithm (PASCAL) algorithm to correct for artifacts caused by variations in atmospheric smearing. This analysis confirms the artifactual nature of the apparent blueshift in the HMI observations, finding weak redshifts at the footpoints instead. We describe the use of PASCAL with GONG++ observations as a complement to the SDO observations and discuss constraints imposed by the use of HMI far from its design conditions. With proper precautions, these data provide rich information on flares and transients.     

Spatial profiles of lines in active region loops

From the statistical treatment of magnetohydrodynamically turbulent plasma, a steady-state density, temperature and magnetic field structure is derived for a coronal loop emitting in UV and EUV range. Spatial variation of line flux is presented for the lines C ii, C iii, O iv, O vi, Ne vii, and Mgx. It is found that the hotter lines which are emitted near the surface of the loop have larger spatial extents compared to the lines originating in the cool core of the loop, in agreement with the observations.     

Lifetimes of some excited states in the rare earths: La ii,Ce ii,Pr ii,Nd ii,Sm ii,Yb i,Yb ii,and Lu ii

Radiative lifetimes for excited states in La ii, Ce ii, Pr ii, Nd ii, Sm ii, Yb i, Yb ii, and Lu ii have been determined by means of the beam-foil technique or the zero-field level-crossing method. The lifetimes for La, Ce, Pr, Nd, Sm, and Yb are shorter than those computed by summing the transition probabilities of Corliss and Bozman by a factor of up to ~5. The large discrepancies between the abundance of La, Ce, Pr, Nd, and Sm in the solar photosphere and in meteorites are eliminated or greatly reduced if the abundance determinations of the solar photosphere are based upon the gf values of Corliss and Bozman, corrected for by the present lifetimes.     

Sub-image data processing in Astro-WISE

Most often, astronomers are interested in a source (e.g., moving, variable, or extreme in some colour index) that lies on a few pixels of an image. However, the classical approach in astronomical data processing is the processing of the entire image or set of images even when the sole source of interest may exist on only a few pixels of one or a few images. This is because pipelines have been written and designed for instruments with fixed detector properties (e.g., image size, calibration frames, overscan regions, etc.). Furthermore, all metadata and processing parameters are based on an instrument or a detector. Accordingly, out of many thousands of images for a survey, this can lead to unnecessary processing of data that is both time-consuming and wasteful. We describe the architecture and an implementation of sub-image processing in Astro-WISE. The architecture enables a user to select, retrieve and process only the relevant pixels in an image where the source exists. We show that lineage data collected during the processing and analysis of datasets can be reused to perform selective reprocessing (at sub-image level) on datasets while the remainder of the dataset is untouched, a difficult process to automate without lineage.     

Physical conditions in a quiescent prominence derived from UV spectra obtained with the UVSP instrument on the SMM

A quiescent prominence observed above the north-west limb on November 20, 1980, is analyzed using data obtained with the Ultraviolet Spectrometer and Polarimeter (UVSP) on the Solar Maximum Mission (SMM). The spectral data include the lines 1215 Å of Hi, 1401 Å of Oiv, 1402 Å of Siiv, 1548 Å of Civ, 1640 Å of Hei, and 1655 Å of Ci. From an analysis of these lines and their emission patterns we deduce physical characteristics of the prominence plasma, and suggest in particular that the prominence consisted of flux tubes at various temperatures. In the hotter parts of the plasma the number density reached values of about 3 × 10¹¹ cm^#X2212;3.     

Fourier analysis of light curves of eclipsing variable stars

The photometric perturbationsB _h ^(l) arising from both tidal and rotational distortion of a close eclipsing binary have been given in two previous papers (Livaniou, 1977; Rovithis-Livaniou, 1977). The aim of the present paper will be to find the eclipse perturbationsB _2m =B _{2m, tid} +B _{2m, rot} of a close binary exhibiting partial eclipses. This will be done giving the suitable combinations of theB _h ^(l) 's and will make easier the application to real stars. After a very brief introduction, Section 2 gives both theB _{2m, tid} andB _{2m, rot} for uniformly bright discs; while in Sections 3 and 4 they are given for linear and quadratic limb-darkening, respectively. Finally, Section 5 gives a brief discussion of the results.