A hierarchical indexing strategy for optimizing Apache Spark with HDFS to efficiently query big geospatial raster data

ABSTRACT

Earth observations and model simulations are generating big multidimensional array-based raster data. However, it is difficult to efficiently query these big raster data due to the inconsistency among the geospatial raster data model, distributed physical data storage model, and the data pipeline in distributed computing frameworks. To efficiently process big geospatial data, this paper proposes a three-layer hierarchical indexing strategy to optimize Apache Spark with Hadoop Distributed File System (HDFS) from the following aspects: (1) improve I/O efficiency by adopting the chunking data structure; (2) keep the workload balance and high data locality by building the global index (k-d tree); (3) enable Spark and HDFS to natively support geospatial raster data formats (e.g., HDF4, NetCDF4, GeoTiff) by building the local index (hash table); (4) index the in-memory data to further improve geospatial data queries; (5) develop a data repartition strategy to tune the query parallelism while keeping high data locality. The above strategies are implemented by developing the customized RDDs, and evaluated by comparing the performance with that of Spark SQL and SciSpark. The proposed indexing strategy can be applied to other distributed frameworks or cloud-based computing systems to natively support big geospatial data query with high efficiency.     

Sediment routing in the Zagros foreland basin: Drainage reorganization and a shift from axial to transverse sediment dispersal in the Kurdistan region of Iraq

In the northwestern sector of the Zagros foreland basin, axial fluvial systems initially delivered fine-grained sediments from northwestern source regions into a contiguous basin, and later transverse fluvial systems delivered coarse-grained sediments from northeastern sources into a structurally partitioned basin by fold-thrust deformation. Here we integrate sedimentologic, stratigraphic, palaeomagnetic and geochronologic data from the northwestern Zagros foreland basin to define the Neogene history of deposition and sediment routing in response to progressive advance of the Zagros fold-thrust belt. This study constrains the depositional environments, timing of deposition and provenance of nonmarine clastic deposits of the Injana (Upper Fars), Mukdadiya (Lower Bakhtiari) and Bai-Hasan (Upper Bakhtiari) Formations in the Kurdistan region of Iraq. Sediments of the Injana Formation (~12.4–7.75 Ma) were transported axially (orogen-parallel) from northwest to southeast by meandering and low-sinuosity channel belt system. In contrast, during deposition of the Mukdadiya Formation (~7.75–5 Ma), sediments were delivered transversely (orogen-perpendicular) from northeast to southwest by braided and low-sinuosity channel belt system in distributive fluvial megafans. By ~5 Ma, the northwestern Zagros foreland basin became partitioned by growth of the Mountain Front Flexure and considerable gravel was introduced in localized alluvial fans derived from growing topographic highs. Foredeep accumulation rates during deposition of the Injana, Mukdadiya and Bai-Hasan Formations averaged 350, 400 and 600 m/Myr respectively, suggesting accelerated accommodation generation in a rapidly subsiding basin governed by flexural subsidence. Detrital zircon U-Pb age spectra show that in addition to sources of Mesozoic-Cenozoic cover strata, the Injana Formation was derived chiefly from Palaeozoic-Precambrian (including Carboniferous and latest Neoproterozoic) strata in an axial position to the northwest, likely from the Bitlis-Puturge Massif and broader Eastern Anatolia. In contrast, the Mukdadiya and Bai-Hasan Formations yield distinctive Palaeogene U-Pb age peaks, particularly in the southeastern sector of the study region, consistent with transverse delivery from the arc-related terranes of the Walash and Naopurdan volcano-sedimentary groups (Gaveh-Rud domain?) and Urumieh-Dokhtar magmatic arc to the northeast. These temporal and spatial variations in stratigraphic framework, depositional environments, sediment routing and compositional provenance reveal a major drainage reorganization during Neogene shortening in the Zagros fold-thrust belt. Whereas axial fluvial systems initially dominated the foreland basin during early orogenesis in the Kurdistan region of Iraq, transverse fluvial systems were subsequently established and delivered major sediment volumes to the foreland as a consequence of the abrupt deformation advance and associated topographic growth in the Zagros.     

A comparison of network model creation algorithms based on the quality of wayfinding results

Various network model creation algorithms have been introduced to demonstrate a better approximation of the actual walking pattern and to provide a better wayfinding guide. However, it is under‐investigated which algorithm creates the most appropriate indoor navigation network model in the context of wayfinding applications. Due to the lack of discussion, some studies unconsciously extended an algorithm designed for creating an outdoor navigation network model to indoor space applications. This is problematic because indoor space has different spatial contexts from outdoor space, such as non‐linear space and no‐designated walking space. Our solution is to select five well‐known algorithms that have been introduced, to reproduce the algorithm for the automated construction of the indoor navigation network model, and to evaluate the applicability of algorithms for indoor wayfinding applications. This article compares the quality of wayfinding results from the output of the indoor navigation network model against two criteria: route efficiency (i.e., length) and route simplicity (i.e., number of directions). Our statistical analysis illustrates that the visibility graph algorithm is the most appropriate for indoor wayfinding applications.     

Spatially explicit models for exploring COVID‐19 lockdown strategies

This article describes two spatially explicit models created to allow experimentation with different societal responses to the COVID‐19 pandemic. We outline the work to date on modeling spatially explicit infective diseases and show that there are gaps that remain important to fill. We demonstrate how geographical regions, rather than a single, national approach, are likely to lead to better outcomes for the population. We provide a full account of how our models function, and how they can be used to explore many different aspects of contagion, including: experimenting with different lockdown measures, with connectivity between places, with the tracing of disease clusters, and the use of improved contact tracing and isolation. We provide comprehensive results showing the use of these models in given scenarios, and conclude that explicitly regionalized models for mitigation provide significant advantages over a “one‐size‐fits‐all” approach. We have made our models, and their data, publicly available for others to use in their own locales, with the hope of providing the tools needed for geographers to have a voice during this difficult time.     

A probabilistic framework for improving reverse geocoding output

Reverse geocoding, which transforms machine‐readable GPS coordinates into human‐readable location information, is widely used in a variety of location‐based services and analysis. The output quality of reverse geocoding is critical because it can greatly impact these services provided to end‐users. We argue that the output of reverse geocoding should be spatially close to and topologically correct with respect to the input coordinates, contain multiple suggestions ranked by a uniform standard, and incorporate GPS uncertainties. However, existing reverse geocoding systems often fail to fulfill these aims. To further improve the reverse geocoding process, we propose a probabilistic framework that includes: (1) a new workflow that can adapt all existing address models and unitizes distance and topology relations among retrieved reference data for candidate selections; (2) an advanced scoring mechanism that quantifies characteristics of the entire workflow and orders candidates according to their likelihood of being the best candidate; and (3) a novel algorithm that derives statistical surfaces for input GPS uncertainties and propagates such uncertainties into final output lists. The efficiency of the proposed approaches is demonstrated through comparisons to the four commercial reverse geocoding systems and through human judgments. We envision that more advanced reverse geocoding output ranking algorithms specific to different application scenarios can be built upon this work.     

Effects of pulsed nutrient inputs on phytoplankton assemblage structure and blooms in an enclosed coastal area

The response of phytoplankton assemblage structure to terrestrial nutrient inputs was examined for the Gulf of Kalloni in the Northern Aegean Sea, a productive semi-enclosed coastal marine ecosystem. The study was focused on a typical annual cycle, and emphasis was placed on the comparative analysis between blooms developing after significant nutrient inputs from the watershed, and naturally occurring blooms. Baseline information was collected on a monthly basis from a network of stations located in the oligotrophic open sea and the interior and more productive part of the embayment. Intensive sampling was also carried out along a gradient in the vicinity of a river which was the most important source of freshwater and nutrient input for the Gulf. Phytoplankton assemblage structure was analyzed from 188 samples using diversity indices (Shannon and Average Taxonomic Distinctness), multivariate plotting methods (NMDS), multivariate statistics (PERMANOVA), and canonical correspondence analysis (CCA). Three characteristic assemblages were recognized: (1) an autumn assemblage developed under nutrient depleted conditions, having low diversity due to the dominance of two small diatoms, (2) a winter bloom of the potentially toxic species Pseudo-nitzschia calliantha occurring immediately after a nutrient peak and characterized by very low diversity, and (3) a naturally occurring early summer bloom of centric diatoms with relatively high diversity. The results of the study support the view that moderate nutrient inputs may have a beneficial effect on the functioning of coastal ecosystems, stimulating the taxonomic diversity through the growth of different taxonomic groups and taxa. On the other hand, a sudden pulse of high nutrient concentrations may greatly affect the natural succession of organisms, have a negative effect on the diversity through the dominance of a single species, and can increase the possibility of a harmful algal bloom development.     

Multibeam sonar backscatter data processing

Multibeam sonar systems now routinely record seafloor backscatter data, which are processed into backscatter mosaics and angular responses, both of which can assist in identifying seafloor types and morphology. Those data products are obtained from the multibeam sonar raw data files through a sequence of data processing stages that follows a basic plan, but the implementation of which varies greatly between sonar systems and software. In this article, we provide a comprehensive review of this backscatter data processing chain, with a focus on the variability in the possible implementation of each processing stage. Our objective for undertaking this task is twofold: (1) to provide an overview of backscatter data processing for the consideration of the general user and (2) to provide suggestions to multibeam sonar manufacturers, software providers and the operators of these systems and software for eventually reducing the lack of control, uncertainty and variability associated with current data processing implementations and the resulting backscatter data products. One such suggestion is the adoption of a nomenclature for increasingly refined levels of processing, akin to the nomenclature adopted for satellite remote-sensing data deliverables.     

Combining pixel and object based image analysis of ultra-high resolution multibeam bathymetry and backscatter for habitat mapping in shallow marine waters

Habitat mapping data are increasingly being recognised for their importance in underpinning marine spatial planning. The ability to collect ultra-high resolution (cm) multibeam echosounder (MBES) data in shallow waters has facilitated understanding of the fine-scale distribution of benthic habitats in these areas that are often prone to human disturbance. Developing quantitative and objective approaches to integrate MBES data with ground observations for predictive modelling is essential for ensuring repeatability and providing confidence measures for habitat mapping products. Whilst supervised classification approaches are becoming more common, users are often faced with a decision whether to implement a pixel based (PB) or an object based (OB) image analysis approach, with often limited understanding of the potential influence of that decision on final map products and relative importance of data inputs to patterns observed. In this study, we apply an ensemble learning approach capable of integrating PB and OB Image Analysis from ultra-high resolution MBES bathymetry and backscatter data for mapping benthic habitats in Refuge Cove, a temperate coastal embayment in south-east Australia. We demonstrate the relative importance of PB and OB seafloor derivatives for the five broad benthic habitats that dominate the site. We found that OB and PB approaches performed well with differences in classification accuracy but not discernible statistically. However, a model incorporating elements of both approaches proved to be significantly more accurate than OB or PB methods alone and demonstrate the benefits of using MBES bathymetry and backscatter combined for class discrimination.     

Fuel optimization for low-thrust Earth–Moon transfer via indirect optimal control

The problem of designing low-energy transfers between the Earth and the Moon has attracted recently a major interest from the scientific community. In this paper, an indirect optimal control approach is used to determine minimum-fuel low-thrust transfers between a low Earth orbit and a Lunar orbit in the Sun–Earth–Moon Bicircular Restricted Four-Body Problem. First, the optimal control problem is formulated and its necessary optimality conditions are derived from Pontryagin’s Maximum Principle. Then, two different solution methods are proposed to overcome the numerical difficulties arising from the huge sensitivity of the problem’s state and costate equations. The first one consists in the use of continuation techniques. The second one is based on a massive exploration of the set of unknown variables appearing in the optimality conditions. The dimension of the search space is reduced by considering adapted variables leading to a reduction of the computational time. The trajectories found are classified in several families according to their shape, transfer duration and fuel expenditure. Finally, an analysis based on the dynamical structure provided by the invariant manifolds of the two underlying Circular Restricted Three-Body Problems, Earth–Moon and Sun–Earth is presented leading to a physical interpretation of the different families of trajectories.