A general-purpose framework for parallel processing of large-scale LiDAR data

ABSTRACT

Light detection and ranging (LiDAR) data are essential for scientific discoveries such as Earth and ecological sciences, environmental applications, and responding to natural disasters. While collecting LiDAR data over large areas is quite possible the subsequent processing steps typically involve large computational demands. Efficiently storing, managing, and processing LiDAR data are the prerequisite steps for enabling these LiDAR-based applications. However, handling LiDAR data poses grand geoprocessing challenges due to data and computational intensity. To tackle such challenges, we developed a general-purpose scalable framework coupled with a sophisticated data decomposition and parallelization strategy to efficiently handle ‘big’ LiDAR data collections. The contributions of this research were (1) a tile-based spatial index to manage big LiDAR data in the scalable and fault-tolerable Hadoop distributed file system, (2) two spatial decomposition techniques to enable efficient parallelization of different types of LiDAR processing tasks, and (3) by coupling existing LiDAR processing tools with Hadoop, a variety of LiDAR data processing tasks can be conducted in parallel in a highly scalable distributed computing environment using an online geoprocessing application. A proof-of-concept prototype is presented here to demonstrate the feasibility, performance, and scalability of the proposed framework.     

Exploring Real‐Time Geoprocessing in Cloud Computing: Navigation Services Case Study

Today, many real‐time geospatial applications (e.g. navigation and location‐based services) involve data‐ and/or compute‐intensive geoprocessing tasks where performance is of great importance. Cloud computing, a promising platform with a large pool of storage and computing resources, could be a practical solution for hosting vast amounts of data and for real‐time processing. In this article, we explored the feasibility of using Google App Engine (GAE), the cloud computing technology by Google, for a module in navigation services, called Integrated GNSS (iGNSS) QoS prediction. The objective of this module is to predict quality of iGNSS positioning solutions for prospective routes in advance. iGNSS QoS prediction involves the real‐time computation of large Triangulated Irregular Networks (TINs) generated from LiDAR data. We experimented with the Google App Engine (GAE) and stored a large TIN for two geoprocessing operations (proximity and bounding box) required for iGNSS QoS prediction. The experimental results revealed that while cloud computing can potentially be used for development and deployment of data‐ and/or compute‐intensive geospatial applications, current cloud platforms require improvements and special tools for handling real‐time geoprocessing, such as iGNSS QoS prediction, efficiently. The article also provides a set of general guidelines for future development of real‐time geoprocessing in clouds.     

A Service Oriented Geoprocessing System for Real‐Time Road Traffic Monitoring

The emergence of the Sensor Web has paved the way for a new set of innovative software applications that exploit the enhanced availability of real‐time information. This article describes one such application built from Sensor Web components that aggregates GPS track data from a fleet of vehicles to provide an overview of road traffic congestion at the city scale. The application embodies a Service Oriented Architecture; web service components are used to archive and pre‐process incoming sensor observations, to encapsulate a horizontally partitioned spatial database that performs geoprocessing, and to disseminate results to client applications. Our results confirm that floating car data can provide an accurate depiction of current road traffic conditions. The presented solution uses Open Geospatial Consortium web services where possible and serves to highlight the difficulties inherent in achieving horizontal database scalability in sensor based geoprocessing systems.     

GeoPW: Laying Blocks for the Geospatial Processing Web

Recent advances in Web‐related technologies have significantly promoted the wide sharing and integrated analysis of distributed geospatial data. Geospatial applications often involve diverse sources of data and complex geoprocessing functions. Existing Web‐based GIS focuses more on access to distributed geospatial data. In scientific problem solving, the ability to carry out geospatial analysis is essential to geoscientific discovery. This article presents the design and implementation of GeoPW, a set of services providing geoprocessing functions over the Web. The concept of the Geospatial Processing Web is discussed to address the geoprocessing demands in the emerging information infrastructure, and the role of GeoPW in establishing the Geospatial Processing Web is identified. The services in GeoPW are implemented by developing middleware that wraps legacy GIS analysis components to provide a large number of geoprocessing utilities over the Web. These services are open and accessible to the public, and they support integrated geoprocessing on the Web.     

Ontological Engineering for Interpreting Geospatial Queries

Despite advancements in geographic information system (GIS) technology, the efficient and effective utilization of GIS to solve geospatial problems is a daunting process requiring specialized knowledge and skills. Two of the most important and burdensome tasks in this process are interpretation of geospatial queries and mapping the interpreted results into geospatial data models and geoprocessing operations. With the current state of GIS, there exists a gap between the knowledge user's possess and the knowledge and skills they need to utilize GIS for solving problems. Currently, users resort to training and practice on GIS technology or involving GIS experts. Neither of these options is optimal and there is a need for a new approach that automates geoprocessing tasks using GIS technology. This paper presents an ontological engineering methodology that uses multiple ontologies and the mappings among them to automate certain tasks related to interpretation of geospatial queries and mapping the interpreted results into geospatial data models and geoprocessing operations. The presented methodology includes conceptualization of geospatial queries, knowledge representation for queries, techniques for relating elements in different ontologies, and an algorithm that uses ontologies to map queries to geoprocessing operations.     

Agent技术在分布式GIS中的应用研究

首先分析了分布式计算技术的发展，详细介绍了当今主流的分布计算技术——Agent技术、分布式GIS技术是以分布式计算技术为依托的，Agent技术将为分布GIS带来新的技术发展；随后，深入分析了Agent技术在分布式GIS中的应用，并结合已有的研究重点介绍了基于Agent的分布式GIS建模、空间信息查找与获取、WebGIS服务体系、空间辅助决策以及GIS互操作等。     

A Task Ontology Driven Approach for Live Geoprocessing in a Service‐Oriented Environment

In a service‐oriented environment, Web geoprocessing services can provide geoprocessing functions for a variety of applications including Sensor Web. Connecting Sensor Web and geoprocessing services together shows great potentail to support live geoprocessing using real‐time data inputs. This article proposes a task ontology driven approach to live geoprocessing. The task in the ontology contains five aspects: task type, task priority, task constraints, task model, and task process. The use of the task ontology in driving live geoprocessing includes the following steps: (1) Task model generation, which generates a concrete process model to fulfill user demands; (2) Process model instantiation, which transforms the process model into an executable workflow; (3) Workflow execution: the workflow engine executes the workflow to generate value‐added data products using Sensor Web data as inputs. The approach not only helps create semantically correct connections between Sensor Web and Web geoprocessing services, but also provides sharable problem solving knowledge using process models. A prototype system, which leverages Web 2.0, Sensor Web, Semantic Web, and geoprocessing services, is developed to demonstrate the applicability of the approach.     

Advancements in web‐mapping tools for land use and marine spatial planning

Land use and marine spatial planning processes are increasingly supported by systematic assessment techniques, particularly by multi‐criteria spatial analysis methods. This has been facilitated by the growing release and uptake of web‐mapping tools, which contribute to transparent, consistent, and informed planning processes and decisions. This article reviews the usability, functionality, and applicability of contemporary planning web‐mapping tools to identify the state‐of‐the‐art and future prospects. The review reveals that interfaces are increasingly available and intuitively applicable by non‐specialized users. Basic map navigation and data querying functionality is being expanded to incorporate advanced map‐making and online data geoprocessing capabilities that enable deriving new data and insights. However, the majority of published planning web tools are one‐off solutions, and a disconnect between research and practice is rendering many of these inaccessible or obsolete. Despite the significant progress made in advancing their provision in the last decade, there is a need for developing transferable interfaces that are maintained beyond project end dates, for them to effectively and consistently support planning processes.     

Interpreting map usage patterns using geovisual analytics and spatio-temporal clustering

Extracting meaningful information from the growing quantity of spatial data is a challenge. The issues are particularly evident with spatio-temporal data describing movement. Such data typically corresponds to movement of humans, animals and machines in the physical environment. This article considers a special form of movement data generated through human–computer interactions with online web maps. As a user interacts with a web map using a mouse as a pointing tool, invisible trajectories are generated. By examining the spatial features on the map where the mouse cursor visits, a user's interests and experience can be detected. To analyse this valuable information, we have developed a geovisual analysis tool which provides a rich insight into such user behaviour. The focus of this paper is on a clustering technique which we apply to mouse trajectories to group trajectories with similar behavioural properties. Our experiments reveal that it is possible to identify experienced and novice users of web mapping environments using an incremental clustering approach. The results can be used to provide personalised map interfaces to users and provide appropriate interventions for completing spatial tasks.     

Semantic typing of linked geoprocessing workflows

In Geographic Information Systems (GIS), geoprocessing workflows allow analysts to organize their methods on spatial data in complex chains. We propose a method for expressing workflows as linked data, and for semi-automatically enriching them with semantics on the level of their operations and datasets. Linked workflows can be easily published on the Web and queried for types of inputs, results, or tools. Thus, GIS analysts can reuse their workflows in a modular way, selecting, adapting, and recommending resources based on compatible semantic types. Our typing approach starts from minimal annotations of workflow operations with classes of GIS tools, and then propagates data types and implicit semantic structures through the workflow using an OWL typing scheme and SPARQL rules by backtracking over GIS operations. The method is implemented in Python and is evaluated on two real-world geoprocessing workflows, generated with Esri's ArcGIS. To illustrate the potential applications of our typing method, we formulate and execute competency questions over these workflows.     

A Task-Based Approach to User Interface Design for a Web-Based Hydrologic Information Systems

Hydroinformatics is a new and rapidly developing field that integrates knowledge and understanding of water resources with the latest developments in information technology to improve decision‐making in many critical applications. It encompasses methods for data capture, storage, processing, analysis and visualization, and the use of advanced modeling, simulation, optimization and knowledge‐based tools and systems infrastructure. Three types of hydrological data are most commonly used: flow rate in major rivers and streams, height of water in wells, and precipitation. To get a complete view of the state of water at a given point in space and time, one must analyze many different types of hydrological data together to derive information using an online GIS tool. To help use these disparate data sources more effectively and efficiently, we have built an online interface called the IJEDI WebCenter for Hydroinformatics using a task‐based approach. In this design, we first identify the tasks that users perform to study water‐related issues, then organize data for each task, and build task‐specific tools to present and analyze data and information. In a study involving both novices and experts in hydrology, we found that the both groups performed water‐related studies more effectively and efficiently than they would have without the WebCenter.     

基于SVG的多尺度电子地图共享模型研究

空间数据共享及互操作是伴随WebGIS技术发展的重要研究课题,如何有针对性地访问可共享的数据资源是一个值得迫切解决的问题。本文针对网上空间数据访问引入SVG技术对地图要素进行表达、组织,通过元数据信息内容为用户提供数据的诸如质量、加工过程、表达形式等＂预见性＂信息,综合运用SVG 1.0标准、VC＋＋,JavaScript等技术建立多尺度电子地图网上共享模型,该模型为用户提供了在因特网上共享空间数据的一种方法。     

Geoprocessing in Cloud Computing platforms – a comparative analysis

Abstract

The emergence of Cloud Computing technologies brings a new information infrastructure to users. Providing geoprocessing functions in Cloud Computing platforms can bring scalable, on-demand, and cost–effective geoprocessing services to geospatial users. This paper provides a comparative analysis of geoprocessing in Cloud Computing platforms – Microsoft Windows Azure and Google App Engine. The analysis compares differences in the data storage, architecture model, and development environment based on the experience to develop geoprocessing services in the two Cloud Computing platforms; emphasizes the importance of virtualization; recommends applications of hybrid geoprocessing Clouds, and suggests an interoperable solution on geoprocessing Cloud services. The comparison allows one to selectively utilize Cloud Computing platforms or hybrid Cloud pattern, once it is understood that the current development of geoprocessing Cloud services is restricted to specific Cloud Computing platforms with certain kinds of technologies. The performance evaluation is also performed over geoprocessing services deployed in public Cloud platforms. The tested services are developed using geoprocessing algorithms from different vendors, GeoSurf and Java Topology Suite. The evaluation results provide a valuable reference on providing elastic and cost-effective geoprocessing Cloud services.     

基于OGC规范的WebGIS开源平台研究

为了更好地实现地理空间数据和信息共享,在分析Web GIS平台、开源GIS、OGC规范以及Web服务的基础上,设计与开发一个基于OGC规范的Web GIS开源服务平台,并给出具体的实现步骤。该方案可高效地完成矢量数据入库、可视化浏览、空间对象属性查询、地图编辑和空间分析等任务,能够为集成应用提供标准的数据访问和获取接口。最后,将其应用到"太湖流域雨量水情信息服务"中,取得较好的效果。     

多种数据源地理信息处理的Internet GIS 方法

介绍了多种数据源获取、管理和地理信息处理的InternetGIS方法和用这种方法设计的InternetGIS的原理及功能特征。     

Enhancing the OGC WPS interface with GeoPipes support for real-time geoprocessing

ABSTRACT

Real-time geospatial information is used in various applications such as risk management or alerting services. Especially, the rise of new sensing technologies also increases the demand for processing the data in real time. Today’s spatial data infrastructures, however, do not meet the requirements for real-time geoprocessing. The OpenGIS^® Web Processing Service (WPS) is not designed to process real-time workflows. It has some major drawbacks in asynchronous processing and cannot handle (geo) data streams out of the box. In previous papers, we introduced the GeoPipes approach to share spatiotemporal data in real time. We implemented the concept extending the Message Queue and Telemetry Transport (MQTT) protocol by a spatial and temporal dimension, which we call GeoMQTT. In this paper, we demonstrate the integration of the GeoPipes idea in the WPS interface to expose standardized real-time geoprocessing services. The proof of the concept is illustrated in some exemplary real-time geo processes.     

Moving Code in Spatial Data Infrastructures – Web Service Based Deployment of Geoprocessing Algorithms

This article proposes a concept for offering complex geoprocessing functionality in service‐based Spatial Data Infrastructures (SDI). Today, geoprocessing in SDI is typically realized in a data driven manner. Applying the suggested “moving code” approach in a case study in the field of Spatial Decision Support proves its applicability. The proposed solution is analyzed and assessed in terms of gained efficiency, performance behavior and support for distributed development of geoprocessing functionality. In data and computation intensive SDI applications the deployment of moving code proves to be beneficial.     

Spatial cloud computing: how can the geospatial sciences use and help shape cloud computing?

Abstract

The geospatial sciences face grand information technology (IT) challenges in the twenty-first century: data intensity, computing intensity, concurrent access intensity and spatiotemporal intensity. These challenges require the readiness of a computing infrastructure that can: (1) better support discovery, access and utilization of data and data processing so as to relieve scientists and engineers of IT tasks and focus on scientific discoveries; (2) provide real-time IT resources to enable real-time applications, such as emergency response; (3) deal with access spikes; and (4) provide more reliable and scalable service for massive numbers of concurrent users to advance public knowledge. The emergence of cloud computing provides a potential solution with an elastic, on-demand computing platform to integrate – observation systems, parameter extracting algorithms, phenomena simulations, analytical visualization and decision support, and to provide social impact and user feedback – the essential elements of the geospatial sciences. We discuss the utilization of cloud computing to support the intensities of geospatial sciences by reporting from our investigations on how cloud computing could enable the geospatial sciences and how spatiotemporal principles, the kernel of the geospatial sciences, could be utilized to ensure the benefits of cloud computing. Four research examples are presented to analyze how to: (1) search, access and utilize geospatial data; (2) configure computing infrastructure to enable the computability of intensive simulation models; (3) disseminate and utilize research results for massive numbers of concurrent users; and (4) adopt spatiotemporal principles to support spatiotemporal intensive applications. The paper concludes with a discussion of opportunities and challenges for spatial cloud computing (SCC).     

User Assessment as Input for Useful Geospatial Representations within Mobile Location-Based Services

Mobile Location‐Based Services (mLBS) are an increasingly consumer‐based concept borne from, and continually driven by, technology‐centred development; as opposed to the needs of end users. Where users have been made a focus, the research generally concerns issues of overall system appearance, functionality, information content and interaction methods, with little emphasis on the component geospatial representations. This paper describes the initial stages of a research project aimed at filling this void through the application of a qualitative User‐Centred Design (UCD) methodology for optimising geospatial representations within mLBS applications, in order to support a selected user group: Australian ‘leisure‐based travellers’. Presented in this paper is an account of two UCD activities adopted for the research. The first, user profiling, served to define the target user population in terms of their technological, geospatial and travel experiences, using an online questionnaire. The second, user task analysis, involved in‐depth interviews with a subset of users in order to obtain a deeper understanding of the geospatial goals, tasks, needs and preferences within the population, as well as the range of user differences and variability in tasks present. An overall user assessment, through combined analysis of the two result sets, highlighted considerations for the ongoing research, including a set of specific implications for the design of alternative models for geospatial services, representations and interactions. The themes described in this paper represent an initial and necessary component of UCD, which has been largely overlooked in research relating to mLBS. Whilst the focus here is on a specific user group and context of use, it is envisaged that many of the concepts tested and ratified by the resulting models will be relevant to mLBS applications in general.