Assessing quality of volunteer crowdsourcing contributions: lessons from the Cropland Capture game

Volunteered geographic information (VGI) is the assembly of spatial information based on public input. While VGI has proliferated in recent years, assessing the quality of volunteer-contributed data has proven challenging, leading some to question the efficiency of such programs. In this paper, we compare several quality metrics for individual volunteers’ contributions. The data were the product of the ‘Cropland Capture’ game, in which several thousand volunteers assessed 165,000 images for the presence of cropland over the course of 6 months. We compared agreement between volunteer ratings and an image's majority classification with volunteer self-agreement on repeated images and expert evaluations. We also examined the impact of experience and learning on performance. Volunteer self-agreement was nearly always higher than agreement with majority classifications, and much greater than agreement with expert validations although these metrics were all positively correlated. Volunteer quality showed a broad trend toward improvement with experience, but the highest accuracies were achieved by a handful of moderately active contributors, not the most active volunteers. Our results emphasize the importance of a universal set of expert-validated tasks as a gold standard for evaluating VGI quality.     

The Scale of VGI in Map Production: A Perspective on European National Mapping Agencies

The perspective of European National Mapping Agencies (NMA) on the role of citizen sensing in map production was explored. The NMAs varied greatly in their engagement with the community generating volunteered geographic information (VGI) and in their future plans. From an assessment of NMA standard practices, it was evident that much VGI was acquired with a positional accuracy that, while less than that typically acquired by NMAs, actually exceeded the requirements of the nominal data capture scale used by most NMAs. Opportunities for VGI use in map revision and updating were evident, especially for agencies that use a continuous rather than cyclical updating policy. Some NMAs had also developed systems to engage with citizen sensors and examples are discussed. Only rarely was VGI used to collect data on features beyond the standard set used by the NMAs. The potential role of citizen sensing and so its current scale of use by NMAs is limited by a series of concerns, notably relating to issues of data quality, the nature and motivation of the contributors, legal issues, the sustainability of data source, and employment fears of NMA staff. Possible priorities for future research and development are identified to help ensure that the potential of VGI in mapping is realized.     

Improvement of a location-aware recommender system using volunteered geographic information

Abstract

Recommender systems (RS), as supportive tools, filter information from a massive amount of data based on the determined preferences. Most of the RS require information about the context of users such as their locations. In such cases, location-aware recommender systems (LARS) can be employed to suggest more personalized items to the users. The most current research projects on LARS focus on the development of algorithms, evaluation methods and applications. However, the role of up-to-date spatial databases in LARS is not a well-researched area. The up-to-date spatial information would potentially improve the accuracy of items which are recommended by LARS. Volunteered geographic information (VGI) could be a low-cost source of up-to-date spatial information for LARS. This article proposes an approach to enrich spatial databases of LARS by VGI. Since not all records of VGI are fitted for use in LARS, a mechanism is developed to identify useful information. Some VGI data sets refer to existing spatial data in the database while other VGI data sets are shared for the first time. Therefore, the proposed method assessed the quality of VGI with reference source (for VGI which is existed in the database) and VGI without reference source (for VGI which is shared for the first time). To demonstrate the feasibility of the proposed approach, a mobile application has been developed to recommend suitable restaurants to the users based on their geospatial locations. The evaluation of the method indicates that VGI can potentially enhance the functionality of the LARS in predicting the users’ interests.     

On predicting and improving the quality of Volunteer Geographic Information projects

Initiatives that rely upon the contributions of volunteers to reach a specific goal are growing more and more with the success of Web 2.0–interactive applications. Also scientific projects are testing and exploiting volunteers' collaboration, but the quality of information obtained with this approach is often puzzling. This paper offers a rich overview of many scientific projects where geographic contributions are committed to volunteers, to the aim of defining strategies to improve information quality. By describing real examples of Volunteer Geographic Information (VGI), the contribution establishes a categorization based on the characteristics of the information, tasks, and scopes of the projects. After a discussion on the relationships of categories and VGI quality, the paper analyses techniques to improve the quality of volunteered information according to the moment of its assessment (i.e., ex ante, ex post, or both with respect to information creation). The paper outlines the main limitations of the different approaches and indicates some guidelines for future developments.     

Matching authority and VGI road networks using an extended node-based matching algorithm

The amount of volunteered geographic information (VGI) has increased over the past decade, and several studies have been conducted to evaluate the quality of VGI data. In this study, we evaluate the completeness of the road network in the VGI data set OpenStreetMap (OSM). The evaluation is based on an accurate and efficient network-matching algorithm. The study begins with a comparison of the two main strategies for network matching: segment-based and node-based matching. The comparison shows that the result quality is comparable for the two strategies, but the node-based result is considerably more computationally efficient. Therefore, we improve the accuracy of node-based algorithm by handling topological relationships and detecting patterns of complicated network components. Finally, we conduct a case study on the extended node-based algorithm in which we match OSM to the Swedish National Road Database (NVDB) in Scania, Sweden. The case study reveals that OSM has a completeness of 87% in the urban areas and 69% in the rural areas of Scania. The accuracy of the matching process is approximately 95%. The conclusion is that the extended node-based algorithm is sufficiently accurate and efficient for conducting surveys of the quality of OSM and other VGI road data sets in large geographic regions.     

Assessing the Accuracy of Volunteered Geographic Information arising from Multiple Contributors to an Internet Based Collaborative Project

The recent rise of neogeography and citizen sensing has increased the opportunities for the use of crowdsourcing as a means to acquire data to support geographical research. The value of the resulting volunteered geographic information is, however, often limited by concerns associated with its quality and the degree to which the contributing data sources may be trusted. Here, information on the quality of sources of volunteered geographic information was derived using a latent class analysis. The volunteered information was on land cover interpreted visually from satellite sensor images and the main focus was on the labeling of 299 sites by seven of the 65 volunteers who contributed to an Internet‐based collaborative project. Using the information on land cover acquired by the multiple volunteers it was shown that the relative, but not absolute, quality of the data from different volunteers could be characterized accurately. Additionally, class‐specific variations in the quality of the information provided by a single volunteer could be characterized by the analysis. The latent class analysis, therefore, was able to provide information on the quality of information provided on an inter‐ and intra‐volunteer basis.     

面向道路网VGI数据的匹配与融合方法

自发地理信息(VGI)是一种新兴的地理数据采集方式,具有数据更新快、细节丰富、覆盖范围广等优势。利用VGI数据可以对道路网实现快速更新;但是由于VGI数据是非专业自发共享的,且数据采集时多使用非专业设备,所以存在数据质量不高的问题。大量VGI数据对同一地理要素的重复采集与融合处理则可改善数据的质量,文中以多人采集的道路网数据为例,结合矢量要素的匹配与融合理论,设计一种适用于道路网VGI数据的匹配与融合算法。首先在路段结点处建立缓冲区进行结点匹配,再根据路段距离相似度进行路段匹配,最后再利用Delaunay三角剖分融合算法对匹配后的同名路段进行融合。将匹配融合后的道路网与原始道路网VGI数据及Google影像图叠加对比分析,结果表明利用本文算法可有效地实现道路网VGI数据的匹配与融合。     

Volunteered Geographic Information: A Bicycling Enthusiast Perspective

Mapping technologies have made considerable strides in recent decades. Global positioning systems (GPS), remote sensing satellites, Web-based mapping services, and geographic information systems (GIS) have facilitated the collection, distribution, analysis, and ultimately interaction with geospatial information. In particular, portable GPS have altered how individuals participate in mapping. Individuals can use GPS to collect tracings of their personal interactions with the environment. These interactions can then be uploaded to one of many available Web-based mapping services. Once uploaded, the geospatial data can be mapped and shared among the broader community of users. Such volunteered geographic information (VGI) exemplifies the conceptualization of an individual collecting, mapping, and sharing personal geographic information. This paper focuses on challenges surrounding VGI. To help place these challenges in a broader context, specialized Web services and GPS technologies developed for the bicycling community will serve as examples of the current status and future prospects of VGI.     

Understanding Design with VGI using an Information Relevance Framework

Volunteered Geographic Information (VGI) has the potential to offer benefits to end‐users over and above those of Professional Geographic Information (PGI). A multi‐methods approach, consisting of participatory observation, focus groups and diary studies, was used to study how VGI and PGI were actually used by a target user group. A theoretical framework of information relevance was used to understand the attributes of information that were most important in relation to the characteristics of the users' activity and their community. The key finding was that the discussion amongst GIS designers should not be whether to choose VGI or PGI as the information data set, but to consider which combination of VGI and PGI relating to different geographic features and task characteristics would best meet the users' needs. VGI is likely to be most relevant to the user when a geographic feature is dynamic rather than static in nature, and can also provide a level of detail that is unlikely to arise through PGI. These findings have implications for how different forms of information are most effective for different usage situations. Above all, a case is presented for the implementation of User Centred Design (UCD) principles when integrating VGI and PGI together in a single mashup‐based product.     

Quality Assessment of the French OpenStreetMap Dataset

The concept of Volunteered Geographic Information (VGI) has recently emerged from the new Web 2.0 technologies. The OpenStreetMap project is currently the most significant example of a system based on VGI. It aims at producing free vector geographic databases using contributions from Internet users. Spatial data quality becomes a key consideration in this context of freely downloadable geographic databases. This article studies the quality of French OpenStreetMap data. It extends the work of Haklay to France, provides a larger set of spatial data quality element assessments (i.e. geometric, attribute, semantic and temporal accuracy, logical consistency, completeness, lineage, and usage), and uses different methods of quality control. The outcome of the study raises questions such as the heterogeneity of processes, scales of production, and the compliance to standardized and accepted specifications. In order to improve data quality, a balance has to be struck between the contributors' freedom and their respect of specifications. The development of appropriate solutions to provide this balance is an important research issue in the domain of user‐generated content.     

Trust as a proxy indicator for intrinsic quality of Volunteered Geographic Information in biodiversity monitoring programs

In this article, we present a fuzzy model for intrinsic quality assessment of Volunteered Geographic Information (VGI) on species occurrences obtained by Citizen Science (CS) biodiversity monitoring programs. The proposed VGI quality assurance approach evaluates the thematic and positional quality of the crowdsourced biodiversity observation in terms of the trustworthiness of the observation by combining three indicators of consistency with habitat, consistency with surroundings, and reputation of contributor, that characterize the geographical and social aspects of trust in VGI. To evaluate the performance and usability of the proposed approach for evaluating the trustworthiness of crowdsourced observations and detecting thematic and positional errors in crowdsourced observations, the developed approach was applied to the crowdsourced observations on Acer macrophyllum collected through the CS biodiversity monitoring projects of E-Flora BC and iNaturalist. The result of a conformity test at the optimal acceptance threshold (sensitivity = 0.99, specificity = 0.8, and Cohen’s kappa = 0.79), the achieved area under the curve (AUC) value (AUC = 0.98), and the results of the complementary investigation on the predictions of the proposed model indicated that the proposed fuzzy trust model exhibited promising predictive performance and was able to flag the majority of attribute and positional errors in the crowdsourced biodiversity observations.     

面向灾害管理的自发地理信息研究及应用

针对目前自发地理信息(VGI)应用于灾害管理的巨大潜力和需求,该文系统地介绍了VGI应用于灾害管理中所面临的机遇及数据可用性、数据质量、数据管理和法律议题等方面的挑战;在此基础上,针对VGI数据质量问题,结合我国灾害管理实际情况,提出了3种提高VGI数据质量的方法,即VGI数据贡献者分类与权限设计、政府引导下的灾害管理VGI数据采集框架以及VGI数据质量评估体系;最后,以开发的地震灾情信息管理原型系统为例,讨论VGI应用于灾害管理中的数据采集框架、数据模型等关键技术问题,具有一定的推广和应用价值。     

Obituary

Abstract

title/>

In the area of volunteered geographical information (VGI), the issue of spatial data quality is a clear challenge. The data that are contributed to VGI projects do not comply with standard spatial data quality assurance procedures, and the contributors operate without central coordination and strict data collection frameworks. However, similar to the area of open source software development, it is suggested that the data hold an intrinsic quality assurance measure through the analysis of the number of contributors who have worked on a given spatial unit. The assumption that as the number of contributors increases so does the quality is known as ‘Linus’ Law’ within the open source community. This paper describes three studies that were carried out to evaluate this hypothesis for VGI using the OpenStreetMap dataset, showing that this rule indeed applies in the case of positional accuracy.     

Integrating VGI and 2D hydraulic models into a data assimilation framework for real time flood forecasting and mapping

ABSTRACT

Crowdsourced data can effectively observe environmental and urban ecosystem processes. The use of data produced by untrained people into flood forecasting models may effectively allow Early Warning Systems (EWS) to better perform while support decision-making to reduce the fatalities and economic losses due to inundation hazard. In this work, we develop a Data Assimilation (DA) method integrating Volunteered Geographic Information (VGI) and a 2D hydraulic model and we test its performances. The proposed framework seeks to extend the capabilities and performances of standard DA works, based on the use of traditional in situ sensors, by assimilating VGI while managing and taking into account the uncertainties related to the quality, and the location and timing of the entire set of observational data. The November 2012 flood in the Italian Tiber River basin was selected as the case study. Results show improvements of the model in terms of uncertainty with a significant persistence of the model updating after the integration of the VGI, even in the case of use of few-selected observations gathered from social media. This will encourage further research in the use of VGI for EWS considering the exponential increase of quality and quantity of smartphone and social media user worldwide.     

An assessment of a collaborative mapping approach for exploring land use patterns for several European metropolises

Until recently, land surveys and digital interpretation of remotely sensed imagery have been used to generate land use inventories. These techniques however, are often cumbersome and costly, allocating large amounts of technical and temporal costs. The technological advances of web 2.0 have brought a wide array of technological achievements, stimulating the participatory role in collaborative and crowd sourced mapping products. This has been fostered by GPS-enabled devices, and accessible tools that enable visual interpretation of high resolution satellite images/air photos provided in collaborative mapping projects. Such technologies offer an integrative approach to geography by means of promoting public participation and allowing accurate assessment and classification of land use as well as geographical features. OpenStreetMap (OSM) has supported the evolution of such techniques, contributing to the existence of a large inventory of spatial land use information. This paper explores the introduction of this novel participatory phenomenon for land use classification in Europe's metropolitan regions. We adopt a positivistic approach to assess comparatively the accuracy of these contributions of OSM for land use classifications in seven large European metropolitan regions. Thematic accuracy and degree of completeness of OSM data was compared to available Global Monitoring for Environment and Security Urban Atlas (GMESUA) datasets for the chosen metropolises. We further extend our findings of land use within a novel framework for geography, justifying that volunteered geographic information (VGI) sources are of great benefit for land use mapping depending on location and degree of VGI dynamism and offer a great alternative to traditional mapping techniques for metropolitan regions throughout Europe. Evaluation of several land use types at the local level suggests that a number of OSM classes (such as anthropogenic land use, agricultural and some natural environment classes) are viable alternatives for land use classification. These classes are highly accurate and can be integrated into planning decisions for stakeholders and policymakers.     

Data trustworthiness and user reputation as indicators of VGI quality

ABSTRACT

Volunteered geographic information (VGI) has entered a phase where there are both a substantial amount of crowdsourced information available and a big interest in using it by organizations. But the issue of deciding the quality of VGI without resorting to a comparison with authoritative data remains an open challenge. This article first formulates the problem of quality assessment of VGI data. Then presents a model to measure trustworthiness of information and reputation of contributors by analyzing geometric, qualitative, and semantic aspects of edits over time. An implementation of the model is running on a small data-set for a preliminary empirical validation. The results indicate that the computed trustworthiness provides a valid approximation of VGI quality.     

VGI数据与地形图数据的自动融合研究

随着科学技术的不断发展,志愿者地理信息（volunteered geographic information,VGI）已经成为地理空间数据中最为重要的来源之一。为了充分利用志愿者地理信息,需要进行VGI与传统地形图数据的匹配与融合。开发了一种全新的数据自动匹配与融合算法,其目的是将ATKIS道路网数据（由德国联邦测绘局所采集的官方数据）与AOSD数据（由大量志愿者携带定位仪器进行户外徒步或骑行所获取的轨迹数据）匹配并融合起来,从而丰富传统地理信息数据的内容,并实现数据的增值。考虑到ATKIS数据与AOSD数据在空间表达上的差异很大,所开发的算法包括了道路要素的智能化分割、道路要素匹配、道路网数据融合以及融合后道路网内部要素间的匹配运算与数据集成等4个过程。大量实地数据的测试结果表明,该算法具有匹配成功率高、准确率高、运算速度快等优点。     

Modeling aggregated expertise of user contributions to assess the credibility of OpenStreetMap features

The emergence of volunteered geographic information (VGI) during the past decade has fueled a wide range of research and applications. The assessment of VGI quality and fitness‐of‐use is still a challenge because of the non‐standardized and crowdsourced data collection process, as well as the unknown skill and motivation of the contributors. However, the frequent approach of assessing VGI quality against external data sources using ISO quality standard measures is problematic because of a frequent lack of available external (reference) data, and because for certain types of features, VGI might be more up‐to‐date than the reference data. Therefore, a VGI‐intrinsic measure of quality is highly desirable. This study proposes such an intrinsic measure of quality by developing the concept of aggregated expertise based on the characteristics of a feature's contributors. The article further operationalizes this concept and examines its feasibility through a case study using OpenStreetMap (OSM). The comparison of model OSM feature quality with information from a field survey demonstrates the successful implementation of this novel approach.     

Applying VGI to collaborative research in the humanities: the case of ARTL@S

ABSTRACT

Using Artl@s as an example of a project that relies on volunteered geographic information (VGI), this article examines the specific challenges that exist, beyond those frequently discussed in general VGI systems (e.g., participants’ motivation and data quality control) in regard to sharing research data in humanities: (1) most data from the humanities is qualitative and collected from multiple data sources which are often inconsistent and unmappable; (2) data is usually interconnected with multiple relationships among different tables which creates challenges for both mapping and query functionality; (3) data is both geographical and historical. Consequently addresses that no longer exist have to be geolocated and visualized on historical basemaps and spaces must be represented diachronically; (4) the design of web map application needs to balance both sophisticated research requirements and a user-friendly interface; (5) finally contributors expect their data to be cited or acknowledged when used in other studies and users need metadata and citation information in order to reuse and repurpose datasets.

In this article, we discuss how Artl@s, a project which developed a georeferenced historical database of exhibition catalogues, addresses these challenges. Artl@s provides a case study for VGI adoption by digital humanities scholars for research data sharing, as it offers features, such as flexible batch data contribution, interrelated spatial query, automatic geolocalization of historical addresses, and data citation mechanisms.