Assessing quality of volunteer crowdsourcing contributions: lessons from the Cropland Capture game

Volunteered geographic information (VGI) is the assembly of spatial information based on public input. While VGI has proliferated in recent years, assessing the quality of volunteer-contributed data has proven challenging, leading some to question the efficiency of such programs. In this paper, we compare several quality metrics for individual volunteers’ contributions. The data were the product of the ‘Cropland Capture’ game, in which several thousand volunteers assessed 165,000 images for the presence of cropland over the course of 6 months. We compared agreement between volunteer ratings and an image's majority classification with volunteer self-agreement on repeated images and expert evaluations. We also examined the impact of experience and learning on performance. Volunteer self-agreement was nearly always higher than agreement with majority classifications, and much greater than agreement with expert validations although these metrics were all positively correlated. Volunteer quality showed a broad trend toward improvement with experience, but the highest accuracies were achieved by a handful of moderately active contributors, not the most active volunteers. Our results emphasize the importance of a universal set of expert-validated tasks as a gold standard for evaluating VGI quality.     

Improvement of a location-aware recommender system using volunteered geographic information

Abstract

Recommender systems (RS), as supportive tools, filter information from a massive amount of data based on the determined preferences. Most of the RS require information about the context of users such as their locations. In such cases, location-aware recommender systems (LARS) can be employed to suggest more personalized items to the users. The most current research projects on LARS focus on the development of algorithms, evaluation methods and applications. However, the role of up-to-date spatial databases in LARS is not a well-researched area. The up-to-date spatial information would potentially improve the accuracy of items which are recommended by LARS. Volunteered geographic information (VGI) could be a low-cost source of up-to-date spatial information for LARS. This article proposes an approach to enrich spatial databases of LARS by VGI. Since not all records of VGI are fitted for use in LARS, a mechanism is developed to identify useful information. Some VGI data sets refer to existing spatial data in the database while other VGI data sets are shared for the first time. Therefore, the proposed method assessed the quality of VGI with reference source (for VGI which is existed in the database) and VGI without reference source (for VGI which is shared for the first time). To demonstrate the feasibility of the proposed approach, a mobile application has been developed to recommend suitable restaurants to the users based on their geospatial locations. The evaluation of the method indicates that VGI can potentially enhance the functionality of the LARS in predicting the users’ interests.     

Obituary

Abstract

title/>

In the area of volunteered geographical information (VGI), the issue of spatial data quality is a clear challenge. The data that are contributed to VGI projects do not comply with standard spatial data quality assurance procedures, and the contributors operate without central coordination and strict data collection frameworks. However, similar to the area of open source software development, it is suggested that the data hold an intrinsic quality assurance measure through the analysis of the number of contributors who have worked on a given spatial unit. The assumption that as the number of contributors increases so does the quality is known as ‘Linus’ Law’ within the open source community. This paper describes three studies that were carried out to evaluate this hypothesis for VGI using the OpenStreetMap dataset, showing that this rule indeed applies in the case of positional accuracy.     

面向灾害管理的自发地理信息研究及应用

针对目前自发地理信息(VGI)应用于灾害管理的巨大潜力和需求,该文系统地介绍了VGI应用于灾害管理中所面临的机遇及数据可用性、数据质量、数据管理和法律议题等方面的挑战;在此基础上,针对VGI数据质量问题,结合我国灾害管理实际情况,提出了3种提高VGI数据质量的方法,即VGI数据贡献者分类与权限设计、政府引导下的灾害管理VGI数据采集框架以及VGI数据质量评估体系;最后,以开发的地震灾情信息管理原型系统为例,讨论VGI应用于灾害管理中的数据采集框架、数据模型等关键技术问题,具有一定的推广和应用价值。     

Data trustworthiness and user reputation as indicators of VGI quality

ABSTRACT

Volunteered geographic information (VGI) has entered a phase where there are both a substantial amount of crowdsourced information available and a big interest in using it by organizations. But the issue of deciding the quality of VGI without resorting to a comparison with authoritative data remains an open challenge. This article first formulates the problem of quality assessment of VGI data. Then presents a model to measure trustworthiness of information and reputation of contributors by analyzing geometric, qualitative, and semantic aspects of edits over time. An implementation of the model is running on a small data-set for a preliminary empirical validation. The results indicate that the computed trustworthiness provides a valid approximation of VGI quality.     

Accurate Attribute Mapping from Volunteered Geographic Information: Issues of Volunteer Quantity and Quality

Crowdsourcing is a popular means of acquiring data, but the use of such data is limited by concerns with its quality. This is evident within cartography and geographical sciences more generally, with the quality of volunteered geographic information (VGI) recognized as a major challenge to address if the full potential of citizen sensing in mapping applications is to be realized. Here, a means to characterize the quality of volunteers, based only on the data they contribute, was used to explore issues connected with the quantity and quality of volunteers for attribute mapping. The focus was on data in the form of annotations or class labels provided by volunteers who visually interpreted an attribute, land cover, from a series of satellite sensor images. A latent class model was found to be able to provide accurate characterisations of the quality of volunteers in terms of the accuracy of their labelling, irrespective of the number of cases that they labelled. The accuracy with which a volunteer could be characterized tended to increase with the number of volunteers contributing but was typically good at all but small numbers of volunteers. Moreover, the ability to characterize volunteers in terms of the quality of their labelling could be used constructively. For example, volunteers could be ranked in terms of quality which could then be used to select a sub-set as input to a subsequent mapping task. This was particularly important as an identified subset of volunteers could undertake a task more accurately than when part of a larger group of volunteers. The results highlight that both the quantity and quality of volunteers need consideration and that the use of VGI may be enhanced through information on the quality of the volunteers derived entirely from the data provided without any additional information.     

面向道路网VGI数据的匹配与融合方法

自发地理信息(VGI)是一种新兴的地理数据采集方式,具有数据更新快、细节丰富、覆盖范围广等优势。利用VGI数据可以对道路网实现快速更新;但是由于VGI数据是非专业自发共享的,且数据采集时多使用非专业设备,所以存在数据质量不高的问题。大量VGI数据对同一地理要素的重复采集与融合处理则可改善数据的质量,文中以多人采集的道路网数据为例,结合矢量要素的匹配与融合理论,设计一种适用于道路网VGI数据的匹配与融合算法。首先在路段结点处建立缓冲区进行结点匹配,再根据路段距离相似度进行路段匹配,最后再利用Delaunay三角剖分融合算法对匹配后的同名路段进行融合。将匹配融合后的道路网与原始道路网VGI数据及Google影像图叠加对比分析,结果表明利用本文算法可有效地实现道路网VGI数据的匹配与融合。     

Finding meaningful participation in volunteer geographic information and citizen science: a case comparison in environmental application

ABSTRACT

Volunteer geographic information (VGI) and citizen science (CS) have been used as a way to promote a form of democratization of knowledge. Crowdsourcing facilitates the massive collection of data through nonexpert volunteers. This study compares 57 VGI and CS projects through a cross-case comparison method to explore the level of involvement of participants, based on three main analytic dimensions (project rationale, actor interaction, and data flow) and 12 sub-variables. The primary goal of this research is to analyze the purpose and the role of technology in VGI and CS projects, using three matrices to explore emerging trade-offs. Finally, we synthesized the workflow of VGI and CS project and compared based on project purposes. At the end of this study, five main purposes were found among the selected projects, uncovering that data collection is the most common purpose. A general workflow can be conceptualized, but detail implementation will involve various trade-offs, especially between local involvement and large-area expert-oriented protocol implementations. By exploring diverse relationships among selected variables of analysis in VGI and CS projects, this study pretends to build a foundation for larger, global meta-analysis that can bring some transparency in the epistemic interactions between experts and nonexperts.     

Towards Usability Engineering for Online Editors of Volunteered Geographic Information: A Perspective on Learnability

This article presents one of the first systematic usability investigations for a Volunteered Geographic Information (VGI) editor. This research is motivated by the fact that although VGI is now widely consumed, contribution rates are lagging considerably. Compared to traditional GIS interfaces, with complex interfaces resulting in high cognitive loads and barriers to participation, VGI tools and interfaces need to be easy to use and learn to encourage and facilitate contributions. This article develops a case study of OpenStreetMap, one of the most successful VGI projects, and its default editor, Potlatch2. Ten participants with no prior experience of VGI contribution, were instructed to contribute data to OSM in a structured exercise, while being monitored using an eye tracker and audio/video screen capture software. Each participant was asked to Think Aloud, i.e. describe what they were thinking and doing as they completed the tasks. The results highlight significant usability issues impacting learnability, especially from the perspective of a new contributor: hidden functionality, lack of user feedback between interactions and the inefficient and inconsistent placement of functionality and map controls. The facilitation of VGI contributions clearly depends on designing targeted interfaces, optimized to the needs of specific levels of contributors with defined goals and expectations.     

浅论自发地理信息的数据管理

分析了自发地理信息(volunteered geographic information,VGI)数据的来源、分类、特点与管理要求,探讨了VGI数据清理与质量控制,研究了以高效处理绘图查询与数据更新为目标的VGI图形数据管理问题,提出了动态线综合二叉树与缩放四叉树的设计思想,以解决VGI图形数据管理中的难点问题。     

众源地理信息研究现状与展望

众源地理信息作为新型地理信息模式,是指由互联网用户借助交互式平台,通过直接上传传感器数据或提供数字化劳动而贡献的地理信息数据。近20年来,众源地理信息热度上升,并受到越来越多研究者的关注,正在逐渐成为一种重要的地理信息数据来源。本文对近十几年来关于众源地理信息的中文和英文文献进行了系统性的分析与研究,力图展现出众源地理信息研究的发展现状,同时以这些研究发现的规律为基础,结合当前的国际形势,探讨了众源地理信息的发展机遇与挑战,同时对发展自主众源地理信息给出了建议。     

Matching authority and VGI road networks using an extended node-based matching algorithm

The amount of volunteered geographic information (VGI) has increased over the past decade, and several studies have been conducted to evaluate the quality of VGI data. In this study, we evaluate the completeness of the road network in the VGI data set OpenStreetMap (OSM). The evaluation is based on an accurate and efficient network-matching algorithm. The study begins with a comparison of the two main strategies for network matching: segment-based and node-based matching. The comparison shows that the result quality is comparable for the two strategies, but the node-based result is considerably more computationally efficient. Therefore, we improve the accuracy of node-based algorithm by handling topological relationships and detecting patterns of complicated network components. Finally, we conduct a case study on the extended node-based algorithm in which we match OSM to the Swedish National Road Database (NVDB) in Scania, Sweden. The case study reveals that OSM has a completeness of 87% in the urban areas and 69% in the rural areas of Scania. The accuracy of the matching process is approximately 95%. The conclusion is that the extended node-based algorithm is sufficiently accurate and efficient for conducting surveys of the quality of OSM and other VGI road data sets in large geographic regions.     

Modeling aggregated expertise of user contributions to assess the credibility of OpenStreetMap features

The emergence of volunteered geographic information (VGI) during the past decade has fueled a wide range of research and applications. The assessment of VGI quality and fitness‐of‐use is still a challenge because of the non‐standardized and crowdsourced data collection process, as well as the unknown skill and motivation of the contributors. However, the frequent approach of assessing VGI quality against external data sources using ISO quality standard measures is problematic because of a frequent lack of available external (reference) data, and because for certain types of features, VGI might be more up‐to‐date than the reference data. Therefore, a VGI‐intrinsic measure of quality is highly desirable. This study proposes such an intrinsic measure of quality by developing the concept of aggregated expertise based on the characteristics of a feature's contributors. The article further operationalizes this concept and examines its feasibility through a case study using OpenStreetMap (OSM). The comparison of model OSM feature quality with information from a field survey demonstrates the successful implementation of this novel approach.     

基于大众点评网的志愿者地理信息空间分异研究——以武汉市为例

在Web 2.0的驱动下,伴随着UGC(User Generated Content)的爆炸式增长,出现了一种新形式的地理信息——志愿者地理信息(Volunteered Geography Information,以下简称VGI)。其中点评类VGI提供了传统地理信息难以提供的商户环境、服务质量等信息,从而增强了地理信息对环境的表达能力,为消费者决策提供了重要参考。通过对大众点评网的点评信息进行分析,发现商户实体对点评信息的空间分布起着很大的制约作用,使得点评信息延续了商户不均衡的分布态势。但这种制约并不起决定性作用,点评信息仍然表现出一些区别于商户分布格局的特点。随后通过对不同价位餐馆的点评量进行分析,发现消费者更倾向对平均价格较高的餐馆进行点评,从而可能使得部分低价位的餐馆面临信息缺失的危机。     

OpenStreetMap data quality enrichment through awareness raising and collective action tools—experiences from a European project

ABSTRACT

Nowadays, several research projects show interest in employing volunteered geographic information (VGI) to improve their systems through using up-to-date and detailed data. The European project CAP4Access is one of the successful examples of such international-wide research projects that aims to improve the accessibility of people with restricted mobility using crowdsourced data. In this project, OpenStreetMap (OSM) is used to extend OpenRouteService, a well-known routing platform. However, a basic challenge that this project tackled was the incompleteness of OSM data with regards to certain information that is required for wheelchair accessibility (e.g. sidewalk information, kerb data, etc.). In this article, we present the results of initial assessment of sidewalk data in OSM at the beginning of the project as well as our approach in awareness raising and using tools for tagging accessibility data into OSM database for enriching the sidewalk data completeness. Several experiments have been carried out in different European cities, and discussion on the results of the experiments as well as the lessons learned are provided. The lessons learned provide recommendations that help in organizing better mapping party events in the future. We conclude by reporting on how and to what extent the OSM sidewalk data completeness in these study areas have benefited from the mapping parties by the end of the project.     

Trust as a proxy indicator for intrinsic quality of Volunteered Geographic Information in biodiversity monitoring programs

In this article, we present a fuzzy model for intrinsic quality assessment of Volunteered Geographic Information (VGI) on species occurrences obtained by Citizen Science (CS) biodiversity monitoring programs. The proposed VGI quality assurance approach evaluates the thematic and positional quality of the crowdsourced biodiversity observation in terms of the trustworthiness of the observation by combining three indicators of consistency with habitat, consistency with surroundings, and reputation of contributor, that characterize the geographical and social aspects of trust in VGI. To evaluate the performance and usability of the proposed approach for evaluating the trustworthiness of crowdsourced observations and detecting thematic and positional errors in crowdsourced observations, the developed approach was applied to the crowdsourced observations on Acer macrophyllum collected through the CS biodiversity monitoring projects of E-Flora BC and iNaturalist. The result of a conformity test at the optimal acceptance threshold (sensitivity = 0.99, specificity = 0.8, and Cohen’s kappa = 0.79), the achieved area under the curve (AUC) value (AUC = 0.98), and the results of the complementary investigation on the predictions of the proposed model indicated that the proposed fuzzy trust model exhibited promising predictive performance and was able to flag the majority of attribute and positional errors in the crowdsourced biodiversity observations.     

Mapping the food environment: The reliability of volunteered geographical information and institutional data sources in France

The obesity epidemic and inequalities in access to food are prompting increasing numbers of food environment studies, which rely on secondary data sources for mapping. This article assesses the reliability of the two main food outlet data sources in France: the volunteered geographical information (VGI) collaborative map OpenStreetMap (OSM) and the national business register Sirene. Their information on food outlets was assessed through ground-truthing in the city-region of Montpellier. Sensitivity, positive predictive value, and concordance were computed for each database. We analyzed the socio-spatial variability of these measures according to households' income level. The sensitivity of Sirene is good and that of OSM moderate, while the opposite holds for positive predictive value, and the concordance of both OSM and Sirene is fair. Sirene provides more reliable data on deprived neighborhoods and OSM on wealthy neighborhoods. Caution is recommended regarding the classifications on which they are based, the time required to update the institutional database, and socially influenced contributions to VGI.     

Quality Assessment of the French OpenStreetMap Dataset

The concept of Volunteered Geographic Information (VGI) has recently emerged from the new Web 2.0 technologies. The OpenStreetMap project is currently the most significant example of a system based on VGI. It aims at producing free vector geographic databases using contributions from Internet users. Spatial data quality becomes a key consideration in this context of freely downloadable geographic databases. This article studies the quality of French OpenStreetMap data. It extends the work of Haklay to France, provides a larger set of spatial data quality element assessments (i.e. geometric, attribute, semantic and temporal accuracy, logical consistency, completeness, lineage, and usage), and uses different methods of quality control. The outcome of the study raises questions such as the heterogeneity of processes, scales of production, and the compliance to standardized and accepted specifications. In order to improve data quality, a balance has to be struck between the contributors' freedom and their respect of specifications. The development of appropriate solutions to provide this balance is an important research issue in the domain of user‐generated content.     

Using control data to determine the reliability of volunteered geographic information about land cover

There is much interest in using volunteered geographic information (VGI) in formal scientific analyses. This analysis uses VGI describing land cover that was captured using a web-based interface, linked to Google Earth. A number of control points, for which the land cover had been determined by experts allowed measures of the reliability of each volunteer in relation to each land cover class to be calculated. Geographically weighted kernels were used to estimate surfaces of volunteered land cover information accuracy and then to develop spatially distributed correspondences between the volunteer land cover class and land cover from 3 contemporary global datasets (GLC-2000, GlobCover and MODIS v.5). Specifically, a geographically weighted approach calculated local confusion matrices (correspondences) at each location in a central African study area and generated spatial distributions of user's, producer's, portmanteau, and partial portmanteau accuracies. These were used to evaluate the global datasets and to infer which of them was 'best’ at describing Tree cover at each location in the study area. The resulting maps show where specific global datasets are recommended for analyses requiring Tree cover information. The methods presented in this research suggest that some of the concerns about the quality of VGI can be addressed through careful data collection, the use of control points to evaluate volunteer performance and spatially explicit analyses. A research agenda for the use and analysis of VGI about land cover is outlined.