Limitations of Majority Agreement in Crowdsourced Image Interpretation

Crowdsourcing can efficiently complete tasks that are difficult to automate, but the quality of crowdsourced data is tricky to evaluate. Algorithms to grade volunteer work often assume that all tasks are similarly difficult, an assumption that is frequently false. We use a cropland identification game with over 2,600 participants and 165,000 unique tasks to investigate how best to evaluate the difficulty of crowdsourced tasks and to what extent this is possible based on volunteer responses alone. Inter‐volunteer agreement exceeded 90% for about 80% of the images and was negatively correlated with volunteer‐expressed uncertainty about image classification. A total of 343 relatively difficult images were independently classified as cropland, non‐cropland or impossible by two experts. The experts disagreed weakly (one said impossible while the other rated as cropland or non‐cropland) on 27% of the images, but disagreed strongly (cropland vs. non‐cropland) on only 7%. Inter‐volunteer disagreement increased significantly with inter‐expert disagreement. While volunteers agreed with expert classifications for most images, over 20% would have been mis‐categorized if only the volunteers’ majority vote was used. We end with a series of recommendations for managing the challenges posed by heterogeneous tasks in crowdsourcing campaigns.     

On predicting and improving the quality of Volunteer Geographic Information projects

Initiatives that rely upon the contributions of volunteers to reach a specific goal are growing more and more with the success of Web 2.0–interactive applications. Also scientific projects are testing and exploiting volunteers' collaboration, but the quality of information obtained with this approach is often puzzling. This paper offers a rich overview of many scientific projects where geographic contributions are committed to volunteers, to the aim of defining strategies to improve information quality. By describing real examples of Volunteer Geographic Information (VGI), the contribution establishes a categorization based on the characteristics of the information, tasks, and scopes of the projects. After a discussion on the relationships of categories and VGI quality, the paper analyses techniques to improve the quality of volunteered information according to the moment of its assessment (i.e., ex ante, ex post, or both with respect to information creation). The paper outlines the main limitations of the different approaches and indicates some guidelines for future developments.     

Accurate Attribute Mapping from Volunteered Geographic Information: Issues of Volunteer Quantity and Quality

Crowdsourcing is a popular means of acquiring data, but the use of such data is limited by concerns with its quality. This is evident within cartography and geographical sciences more generally, with the quality of volunteered geographic information (VGI) recognized as a major challenge to address if the full potential of citizen sensing in mapping applications is to be realized. Here, a means to characterize the quality of volunteers, based only on the data they contribute, was used to explore issues connected with the quantity and quality of volunteers for attribute mapping. The focus was on data in the form of annotations or class labels provided by volunteers who visually interpreted an attribute, land cover, from a series of satellite sensor images. A latent class model was found to be able to provide accurate characterisations of the quality of volunteers in terms of the accuracy of their labelling, irrespective of the number of cases that they labelled. The accuracy with which a volunteer could be characterized tended to increase with the number of volunteers contributing but was typically good at all but small numbers of volunteers. Moreover, the ability to characterize volunteers in terms of the quality of their labelling could be used constructively. For example, volunteers could be ranked in terms of quality which could then be used to select a sub-set as input to a subsequent mapping task. This was particularly important as an identified subset of volunteers could undertake a task more accurately than when part of a larger group of volunteers. The results highlight that both the quantity and quality of volunteers need consideration and that the use of VGI may be enhanced through information on the quality of the volunteers derived entirely from the data provided without any additional information.     

地理信息志愿者信誉度评估的版本相似度模型——以面目标为例

针对自发地理信息中存在大量恶意、虚假、低质量数据,提出了一种基于版本相似度的VGI志愿者信誉度计算模型。该模型将每个志愿者对某个地理空间目标的一次编辑结果定义为一个版本,当其他用户编辑该目标版本时,通过计算版本相似度来获得其他用户对该志愿者这个目标版本的支持度;然后通过计算其他贡献者对某志愿者所贡献的每个目标的支持度,对所有目标的支持度进行加权平均获得该志愿者的综合信誉度。其中版本相似度综合目标空间和属性相似性的主要因素进行加权计算。然后以面目标为例阐述了本文志愿者信誉度计算过程。为了验证本文信誉度计算模型的合理性,笔者采用德国柏林OpenStreetMap数据进行试验,试验表明通过本文模型计算获得的用户信誉度与其贡献的要素质量总体上正相关。     

Finding meaningful participation in volunteer geographic information and citizen science: a case comparison in environmental application

ABSTRACT

Volunteer geographic information (VGI) and citizen science (CS) have been used as a way to promote a form of democratization of knowledge. Crowdsourcing facilitates the massive collection of data through nonexpert volunteers. This study compares 57 VGI and CS projects through a cross-case comparison method to explore the level of involvement of participants, based on three main analytic dimensions (project rationale, actor interaction, and data flow) and 12 sub-variables. The primary goal of this research is to analyze the purpose and the role of technology in VGI and CS projects, using three matrices to explore emerging trade-offs. Finally, we synthesized the workflow of VGI and CS project and compared based on project purposes. At the end of this study, five main purposes were found among the selected projects, uncovering that data collection is the most common purpose. A general workflow can be conceptualized, but detail implementation will involve various trade-offs, especially between local involvement and large-area expert-oriented protocol implementations. By exploring diverse relationships among selected variables of analysis in VGI and CS projects, this study pretends to build a foundation for larger, global meta-analysis that can bring some transparency in the epistemic interactions between experts and nonexperts.     

面向道路网VGI数据的匹配与融合方法

自发地理信息(VGI)是一种新兴的地理数据采集方式,具有数据更新快、细节丰富、覆盖范围广等优势。利用VGI数据可以对道路网实现快速更新;但是由于VGI数据是非专业自发共享的,且数据采集时多使用非专业设备,所以存在数据质量不高的问题。大量VGI数据对同一地理要素的重复采集与融合处理则可改善数据的质量,文中以多人采集的道路网数据为例,结合矢量要素的匹配与融合理论,设计一种适用于道路网VGI数据的匹配与融合算法。首先在路段结点处建立缓冲区进行结点匹配,再根据路段距离相似度进行路段匹配,最后再利用Delaunay三角剖分融合算法对匹配后的同名路段进行融合。将匹配融合后的道路网与原始道路网VGI数据及Google影像图叠加对比分析,结果表明利用本文算法可有效地实现道路网VGI数据的匹配与融合。     

Using control data to determine the reliability of volunteered geographic information about land cover

There is much interest in using volunteered geographic information (VGI) in formal scientific analyses. This analysis uses VGI describing land cover that was captured using a web-based interface, linked to Google Earth. A number of control points, for which the land cover had been determined by experts allowed measures of the reliability of each volunteer in relation to each land cover class to be calculated. Geographically weighted kernels were used to estimate surfaces of volunteered land cover information accuracy and then to develop spatially distributed correspondences between the volunteer land cover class and land cover from 3 contemporary global datasets (GLC-2000, GlobCover and MODIS v.5). Specifically, a geographically weighted approach calculated local confusion matrices (correspondences) at each location in a central African study area and generated spatial distributions of user's, producer's, portmanteau, and partial portmanteau accuracies. These were used to evaluate the global datasets and to infer which of them was 'best’ at describing Tree cover at each location in the study area. The resulting maps show where specific global datasets are recommended for analyses requiring Tree cover information. The methods presented in this research suggest that some of the concerns about the quality of VGI can be addressed through careful data collection, the use of control points to evaluate volunteer performance and spatially explicit analyses. A research agenda for the use and analysis of VGI about land cover is outlined.     

Assessing the Accuracy of Volunteered Geographic Information arising from Multiple Contributors to an Internet Based Collaborative Project

The recent rise of neogeography and citizen sensing has increased the opportunities for the use of crowdsourcing as a means to acquire data to support geographical research. The value of the resulting volunteered geographic information is, however, often limited by concerns associated with its quality and the degree to which the contributing data sources may be trusted. Here, information on the quality of sources of volunteered geographic information was derived using a latent class analysis. The volunteered information was on land cover interpreted visually from satellite sensor images and the main focus was on the labeling of 299 sites by seven of the 65 volunteers who contributed to an Internet‐based collaborative project. Using the information on land cover acquired by the multiple volunteers it was shown that the relative, but not absolute, quality of the data from different volunteers could be characterized accurately. Additionally, class‐specific variations in the quality of the information provided by a single volunteer could be characterized by the analysis. The latent class analysis, therefore, was able to provide information on the quality of information provided on an inter‐ and intra‐volunteer basis.     

Assessing Resistance to Volunteered Geographic Information Reporting within Local Government

Using volunteered geographic information (VGI) as a cost‐saving environmental monitoring and public participation mechanism has been thought to be a proof of concept to assist environmental decision‐making with insufficient managerial resources. However, research literature also often found institutional resistance when implementing VGI as part of the decision‐making process. This research develops a conceptual framework as the foundation to conduct a cross‐case comparison for four empirical cases in recreation and invasive species monitoring, but general enough for other such case comparisons. The VGI Reporting Assessment Framework (VGI‐RAF) adopted aspects of the Decision Situation Assessment (DSA) framework as the foundation to address institutional flexibility, stakeholder dynamics, and technical considerations. Using a cross‐case comparison can accumulate deep observations from cases and increase validity of insights through comparing processes. Results show that eight aspects of the VGI‐RAF reveal the suitability of embedding voluntary information within existing institutional decision‐making processes. VGI‐RAF depicts considerations for: (1) existing participation arrangements, (2) institutional arrangements, (3) managers’ needs and stakeholder expectation, and (4) interaction mechanism and technical needs. The cross‐comparison results indicate that a public participation mechanism should be accompanied by VGI development to increase effectiveness. Institutional arrangements should match local government's information needs with participants’ expectations. Knowledge gaps between institutional information needs and volunteer knowledge should be filled through technology enhancements. For ease of implementation and to promote long‐term success, we recommend that volunteered information be connected with existing communication channels to minimize changes to existing information systems.     

VGI数据与地形图数据的自动融合研究

随着科学技术的不断发展，志愿者地理信息（volunteered geographic information，VGI）已经成为地理空间数据中最为重要的来源之一。为了充分利用志愿者地理信息，需要进行VGI与传统地形图数据的匹配与融合。开发了一种全新的数据自动匹配与融合算法，其目的是将ATKIS道路网数据（由德国联邦测绘局所采集的官方数据）与AOSD数据（由大量志愿者携带定位仪器进行户外徒步或骑行所获取的轨迹数据）匹配并融合起来，从而丰富传统地理信息数据的内容，并实现数据的增值。考虑到ATKIS数据与AOSD数据在空间表达上的差异很大，所开发的算法包括了道路要素的智能化分割、道路要素匹配、道路网数据融合以及融合后道路网内部要素间的匹配运算与数据集成等4个过程。大量实地数据的测试结果表明，该算法具有匹配成功率高、准确率高、运算速度快等优点。     

Improvement of a location-aware recommender system using volunteered geographic information

Abstract

Recommender systems (RS), as supportive tools, filter information from a massive amount of data based on the determined preferences. Most of the RS require information about the context of users such as their locations. In such cases, location-aware recommender systems (LARS) can be employed to suggest more personalized items to the users. The most current research projects on LARS focus on the development of algorithms, evaluation methods and applications. However, the role of up-to-date spatial databases in LARS is not a well-researched area. The up-to-date spatial information would potentially improve the accuracy of items which are recommended by LARS. Volunteered geographic information (VGI) could be a low-cost source of up-to-date spatial information for LARS. This article proposes an approach to enrich spatial databases of LARS by VGI. Since not all records of VGI are fitted for use in LARS, a mechanism is developed to identify useful information. Some VGI data sets refer to existing spatial data in the database while other VGI data sets are shared for the first time. Therefore, the proposed method assessed the quality of VGI with reference source (for VGI which is existed in the database) and VGI without reference source (for VGI which is shared for the first time). To demonstrate the feasibility of the proposed approach, a mobile application has been developed to recommend suitable restaurants to the users based on their geospatial locations. The evaluation of the method indicates that VGI can potentially enhance the functionality of the LARS in predicting the users’ interests.     

面向灾害管理的自发地理信息研究及应用

针对目前自发地理信息(VGI)应用于灾害管理的巨大潜力和需求,该文系统地介绍了VGI应用于灾害管理中所面临的机遇及数据可用性、数据质量、数据管理和法律议题等方面的挑战;在此基础上,针对VGI数据质量问题,结合我国灾害管理实际情况,提出了3种提高VGI数据质量的方法,即VGI数据贡献者分类与权限设计、政府引导下的灾害管理VGI数据采集框架以及VGI数据质量评估体系;最后,以开发的地震灾情信息管理原型系统为例,讨论VGI应用于灾害管理中的数据采集框架、数据模型等关键技术问题,具有一定的推广和应用价值。     

Obituary

Abstract

title/>

In the area of volunteered geographical information (VGI), the issue of spatial data quality is a clear challenge. The data that are contributed to VGI projects do not comply with standard spatial data quality assurance procedures, and the contributors operate without central coordination and strict data collection frameworks. However, similar to the area of open source software development, it is suggested that the data hold an intrinsic quality assurance measure through the analysis of the number of contributors who have worked on a given spatial unit. The assumption that as the number of contributors increases so does the quality is known as ‘Linus’ Law’ within the open source community. This paper describes three studies that were carried out to evaluate this hypothesis for VGI using the OpenStreetMap dataset, showing that this rule indeed applies in the case of positional accuracy.     

浅论自发地理信息的数据管理

分析了自发地理信息(volunteered geographic information,VGI)数据的来源、分类、特点与管理要求,探讨了VGI数据清理与质量控制,研究了以高效处理绘图查询与数据更新为目标的VGI图形数据管理问题,提出了动态线综合二叉树与缩放四叉树的设计思想,以解决VGI图形数据管理中的难点问题。     

Matching authority and VGI road networks using an extended node-based matching algorithm

The amount of volunteered geographic information (VGI) has increased over the past decade, and several studies have been conducted to evaluate the quality of VGI data. In this study, we evaluate the completeness of the road network in the VGI data set OpenStreetMap (OSM). The evaluation is based on an accurate and efficient network-matching algorithm. The study begins with a comparison of the two main strategies for network matching: segment-based and node-based matching. The comparison shows that the result quality is comparable for the two strategies, but the node-based result is considerably more computationally efficient. Therefore, we improve the accuracy of node-based algorithm by handling topological relationships and detecting patterns of complicated network components. Finally, we conduct a case study on the extended node-based algorithm in which we match OSM to the Swedish National Road Database (NVDB) in Scania, Sweden. The case study reveals that OSM has a completeness of 87% in the urban areas and 69% in the rural areas of Scania. The accuracy of the matching process is approximately 95%. The conclusion is that the extended node-based algorithm is sufficiently accurate and efficient for conducting surveys of the quality of OSM and other VGI road data sets in large geographic regions.     

Modeling aggregated expertise of user contributions to assess the credibility of OpenStreetMap features

The emergence of volunteered geographic information (VGI) during the past decade has fueled a wide range of research and applications. The assessment of VGI quality and fitness‐of‐use is still a challenge because of the non‐standardized and crowdsourced data collection process, as well as the unknown skill and motivation of the contributors. However, the frequent approach of assessing VGI quality against external data sources using ISO quality standard measures is problematic because of a frequent lack of available external (reference) data, and because for certain types of features, VGI might be more up‐to‐date than the reference data. Therefore, a VGI‐intrinsic measure of quality is highly desirable. This study proposes such an intrinsic measure of quality by developing the concept of aggregated expertise based on the characteristics of a feature's contributors. The article further operationalizes this concept and examines its feasibility through a case study using OpenStreetMap (OSM). The comparison of model OSM feature quality with information from a field survey demonstrates the successful implementation of this novel approach.     

A Review of Sampling Effects and Response Bias in Internet Participatory Mapping (PPGIS/PGIS/VGI)

Global interest in participatory mapping described as public participation GIS (PPGIS), participatory GIS (PGIS), and volunteered geographic information (VGI) continues to grow, but systematic study of spatial data quality and sampling effects is limited. This article provides a review and meta‐analysis of Internet‐based PPGIS studies conducted during the period 2006–2015 (n=26) to answer the following research questions: (1) How does mapping effort, as a proxy measure for spatial data quality, differ by sampling group? (2) Does the purpose and context of PPGIS influence mapping results? (3) What is the potential for mapping bias through sampling design? (4) Given the results, what should be the focus of future PPGIS research? Mapping effort was highest in sampling groups whose livelihoods were closely related to the purpose of the study, there was greater mapping effort in household sampling groups compared to volunteer groups, and participant domicile had strong effects on mapped results through spatial discounting. The use of online Internet panels provides higher response rates but lower spatial data quality. Future research should focus on increasing sampling response rates, assessing social trade‐offs using alternative spatial weighting schemes, and examining the capacity of the public to select land use alternatives as a complement to traditional expert‐driven planning systems.     

A building polygonal object matching method based on minimum bounding rectangle combinatorial optimization and relaxation labeling

Volunteered geographic information (VGI) is an emerging phenomenon where anyone can create geographic information and share it with others. Compared with traditional authoritative geospatial data, it has several advantages, such as enriched data, instant updates, and low cost. The object matching method is widely used in VGI quality assessment and data updates. However, VGI matching faces certain challenges, such as the levels of detail that vary from object to object, the uneven distribution of data quality, and the automated matching requirement. To resolve these problems, this article proposes a new matching method that effectively combines the advantages of minimum bounding rectangle combinatorial optimization (MBRCO) and relaxation labeling. The proposed method (1) avoids setting the similarity threshold and weights and does not require training samples. This process is realized based on contextual information and optimization. (2) It overcomes the disadvantage that the MBRCO algorithm cannot distinguish adjacent buildings with similar shapes. Our approach is experimentally validated using two publicly available spatial datasets: OpenStreetMap and AutoNavi map. The experimental studies show that the proposed automatic matching method outperforms all the threshold-based MBRCO methods and achieves high accuracy with a precision of 97.8% and a recall of 99.2%.     

VGI and crowdsourced data credibility analysis using spam email detection techniques

Volunteered geographic information (VGI) can be considered a subset of crowdsourced data (CSD) and its popularity has recently increased in a number of application areas. Disaster management is one of its key application areas in which the benefits of VGI and CSD are potentially very high. However, quality issues such as credibility, reliability and relevance are limiting many of the advantages of utilising CSD. Credibility issues arise as CSD come from a variety of heterogeneous sources including both professionals and untrained citizens. VGI and CSD are also highly unstructured and the quality and metadata are often undocumented. In the 2011 Australian floods, the general public and disaster management administrators used the Ushahidi Crowd-mapping platform to extensively communicate flood-related information including hazards, evacuations, emergency services, road closures and property damage. This study assessed the credibility of the Australian Broadcasting Corporation’s Ushahidi CrowdMap dataset using a Naïve Bayesian network approach based on models commonly used in spam email detection systems. The results of the study reveal that the spam email detection approach is potentially useful for CSD credibility detection with an accuracy of over 90% using a forced classification methodology.