• Record: found
  • Abstract: found
  • Article: found
Is Open Access

Cameras for Public Health Surveillance: A Methods Protocol for Crowdsourced Annotation of Point-of-Sale Photographs

Read this article at

      There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.


      BackgroundPhotographs are an effective way to collect detailed and objective information about the environment, particularly for public health surveillance. However, accurately and reliably annotating (ie, extracting information from) photographs remains difficult, a critical bottleneck inhibiting the use of photographs for systematic surveillance. The advent of distributed human computation (ie, crowdsourcing) platforms represents a veritable breakthrough, making it possible for the first time to accurately, quickly, and repeatedly annotate photos at relatively low cost.ObjectiveThis paper describes a methods protocol, using photographs from point-of-sale surveillance studies in the field of tobacco control to demonstrate the development and testing of custom-built tools that can greatly enhance the quality of crowdsourced annotation.MethodsEnhancing the quality of crowdsourced photo annotation requires a number of approaches and tools. The crowdsourced photo annotation process is greatly simplified by decomposing the overall process into smaller tasks, which improves accuracy and speed and enables adaptive processing, in which irrelevant data is filtered out and more difficult targets receive increased scrutiny. Additionally, zoom tools enable users to see details within photographs and crop tools highlight where within an image a specific object of interest is found, generating a set of photographs that answer specific questions. Beyond such tools, optimizing the number of raters (ie, crowd size) for accuracy and reliability is an important facet of crowdsourced photo annotation. This can be determined in a systematic manner based on the difficulty of the task and the desired level of accuracy, using receiver operating characteristic (ROC) analyses. Usability tests of the zoom and crop tool suggest that these tools significantly improve annotation accuracy. The tests asked raters to extract data from photographs, not for the purposes of assessing the quality of that data, but rather to assess the usefulness of the tool. The proportion of individuals accurately identifying the presence of a specific advertisement was higher when provided with pictures of the product’s logo and an example of the ad, and even higher when also provided the zoom tool (χ2 2=155.7, P<.001). Similarly, when provided cropped images, a significantly greater proportion of respondents accurately identified the presence of cigarette product ads (χ2 1=75.14, P<.001), as well as reported being able to read prices (χ2 2=227.6, P<.001). Comparing the results of crowdsourced photo-only assessments to traditional field survey data, an excellent level of correspondence was found, with area under the ROC curves produced by sensitivity analyses averaging over 0.95, requiring on average 10 to 15 crowdsourced raters to achieve values of over 0.90.ResultsFurther testing and improvement of these tools and processes is currently underway. This includes conducting systematic evaluations that crowdsource photograph annotation and methodically assess the quality of raters’ work.ConclusionsOverall, the combination of crowdsourcing technologies with tiered data flow and tools that enhance annotation quality represents a breakthrough solution to the problem of photograph annotation, vastly expanding opportunities for the use of photographs rich in public health and other data on a scale previously unimaginable.

      Related collections

      Most cited references 25

      • Record: found
      • Abstract: not found
      • Article: not found

      Crowdsourcing systems on the World-Wide Web

        • Record: found
        • Abstract: found
        • Article: not found

        Photovoice: concept, methodology, and use for participatory needs assessment.

         Jean Burris,  C. Wang (1997)
        Photovoice is a process by which people can identify, represent, and enhance their community through a specific photographic technique. As a practice based in the production of knowledge, photovoice has three main goals: (1) to enable people to record and reflect their community's strengths and concerns, (2) to promote critical dialogue and knowledge about important issues through large and small group discussion of photographs, and (3) to reach policymakers. Applying photovoice to public health promotion, the authors describe the methodology and analyze its value for participatory needs assessment. They discuss the development of the photovoice concept, advantages and disadvantages, key elements, participatory analysis, materials and resources, and implications for practice.
          • Record: found
          • Abstract: not found
          • Article: not found



            Author and article information

            1Department of Research and Evaluation Legacy Washington, DCUnited States
            2Survos Washington, DCUnited States
            3Steven A Schroeder Institute for Tobacco Research and Policy Studies Legacy Washington, DCUnited States
            4Department of Health, Behavior, and Society Johns Hopkins Bloomberg School of Public Health Baltimore, MDUnited States
            5Department of Oncology Lombardi Comprehensive Cancer Center Georgetown University Medical Center Washington, DCUnited States
            Author notes
            Corresponding Author: Vinu Ilakkuvan vilakkuvan@
            JMIR Res Protoc
            JMIR Res Protoc
            JMIR Research Protocols
            JMIR Publications Inc. (Toronto, Canada )
            Apr-Jun 2014
            09 April 2014
            : 3
            : 2
            (Editor), (Reviewer), (Reviewer),
            ©Vinu Ilakkuvan, Michael Tacelosky, Keith C. Ivey, Jennifer L Pearson, Jennifer Cantrell, Donna M Vallone, David B. Abrams, Thomas R Kirchner. Originally published in JMIR Research Protocols (, 09.04.2014.

            This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.



            Comment on this article