Building social graphs from images through expert–based crowdsourcing

The extraction of semantic information from multimedia content represents a challenging problem. Despite the continuous refinement of automatic tools, the quality and completeness of the results is not always satisfactory. To overcome this limitation, the vision of the CUbRIK project is to provide a multimedia search and exploration platform that seamlessly integrate human tasks and algorithms. In this paper, as a concrete example, we illustrate the design of amultimedia content processing pipeline that aims at extracting evidence of social relationships from the analysis of a photo collection covering the main events and people that shaped the history of Europe after World War II. We discuss the issues faced by general-purpose crowdsourcing and automatic face detection/recognition algorithms in determining the identities of people portrayed in the photo collection. Hence, we illustrate the design of a system that tackles the uncertainty of the results produced by face detection/recognition with expert-based crowdsourcing.