3
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      A Full Probabilistic Model for Yes/No Type Crowdsourcing in Multi-Class Classification

      Preprint
      , ,

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Crowdsourcing has become widely used in supervised scenarios where training sets are scarce and hard to obtain. Most crowdsourcing models in literature assume labelers can provide answers for full questions. In classification contexts, full questions mean that a labeler is asked to discern among all the possible classes. Unfortunately, that discernment is not always easy in realistic scenarios. Labelers may not be experts in differentiating all the classes. In this work, we provide a full probabilistic model for a shorter type of queries. Our shorter queries just required a 'yes' or 'no' response. Our model estimates a joint posterior distribution of matrices related to the labelers confusions and the posterior probability of the class of every object. We develop an approximate inference approach using Monte Carlo Sampling and Black Box Variational Inference, where we provide the derivation of the necessary gradients. We build two realistic crowdsourcing scenarios to test our model. The first scenario queries for irregular astronomical time-series. The second scenario relies on animal's image classification. Results show that we can achieve comparable results with full query crowdsourcing. Furthermore, we show that modeling the labelers failures plays an important role in estimating the true classes. Finally, we provide the community with two real datasets obtained from our crowdsourcing experiments. All our code is publicly available (Available at: revealed as soon as the paper gets published.)

          Related collections

          Most cited references1

          • Record: found
          • Abstract: not found
          • Article: not found

          Natural Gradient Works Efficiently in Learning

            Bookmark

            Author and article information

            Journal
            02 January 2019
            Article
            1901.00397
            722f613b-192d-48f9-b094-374eb89e247d

            http://arxiv.org/licenses/nonexclusive-distrib/1.0/

            History
            Custom metadata
            SIAM International Conference on Data Mining (SDM19), 9 official pages, 5 supplementary pages
            cs.LG stat.ML

            Machine learning,Artificial intelligence
            Machine learning, Artificial intelligence

            Comments

            Comment on this article