Blog
About

134
views
0
recommends
+1 Recommend
1 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Conference Proceedings: found
      Is Open Access

      Measuring the Agreement Among Relevance Judges

      MIRA '99 (MIRA)

      MIRA '99

      14th - 16th April 1999

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          The importance of the issue of the agreement (or disagreement) between relevance judges is increasing, since new kinds of relevance judgment expression are being used (to the classical dichotomous one, various researches have added scalar, weighted, and orders of various kind) and new media are being introduced (it is far quicker to judge the relevance of an image than a text, and thus the human judgments can be obtained more easily).

          This paper presents a coherent account of the disagreement between relevance judges and groups of judges. Judgment expressions of different kinds, grouped into two categories, are taken into account. To the first category, score judgments, belong the more classical dichotomous, scalar, and weighted. To the second one, order judgments, belong total (or linear) and partial (or weak) orders, both with or without equality. A uniform notation for describing relevance judgments of each kind is proposed; some of the problems arising when one tries to operationally measure the disagreement between judges are described; a measure for the disagreement of two judges expressing two judgments of the same kind is proposed; the disagreement of a group of more than two judges is discussed; and, finally, some experimental activity inspired by this study is sketched.

          Related collections

          Most cited references 9

          • Record: found
          • Abstract: not found
          • Article: not found

          Measuring retrieval effectiveness based on user preference of documents

           Y.Y. Yao (1995)
            Bookmark
            • Record: found
            • Abstract: not found
            • Article: not found

            Relevance assessments and retrieval system evaluation

             G. Salton,  M.E. Lesk (1968)
              Bookmark
              • Record: found
              • Abstract: not found
              • Article: not found

              Variations in relevance judgments and the evaluation of retrieval performance

               Robert Burgin (1992)
                Bookmark

                Author and article information

                Contributors
                Conference
                April 1999
                April 1999
                : 1-13
                Affiliations
                Department of Mathematics and Computer Science

                University of Udine

                Via delle Scienze, 206—Loc. Rizzi

                I 33100 Udine, Italy

                ph.: +39 0432 558456, fax: +39 0432 558499

                WWW: http://www.dimi.uniud.it/mizzaro
                Article
                10.14236/ewic/MIRA1999.5
                © Stefano Mizzaro. Published by BCS Learning and Development Ltd. MIRA '99

                This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

                MIRA '99
                MIRA
                Glasgow, Scotland
                14th - 16th April 1999
                Electronic Workshops in Computing (eWiC)
                MIRA '99
                Product
                Product Information: 1477-9358BCS Learning & Development
                Self URI (journal page): https://ewic.bcs.org/
                Categories
                Electronic Workshops in Computing

                Comments

                Comment on this article