Blog
About

  • Record: found
  • Abstract: found
  • Article: found
Is Open Access

Interrater reliability: the kappa statistic

Biochemia Medica

Croatian Society of Medical Biochemistry and Laboratory Medicine

kappa, reliability, rater, interrater

Read this article at

Bookmark
      There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

      Abstract

      The kappa statistic is frequently used to test interrater reliability. The importance of rater reliability lies in the fact that it represents the extent to which the data collected in the study are correct representations of the variables measured. Measurement of the extent to which data collectors (raters) assign the same score to the same variable is called interrater reliability. While there have been a variety of methods to measure interrater reliability, traditionally it was measured as percent agreement, calculated as the number of agreement scores divided by the total number of scores. In 1960, Jacob Cohen critiqued use of percent agreement due to its inability to account for chance agreement. He introduced the Cohen’s kappa, developed to account for the possibility that raters actually guess on at least some variables due to uncertainty. Like most correlation statistics, the kappa can range from −1 to +1. While the kappa is one of the most commonly used statistics to test interrater reliability, it has limitations. Judgments about what level of kappa should be acceptable for health research are questioned. Cohen’s suggested interpretation may be too lenient for health related studies because it implies that a score as low as 0.41 might be acceptable. Kappa and percent agreement are compared, and levels for both kappa and percent agreement that should be demanded in healthcare studies are suggested.

      Related collections

      Most cited references 13

      • Record: found
      • Abstract: not found
      • Article: not found

      A Coefficient of Agreement for Nominal Scales

        Bookmark
        • Record: found
        • Abstract: not found
        • Article: not found

        The Kappa Statistic: A Second Look

          Bookmark
          • Record: found
          • Abstract: found
          • Article: not found

          Meta-analysis of Pap test accuracy.

          A literature search identified 62 studies published by August 1992 comparing Papanicolaou (Pap) test results with histology. Critical appraisal revealed that 82% of these had potential for verification bias and that only 37% stated that cytology and histology were independently assessed. Estimates of sensitivity and specificity ranged from 11 to 99% and 14 to 97%, respectively, and were highly negatively correlated (r = -0.63). Meta-analysis was used to combine data from 59 studies to estimate the accuracy of the Pap test using a summary receiver operating characteristic curve and to examine the effect of study quality. The summary receiver operating characteristic curve suggests that the Pap test may be unable to achieve concurrently high sensitivity and specificity. For example, specificity in the 90-95% range corresponds to sensitivity in the 20-35% range. Pap test accuracy was not associated with reported study characteristics or dimensions of quality. Future primary studies should pay more attention to methodologic standards for the conduct and reporting of diagnostic test evaluations.
            Bookmark

            Author and article information

            Affiliations
            Department of Nursing, National University, Aero Court, San Diego, California
            Author notes
            Corresponding author: mchugh8688@ 123456gmail.com
            Journal
            Biochem Med (Zagreb)
            Biochem Med (Zagreb)
            Biochemia Medica
            Biochemia Medica
            Croatian Society of Medical Biochemistry and Laboratory Medicine
            1330-0962
            1846-7482
            15 October 2012
            October 2012
            : 22
            : 3
            : 276-282
            23092060
            3900052
            biochem_med-22-3-276-4
            ©Copyright by Croatian Society of Medical Biochemistry and Laboratory Medicine

            This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License ( http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

            Categories
            Lessons in Biostatistics

            interrater, rater, reliability, kappa

            Comments

            Comment on this article