Blog
About

0
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: not found
      • Article: not found

      Reply to the comment of Bertocchi et al.

      ,

      Scientometrics

      Springer Nature

      Read this article at

      ScienceOpenPublisher
      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Related collections

          Most cited references 4

          • Record: found
          • Abstract: not found
          • Article: not found

          Bibliometric evaluation vs. informed peer review: Evidence from Italy

            Bookmark
            • Record: found
            • Abstract: found
            • Article: found
            Is Open Access

            Do they agree? Bibliometric evaluation vs informed peer review in the Italian research assessment exercise

             ,   (2016)
            During the Italian research assessment exercise, the national agency ANVUR performed an experiment to assess agreement between grades attributed to journal articles by informed peer review (IR) and by bibliometrics. A sample of articles was evaluated by using both methods and agreement was analyzed by weighted Cohen's kappas. ANVUR presented results as indicating an overall 'good' or 'more than adequate' agreement. This paper re-examines the experiment results according to the available statistical guidelines for interpreting kappa values, by showing that the degree of agreement, always in the range 0.09-0.42 has to be interpreted, for all research fields, as unacceptable, poor or, in a few cases, as, at most, fair. The only notable exception, confirmed also by a statistical meta-analysis, was a moderate agreement for economics and statistics (Area 13) and its sub-fields. We show that the experiment protocol adopted in Area 13 was substantially modified with respect to all the other research fields, to the point that results for economics and statistics have to be considered as fatally flawed. The evidence of a poor agreement supports the conclusion that IR and bibliometrics do not produce similar results, and that the adoption of both methods in the Italian research assessment possibly introduced systematic and unknown biases in its final results. The conclusion reached by ANVUR must be reversed: the available evidence does not justify at all the joint use of IR and bibliometrics within the same research assessment exercise.
              Bookmark
              • Record: found
              • Abstract: not found
              • Article: not found

              Comment to: Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise

                Bookmark

                Author and article information

                Journal
                Scientometrics
                Scientometrics
                Springer Nature
                0138-9130
                1588-2861
                September 2016
                July 2016
                : 108
                : 3
                : 1675-1684
                Article
                10.1007/s11192-016-2055-6
                © 2016

                Comments

                Comment on this article