28
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Resitting a high-stakes postgraduate medical examination on multiple occasions: nonlinear multilevel modelling of performance in the MRCP(UK) examinations

      research-article
      1 , 2 , , 1
      BMC Medicine
      BioMed Central

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Background

          Failure rates in postgraduate examinations are often high and many candidates therefore retake examinations on several or even many times. Little, however, is known about how candidates perform across those multiple attempts. A key theoretical question to be resolved is whether candidates pass at a resit because they have got better, having acquired more knowledge or skills, or whether they have got lucky, chance helping them to get over the pass mark. In the UK, the issue of resits has become of particular interest since the General Medical Council issued a consultation and is considering limiting the number of attempts candidates may make at examinations.

          Methods

          Since 1999 the examination for Membership of the Royal Colleges of Physicians of the United Kingdom (MRCP(UK)) has imposed no limit on the number of attempts candidates can make at its Part 1, Part2 or PACES (Clinical) examination. The present study examined the performance of candidates on the examinations from 2002/2003 to 2010, during which time the examination structure has been stable. Data were available for 70,856 attempts at Part 1 by 39,335 candidates, 37,654 attempts at Part 2 by 23,637 candidates and 40,303 attempts at PACES by 21,270 candidates, with the maximum number of attempts being 26, 21 and 14, respectively. The results were analyzed using multilevel modelling, fitting negative exponential growth curves to individual candidate performance.

          Results

          The number of candidates taking the assessment falls exponentially at each attempt. Performance improves across attempts, with evidence in the Part 1 examination that candidates are still improving up to the tenth attempt, with a similar improvement up to the fourth attempt in Part 2 and the sixth attempt at PACES. Random effects modelling shows that candidates begin at a starting level, with performance increasing by a smaller amount at each attempt, with evidence of a maximum, asymptotic level for candidates, and candidates showing variation in starting level, rate of improvement and maximum level. Modelling longitudinal performance across the three diets (sittings) shows that the starting level at Part 1 predicts starting level at both Part 2 and PACES, and the rate of improvement at Part 1 also predicts the starting level at Part 2 and PACES.

          Conclusion

          Candidates continue to show evidence of true improvement in performance up to at least the tenth attempt at MRCP(UK) Part 1, although there are individual differences in the starting level, the rate of improvement and the maximum level that can be achieved. Such findings provide little support for arguments that candidates should only be allowed a fixed number of attempts at an examination. However, unlimited numbers of attempts are also difficult to justify because of the inevitable and ever increasing role that luck must play with increasing numbers of resits, so that the issue of multiple attempts might be better addressed by tackling the difficult question of how a pass mark should increase with each attempt at an exam.

          Related collections

          Most cited references20

          • Record: found
          • Abstract: found
          • Article: not found

          The Effect of Different Forms of Centering in Hierarchical Linear Models

          Multilevel models are becoming increasingly used in applied educational social and economic research for the analysis of hierarchically nested data. In these random coefficient regression models the parameters are allowed to differ over the groups in which the observations are nested. For computational ease in deriving parameter estimates, predictors are often centered around the mean. In nested or grouped data, the option of centering around the grand mean is extended with an option to center within groups or contexts. Both are statistically sound ways to improve parameter estimation. In this article we study the effects of these two different ways of centering, in comparison to the use of raw scores, on the parameter estimates in random coefficient models. The conclusion is that centering around the group mean amounts to fitting a different model from that obtained by centering around the grand mean or by using raw scores. The choice between the two options for centering can only be made on a theoretical basis. Based on this study, we conclude that centering rules valid for simple models, such as the fixed coefficients regression model. are no longer applicable to more complicated models, such as the random coefficient model. We think researchers should be made aware of the consequences of the choice of particular centering options.
            Bookmark
            • Record: found
            • Abstract: found
            • Article: not found

            The role of coding time in estimating and interpreting growth curve models.

            The coding of time in growth curve models has important implications for the interpretation of the resulting model that are sometimes not transparent. The authors develop a general framework that includes predictors of growth curve components to illustrate how parameter estimates and their standard errors are exactly determined as a function of receding time in growth curve models. Linear and quadratic growth model examples are provided, and the interpretation of estimates given a particular coding of time is illustrated. How and why the precision and statistical power of predictors of lower order growth curve components changes over time is illustrated and discussed. Recommendations include coding time to produce readily interpretable estimates and graphing lower order effects across time with appropriate confidence intervals to help illustrate and understand the growth process. (c) 2004 APA, all rights reserved
              Bookmark
              • Record: found
              • Abstract: not found
              • Article: not found

              Autoregressive Latent Trajectory (ALT) Models A Synthesis of Two Traditions

                Bookmark

                Author and article information

                Journal
                BMC Med
                BMC Med
                BMC Medicine
                BioMed Central
                1741-7015
                2012
                14 June 2012
                : 10
                : 60
                Affiliations
                [1 ]Academic Centre for Medical Education, Division of Medical Education, University College London, Gower Street, London, WC1E 6BT, UK
                [2 ]Division of Psychology and Language Sciences, University College London, Gower Street, London, WC1E 6BT, UK
                Article
                1741-7015-10-60
                10.1186/1741-7015-10-60
                3394208
                22697599
                0e7afcc5-1ff9-4949-9d2c-cd1619342d78
                Copyright ©2012 McManus and Ludka; licensee BioMed Central Ltd.

                This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

                History
                : 30 November 2011
                : 14 June 2012
                Categories
                Research Article

                Medicine
                Medicine

                Comments

                Comment on this article