41
views
0
recommends
+1 Recommend
0 collections
    3
    shares
      • Record: found
      • Abstract: found
      • Article: not found

      The Work-ability Support Scale: Evaluation of Scoring Accuracy and Rater Reliability

      research-article

      Read this article at

      ScienceOpenPublisherPMC
      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Purpose The Work-ability Support Scale (WSS) is a new tool designed to assess vocational ability and support needs following onset of acquired disability, to assist decision-making in vocational rehabilitation. In this article, we report an iterative process of development through evaluation of inter- and intra-rater reliability and scoring accuracy, using vignettes. The impact of different methodological approaches to analysis of reliability is highlighted. Methods Following preliminary evaluation using case-histories, six occupational therapists scored vignettes, first individually and then together in two teams. Scoring was repeated blind after 1 month. Scoring accuracy was tested against agreed ‘reference standard’ vignette scores using intraclass correlation coefficients (ICCs) for total scores and linear-weighted kappas (kw) for individual items. Item-by-item inter- and intra-rater reliability was evaluated for both individual and team scores, using two different statistical methods. Results ICCs for scoring accuracy ranged from 0.95 (95 % CI 0.78–0.98) to 0.96 (0.89–0.99) for Part A, and from 0.78 (95 % CI 0.67–0.85) to 0.84 (0.69–0.92) for Part B. Item by item analysis of scoring accuracy, inter- and intra-rater reliability all showed ‘substantial’ to ‘almost perfect’ agreement (kw ≥ 0.60) for all Part-A and 8/12 Part-B items, although multi-rater kappa (Fleiss) produced more conservative results (mK = 0.34–0.79). Team rating produced marginal improvements for Part-A but not Part-B. Four problematic contextual items were identified, leading to adjustment of the scoring manual. Conclusion This vignette-based study demonstrates generally acceptable levels of scoring accuracy and reliability for the WSS. Further testing in real-life situations is now warranted.

          Related collections

          Most cited references7

          • Record: found
          • Abstract: found
          • Article: not found

          Exploring the use of vignettes: from validity to trustworthiness.

          In this article one of the authors presents her experience of using vignettes in an action research case study, with particular emphasis on their trustworthiness. The research was carried out with a group of health care professionals who were working together to improve preoperative education for patients awaiting a hip replacement in one National Health Service Trust in the United Kingdom. The authors describe the exploration of three types of vignette--napshots, portraits, and composites--as a means of collecting, analyzing, and representing data. They provide an example of each and briefly present their evaluative use by the health care professionals. The vignettes were constructed to provide one account of the truth that was representative of events. Within this representation was the writer, researcher, and reader feature. The vignettes stimulated reflection and analysis within the action research cycle and thus contributed to the improvement of practice.
            Bookmark
            • Record: found
            • Abstract: found
            • Article: not found

            The UK FIM+FAM: development and evaluation. Functional Assessment Measure.

            The aim of this study was to develop and evaluate the UK version of the Functional Assessment Measure (UK FIM+FAM). Before and after evaluation of inter-rater reliability. Ten 'troublesome' items in the original FIM+FAM were identified as being particularly difficult to score reliably. Revised decision trees were developed and tested for these items over a period of two years to produce the UK FIM+FAM. A multicentre study was undertaken to test agreement between raters for the UK FIM+FAM, in comparison with the original version, by assessing accuracy of scoring for standard vignettes. Baseline testing of the original FIM+FAM was undertaken at the start of the project in 1995. Thirty-seven rehabilitation professionals (11 teams) each rated the same three sets of vignettes - first individually and then as part of a multidisciplinary team. Accuracy was assessed in relation to the agreed 'correct' answers, both for individual and for team scores. Following development of the UK version, the same vignettes (with minimal adaptation to place them in context with the revised version) were rated by 28 individuals (nine teams). Taking all 30 items together, the accuracy for scoring by individuals improved from 74.7% to 77.1% with the UK version, and team scores improved from 83.7% to 86.5%. When the 10 troublesome items were taken together, accuracy of individual raters improved from 69.5% to 74.6% with the UK version (p <0.001), and team scores improved from 78.2% to 84.1% (N/S). For both versions, team ratings were significantly more accurate than individual ratings (p <0.01). Kappa values for team scoring of the troublesome items were all above 0.65 in the UK version. The UK FIM+FAM compares favourably with the original version for scoring accuracy and ease of use, and is now sufficiently well-developed for wider dissemination.
              Bookmark
              • Record: found
              • Abstract: not found
              • Article: not found

              Confidence Intervals for the Kappa Statistic

                Bookmark

                Author and article information

                Contributors
                +44-208-8692800 , +44-208-8692803 , Lynne.turner-stokes@dial.pipex.com
                Journal
                J Occup Rehabil
                J Occup Rehabil
                Journal of Occupational Rehabilitation
                Springer US (Boston )
                1053-0487
                1573-3688
                12 December 2013
                12 December 2013
                2014
                : 24
                : 511-524
                Affiliations
                [ ]Department of Palliative Care, Policy and Rehabilitation, School of Medicine, King’s College London, London, UK
                [ ]Regional Rehabilitation Unit, Northwick Park Hospital, Watford Road, Harrow, London, Middlesex HA1 3UJ UK
                [ ]Person Centred Research Centre, Health and Rehabilitation Research Institute, AUT University, Auckland, New Zealand
                [ ]School of Health Sciences, University of Canterbury, Christchurch, New Zealand
                [ ]School of Public Health and Psychosocial Studies, AUT University, Auckland, New Zealand
                [ ]School of Nursing and Midwifery, University of Queensland, Brisbane, Australia
                Article
                9486
                10.1007/s10926-013-9486-1
                4118042
                24338285
                33670973-9a8d-4182-af93-f3f9e64ed20d
                © The Author(s) 2013

                Open AccessThis article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.

                History
                Categories
                Article
                Custom metadata
                © Springer Science+Business Media New York 2014

                Occupational & Environmental medicine
                rehabilitation,vocational,needs assessment,reliability and validity

                Comments

                Comment on this article