+1 Recommend
0 collections
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Using and Reporting the Delphi Method for Selecting Healthcare Quality Indicators: A Systematic Review


      Read this article at

          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.



          Delphi technique is a structured process commonly used to developed healthcare quality indicators, but there is a little recommendation for researchers who wish to use it. This study aimed 1) to describe reporting of the Delphi method to develop quality indicators, 2) to discuss specific methodological skills for quality indicators selection 3) to give guidance about this practice.

          Methodology and Main Finding

          Three electronic data bases were searched over a 30 years period (1978–2009). All articles that used the Delphi method to select quality indicators were identified. A standardized data extraction form was developed. Four domains (questionnaire preparation, expert panel, progress of the survey and Delphi results) were assessed. Of 80 included studies, quality of reporting varied significantly between items (9% for year's number of experience of the experts to 98% for the type of Delphi used). Reporting of methodological aspects needed to evaluate the reliability of the survey was insufficient: only 39% (31/80) of studies reported response rates for all rounds, 60% (48/80) that feedback was given between rounds, 77% (62/80) the method used to achieve consensus and 57% (48/80) listed quality indicators selected at the end of the survey. A modified Delphi procedure was used in 49/78 (63%) with a physical meeting of the panel members, usually between Delphi rounds. Median number of panel members was 17(Q1:11; Q3:31). In 40/70 (57%) studies, the panel included multiple stakeholders, who were healthcare professionals in 95% (38/40) of cases. Among 75 studies describing criteria to select quality indicators, 28 (37%) used validity and 17(23%) feasibility.


          The use and reporting of the Delphi method for quality indicators selection need to be improved. We provide some guidance to the investigators to improve the using and reporting of the method in future surveys.

          Related collections

          Most cited references55

          • Record: found
          • Abstract: found
          • Article: not found

          Consulting the oracle: ten lessons from using the Delphi technique in nursing research.

          The aim of this paper was to provide insight into the Delphi technique by outlining our personal experiences during its use over a 10-year period in a variety of applications. As a means of achieving consensus on an issue, the Delphi research method has become widely used in healthcare research generally and nursing research in particular. The literature on this technique is expanding, mainly addressing what it is and how it should be used. However, there is still much confusion and uncertainty surrounding it, particularly about issues such as modifications, consensus, anonymity, definition of experts, how 'experts' are selected and how non-respondents are pursued. This issues that arise when planning and carrying out a Delphi study include the definition of consensus; the issue of anonymity vs. quasi-anonymity for participants; how to estimate the time needed to collect the data, analyse each 'round', feed back results to participants, and gain their responses to this feedback; how to define and select the 'experts' who will be asked to participate; how to enhance response rates; and how many 'rounds' to conduct. Many challenges and questions are raised when using the Delphi technique, but there is no doubt that it is an important method for achieving consensus on issues where none previously existed. Researchers need to adapt the method to suit their particular study.
            • Record: found
            • Abstract: found
            • Article: not found

            Defining and classifying clinical indicators for quality improvement.

            J Mainz (2003)
            This paper provides a brief review of definitions, characteristics, and categories of clinical indicators for quality improvement in health care. Clinical indicators assess particular health structures, processes, and outcomes. They can be rate- or mean-based, providing a quantitative basis for quality improvement, or sentinel, identifying incidents of care that trigger further investigation. They can assess aspects of the structure, process, or outcome of health care. Furthermore, indicators can be generic measures that are relevant for most patients or disease-specific, expressing the quality of care for patients with specific diagnoses. Monitoring health care quality is impossible without the use of clinical indicators. They create the basis for quality improvement and prioritization in the health care system. To ensure that reliable and valid clinical indicators are used, they must be designed, defined, and implemented with scientific rigour.
              • Record: found
              • Abstract: not found
              • Article: not found

              Managing Delphi Surveys Using Nonparametric Statistical Techniques


                Author and article information

                Role: Editor
                PLoS One
                PLoS ONE
                Public Library of Science (San Francisco, USA )
                9 June 2011
                : 6
                : 6
                : e20476
                [1 ]AP-HP, Hôpital Robert Debré, Unité d'Epidémiologie Clinique, Paris, France
                [2 ]Inserm, CIE 5, Paris, France
                [3 ]AP-HP, Hôpital Robert Debré, Pôle Gynécologie et périnatalité, Hospitalisation Gynécologie – Obstétrique, Paris, France
                [4 ]Université Paris 7 Denis Diderot, UFR de Médecine, Paris, France
                University of British Columbia, Canada
                Author notes

                Study concept and design: RB CA HA. Had full access to all the study data and takes responsibility for the integrity of the data: RB. Statistical analysis: RB. Analysis and interpretation of data: RB CA HA. Drafting of the manuscript: RB. Critical revision of the manuscript: RB CA ML HA OS. Approved the final version of the paper: RB HA ML OS CA. Guarantor of the study: RB.

                Boulkedid et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
                : 22 March 2011
                : 26 April 2011
                Page count
                Pages: 9
                Research Article
                Clinical Research Design
                Reporting Guidelines
                Survey Research
                Systematic Reviews
                Survey Methods
                Non-Clinical Medicine
                Communication in Health Care
                Health Care Quality
                Public Health



                Comment on this article