Guideline appraisal with AGREE II: Systematic review of the current evidence on how users handle the 2 overall assessments

There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

Abstract

Introduction

The Appraisal of Guidelines for Research & Evaluation (AGREE) II instrument is the most commonly used guideline appraisal tool. It includes 23 appraisal criteria (items) organized within 6 domains and 2 overall assessments (1. overall guideline quality; 2. recommendation for use). The aim of this systematic review was twofold. Firstly, to investigate how often AGREE II users conduct the 2 overall assessments. Secondly, to investigate the influence of the 6 domain scores on each of the 2 overall assessments.

Materials and methods

A systematic bibliographic search was conducted for publications reporting guideline appraisals with AGREE II. The impact of the 6 domain scores on the overall assessment of guideline quality was examined using a multiple linear regression model. Their impact on the recommendation for use (possible answers: “yes”, “yes, with modifications”, “no”) was examined using a multinomial regression model.

Results

118 relevant publications including 1453 guidelines were identified. 77.1% of the publications reported results for at least one overall assessment, but only 32.2% reported results for both overall assessments. The results of the regression analyses showed a statistically significant influence of all domains on overall guideline quality, with Domain 3 (rigour of development) having the strongest influence. For the recommendation for use, the results showed a significant influence of Domains 3 to 5 (“yes” vs. “no”) and Domains 3 and 5 (“yes, with modifications” vs. “no”).

Conclusions

The 2 overall assessments of AGREE II are underreported by guideline assessors. Domains 3 and 5 have the strongest influence on the results of the 2 overall assessments, while the other domains have a varying influence. Within a normative approach, our findings could be used as guidance for weighting individual domains in AGREE II to make the overall assessments more objective. Alternatively, a stronger content analysis of the individual domains could clarify their importance in terms of guideline quality. Moreover, AGREE II should require users to transparently present how they conducted the assessments.

Related collections

Most cited references 51

Record: found
Abstract: not found
Article: not found

Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing

Yoav Benjamini, Yosef Hochberg (1995)

0 comments Cited 23876 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

Effectiveness and efficiency of guideline dissemination and implementation strategies.

Geraint E R Thomas, Gavin C. Donaldson, G MacLennan … (2004)

To undertake a systematic review of the effectiveness and costs of different guideline development, dissemination and implementation strategies. To estimate the resource implications of these strategies. To develop a framework for deciding when it is efficient to develop and introduce clinical guidelines. MEDLINE, Healthstar, Cochrane Controlled Trial Register, EMBASE, SIGLE and the specialised register of the Cochrane Effective Practice and Organisation of Care (EPOC) group. Single estimates of dichotomous process variables were derived for each study comparison based upon the primary end-point or the median measure across several reported end-points. Separate analyses were undertaken for comparisons of different types of intervention. The study also explored whether the effects of multifaceted interventions increased with the number of intervention components. Studies reporting economic data were also critically appraised. A survey to estimate the feasibility and likely resource requirements of guideline dissemination and implementation strategies in UK settings was carried out with key informants from primary and secondary care. In total, 235 studies reporting 309 comparisons met the inclusion criteria; of these 73% of comparisons evaluated multifaceted interventions, although the maximum number of replications of a specific multifaceted intervention was 11 comparisons. Overall, the majority of comparisons reporting dichotomous process data observed improvements in care; however, there was considerable variation in the observed effects both within and across interventions. Commonly evaluated single interventions were reminders, dissemination of educational materials, and audit and feedback. There were 23 comparisons of multifaceted interventions involving educational outreach. The majority of interventions observed modest to moderate improvements in care. No relationship was found between the number of component interventions and the effects of multifaceted interventions. Only 29.4% of comparisons reported any economic data. The majority of studies only reported costs of treatment; only 25 studies reported data on the costs of guideline development or guideline dissemination and implementation. The majority of studies used process measures for their primary end-point, despite the fact that only three guidelines were explicitly evidence based (and may not have been efficient). Respondents to the key informant survey rarely identified existing budgets to support guideline dissemination and implementation strategies. In general, the respondents thought that only dissemination of educational materials and short (lunchtime) educational meetings were generally feasible within current resources. There is an imperfect evidence base to support decisions about which guideline dissemination and implementation strategies are likely to be efficient under different circumstances. Decision makers need to use considerable judgement about how best to use the limited resources they have for clinical governance and related activities to maximise population benefits. They need to consider the potential clinical areas for clinical effectiveness activities, the likely benefits and costs required to introduce guidelines and the likely benefits and costs as a result of any changes in provider behaviour. Further research is required to: develop and validate a coherent theoretical framework of health professional and organisational behaviour and behaviour change to inform better the choice of interventions in research and service settings, and to estimate the efficiency of dissemination and implementation strategies in the presence of different barriers and effect modifiers.

0 comments Cited 211 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

Systematic review and meta-analysis of practice facilitation within primary care settings.

N Baskerville, Clare Liddy, William Hogg (2012)

This study was a systematic review with a quantitative synthesis of the literature examining the overall effect size of practice facilitation and possible moderating factors. The primary outcome was the change in evidence-based practice behavior calculated as a standardized mean difference. In this systematic review, we searched 4 electronic databases and the reference lists of published literature reviews to find practice facilitation studies that identified evidence-based guideline implementation within primary care practices as the outcome. We included randomized and nonrandomized controlled trials and prospective cohort studies published from 1966 to December 2010 in English language only peer-reviewed journals. Reviews of each study were conducted and assessed for quality; data were abstracted, and standardized mean difference estimates and 95% confidence intervals (CIs) were calculated using a random-effects model. Publication bias, influence, subgroup, and meta-regression analyses were also conducted. Twenty-three studies contributed to the analysis for a total of 1,398 participating practices: 697 practice facilitation intervention and 701 control group practices. The degree of variability between studies was consistent with what would be expected to occur by chance alone (I2 = 20%). An overall effect size of 0.56 (95% CI, 0.43-0.68) favored practice facilitation (z = 8.76; P <.001), and publication bias was evident. Primary care practices are 2.76 (95% CI, 2.18-3.43) times more likely to adopt evidence-based guidelines through practice facilitation. Meta-regression analysis indicated that tailoring (P = .05), the intensity of the intervention (P = .03), and the number of intervention practices per facilitator (P = .004) modified evidence-based guideline adoption. Practice facilitation has a moderately robust effect on evidence-based guideline adoption within primary care. Implementation fidelity factors, such as tailoring, the number of practices per facilitator, and the intensity of the intervention, have important resource implications.

0 comments Cited 210 times – based on 0 reviews      Review now

Bookmark

All references

Author and article information

Contributors

Hong-Liang Zhang: Role: Editor

Journal

Journal ID (nlm-ta): PLoS One

Journal ID (iso-abbrev): PLoS ONE

Journal ID (publisher-id): plos

Journal ID (pmc): plosone

Title: PLoS ONE

Publisher: Public Library of Science (San Francisco, CA USA )

ISSN (Electronic): 1932-6203

Publication date (Electronic): 30 March 2017

Publication date Collection: 2017

Volume: 12

Issue: 3

Electronic Location Identifier: e0174831

Affiliations

[1 ]Institute for Quality and Efficiency in Health Care (IQWiG), Cologne, Germany

[2 ]Institute for Research in Operative Medicine (IFOM), University of Witten/Herdecke, Campus Cologne, Cologne, Germany

[3 ]Brandenburg Medical School – Theodor Fontane Neuruppin, Germany & University of Witten/Herdecke, Witten/Herdecke, Germany

[4 ]Medical Advisory Service of the German Social Health Insurance (MDS), Essen, Germany

National Natural Science Foundation of China, CHINA

Author notes

Competing Interests: The authors have declared that no competing interests exist.

Conceptualization: WHE ME EN US.
Data curation: WHE US ACB UL.
Formal analysis: WHE ACB.
Methodology: WHE ACB EN ME.
Project administration: WHE.
Resources: WHE ACB ME.
Supervision: EN ME.
Visualization: WHE ACB.
Writing – original draft: WHE US EN ACB UL ME.

* E-mail: wiebke.hoffmann-esser@ 123456iqwig.de

Author information

Wiebke Hoffmann-Eßer http://orcid.org/0000-0003-4262-3794

Article

Publisher ID: PONE-D-16-48733

DOI: 10.1371/journal.pone.0174831

PMC ID: 5373625

PubMed ID: 28358870

SO-VID: 9578fae1-9c43-4c4a-ac61-2791725b5e8d

License:

This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

History

Date received : 12 December 2016

Date accepted : 15 March 2017

Page count

Figures: 2, Tables: 4, Pages: 15

Funding

Non-monetary support for this research was provided by the Institute for Quality and Efficiency in Health Care (IQWiG). No external funding was received.

Custom metadata

Data Availability All relevant data are within the paper and its Supporting Information files.

Guideline appraisal with AGREE II: Systematic review of the current evidence on how users handle the 2 overall assessments

Read this article at

Abstract

Introduction

Materials and methods

Results

Conclusions

Related collections

PLOS: COVID-19 pandemic

Most cited references 51

Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing

Effectiveness and efficiency of guideline dissemination and implementation strategies.

Systematic review and meta-analysis of practice facilitation within primary care settings.

Author and article information

Contributors

Journal

Affiliations

Author notes

Author information

Article

History

Page count

Funding

Categories

Custom metadata

Comments

Comment on this article

Similar content 178

Cited by 36

Most referenced authors 1,075