Statistical Methods Used to Test for Agreement of Medical Instruments Measuring Continuous Variables in Method Comparison Studies: A Systematic Review

There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

Abstract

Background

Accurate values are a must in medicine. An important parameter in determining the quality of a medical instrument is agreement with a gold standard. Various statistical methods have been used to test for agreement. Some of these methods have been shown to be inappropriate. This can result in misleading conclusions about the validity of an instrument. The Bland-Altman method is the most popular method judging by the many citations of the article proposing this method. However, the number of citations does not necessarily mean that this method has been applied in agreement research. No previous study has been conducted to look into this. This is the first systematic review to identify statistical methods used to test for agreement of medical instruments. The proportion of various statistical methods found in this review will also reflect the proportion of medical instruments that have been validated using those particular methods in current clinical practice.

Methodology/Findings

Five electronic databases were searched between 2007 and 2009 to look for agreement studies. A total of 3,260 titles were initially identified. Only 412 titles were potentially related, and finally 210 fitted the inclusion criteria. The Bland-Altman method is the most popular method with 178 (85%) studies having used this method, followed by the correlation coefficient (27%) and means comparison (18%). Some of the inappropriate methods highlighted by Altman and Bland since the 1980s are still in use.

Conclusions

This study finds that the Bland-Altman method is the most popular method used in agreement research. There are still inappropriate applications of statistical methods in some studies. It is important for a clinician or medical researcher to be aware of this issue because misleading conclusions from inappropriate analyses will jeopardize the quality of the evidence, which in turn will influence quality of care given to patients in the future.

Related collections

Most cited references 53

Record: found
Abstract: found
Article: not found

Body fluid volumes measurements by impedance: A review of bioimpedance spectroscopy (BIS) and bioimpedance analysis (BIA) methods.

Michel Jaffrin, Patrick Morel (2008)

This paper reviews various bioimpedance methods permitting to measure non-invasively, extracellular, intracellular and total body water (TBW) and compares BIA methods based on empirical equations of the wrist-ankle resistance or impedance at 50 kHz, height and weight with BIS methods which rely on an electrical model of tissues and resistances measured at zero and infinite frequencies. In order to compare these methods, impedance measurements were made with a multifrequency Xitron 4200 impedance meter on 57 healthy subjects which had undergone simultaneously a Dual X-ray absorptiometry examination (DXA), in order to estimate their TBW from their fat-free-mass. Extracellular (ECW) and TBW volumes were calculated for these subjects using the original BIS method and modifications of Matthie[Matthie JR. Second generation mixture theory equation for estimating intracellular water using bioimpedance spectroscopy. J Appl Physiol 2005;99:780-1], Jaffrin et al. [Jaffrin MY, Fenech M, Moreno MV, Kieffer R. Total body water measurement by a modification of the bioimpédance spectroscopy method. Med Bio Eng Comput 2006;44:873-82], Moissl et al. [Moissl UM, Wabel P, Chamney PW, Bosaeus I, Levin NW, et al. Body fluid volume determination via body composition spectroscopy in health and disease. Physiol Meas 2006;27:921-33] and their TBW resistivities were compared and discussed. ECW volumes were calculated by BIA methods of Sergi et al. [Sergi G, Bussolotto M, Perini P, Calliari I, et al. Accuracy of bioelectrical bioimpedance analysis for the assessment of extracellular space in healthy subjects and in fluid retention states. Ann Nutr Metab 1994;38(3):158-65] and Hannan et al. [Hannan WJ, Cowen SJ, Fearon KC, Plester CE, Falconer JS, Richardson RA. Evaluation of multi-frequency bio-impedance analysis for the assessment of extracellular and total body water in surgical patients. Clin Sci 1994;86:479-85] and TBW volumes by BIA methods of Kushner and Schoeller [Kushner RF, Schoeller DA. Estimation of total body water by bioelectrical impedance analysis. Am J Clin Nutr 1986;44(3):417-24], Lukaski et al. [Lukaski HC, Bolonchuk WW. Estimation of body fluid volumes using tetrapolar bioelectrical impedance measurements. Aviat Space Environ Med 1988;59:1163-9], Hannan et al. [Hannan WJ, Cowen SJ, Fearon KC, Plester CE, Falconer JS, Richardson RA. Evaluation of multi-frequency bio-impedance analysis for the assessment of extracellular and total body water in surgical patients. Clinical Science 1994;86:479-85], Deurenberg et al. [Deurenberg P, van der Koy K, Leenen R, Westrate JA, Seidell JC. Sex and age specific prediction formulas for estimating body composition from bioelectric impedance: a cross validation study. Int J Obesity 1991;15:17-25] These volumes were compared against those given by BIS method and, in the case of TBW, with those by DXA. For ECW, a good agreement was found between various BIS methods and that of Sergi while Hannan's values were higher. Both Matthie's and Moissl's methods gave mean TBW resistivities and volumes lower than those of Jaffrin's and DXA methods. Kushner et al. method gave values of TBW not significantly different from those of Jaffrin et al. and DXA, as Hannan's method in men, but Lukaski and Deurenberg methods led to an underestimation.

0 comments Cited 114 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

A note on the use of the intraclass correlation coefficient in the evaluation of agreement between two methods of measurement.

J Bland, D Altman (1990)

The intraclass correlation coefficient (rI) has been advocated as a statistic for assessing agreement or consistency between two methods of measurement, in conjunction with a significance test of the difference between means obtained by the two methods. We show that neither technique is appropriate for assessing the interchangeability of measurement methods. We describe an alternative approach based on estimation of the mean and standard deviation of differences between measurements by the two methods.

0 comments Cited 87 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

Statistical techniques for comparing measurers and methods of measurement: a critical review.

John Ludbrook (2002)

1. Clinical and experimental pharmacologists and physiologists often wish to compare two methods of measurement, or two measurers. 2. Biostatisticians insist that what should be sought is not agreement between methods or measurers, but disagreement or bias. 3. If measurements have been made on a continuous scale, the main choice is between the Altman-Bland method of differences and least products regression analysis. It is argued that although the former is relatively simple to execute, it does not distinguish adequately between fixed and proportional bias. Least products regression analysis, although more difficult to execute, does achieve this goal. There is almost universal agreement among biostatisticians that the Pearson product-moment correlation coefficient (r) is valueless as a test for bias. 4. If measurements have been made on a categorical scale, unordered or ordered, the most popular method of analysis is to use the kappa statistic. If the categories are unordered, the unweighted kappa statistic (K) is appropriate. If the categories are ordered, as they are in most rating scales in clinical, psychological and epidemiological research, the weighted kappa statistic (K(w)) is preferable. But K(w) corresponds to the intraclass correlation coefficient, which, like r for continuous variables, is incapable of detecting bias. Simple techniques for detecting bias in the case of ordered categorical variables are described and commended to investigators.

0 comments Cited 61 times – based on 0 reviews      Review now

Bookmark

All references

Author and article information

Contributors

: Role: Editor

Journal

Journal ID (nlm-ta): PLoS One

Journal ID (iso-abbrev): PLoS ONE

Journal ID (publisher-id): plos

Journal ID (pmc): plosone

Title: PLoS ONE

Publisher: Public Library of Science (San Francisco, USA )

ISSN (Electronic): 1932-6203

Publication date Collection: 2012

Publication date (Electronic): 25 May 2012

Volume: 7

Issue: 5

Electronic Location Identifier: e37908

Affiliations

[1 ]Julius Centre University of Malaya, Department of Social and Preventive Medicine, Faculty of Medicine, University of Malaya, Kuala Lumpur, Malaysia

[2 ]Department of Applied Statistics, Faculty of Economics and Administration, University of Malaya, Kuala Lumpur, Malaysia

University of East Piedmont, Italy

Author notes

* E-mail: rafdzah@ 123456hotmail.com

Conceived and designed the experiments: RZ AB RI NAI. Performed the experiments: RZ AB RI NAI. Analyzed the data: RZ AB RI NAI. Contributed reagents/materials/analysis tools: RZ AB RI NAI. Wrote the paper: RZ AB RI NAI.

Article

Publisher ID: PONE-D-12-05103

DOI: 10.1371/journal.pone.0037908

PMC ID: 3360667

PubMed ID: 22662248

SO-VID: fc7ee749-d17f-4a05-8c72-943ecef4732b

Copyright © Zaki et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

History

Date received : 20 February 2012

Date accepted : 30 April 2012

Page count

Pages: 7

Comments

Comment on this article

scite_

Cited by 85

See all cited by

Most referenced authors 194

See all reference authors

- Version 1

Statistical Methods Used to Test for Agreement of Medical Instruments Measuring Continuous Variables in Method Comparison Studies: A Systematic Review

Read this article at

Abstract

Background

Methodology/Findings

Conclusions

Related collections

PLOS Climate

Most cited references 53

Body fluid volumes measurements by impedance: A review of bioimpedance spectroscopy (BIS) and bioimpedance analysis (BIA) methods.

A note on the use of the intraclass correlation coefficient in the evaluation of agreement between two methods of measurement.

Statistical techniques for comparing measurers and methods of measurement: a critical review.

Author and article information

Contributors

Journal

Affiliations

Author notes

Article

History

Page count

Categories

Comments

Comment on this article

Similar content 212

Cited by 85

Most referenced authors 194