Why the C-statistic is not informative to evaluate early warning scores and what metrics to use

There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

Abstract

Metrics typically used to report the performance of an early warning score (EWS), such as the area under the receiver operator characteristic curve or C-statistic, are not useful for pre-implementation analyses. Because physiological deterioration has an extremely low prevalence of 0.02 per patient-day, these metrics can be misleading. We discuss the statistical reasoning behind this statement and present a novel alternative metric more adequate to operationalize an EWS. We suggest that pre-implementation evaluation of EWSs should include at least two metrics: sensitivity; and either the positive predictive value, number needed to evaluate, or estimated rate of alerts. We also argue the importance of reporting each individual cutoff value.

Related collections

Most cited references 19

Record: found
Abstract: found
Article: not found

Use and misuse of the receiver operating characteristic curve in risk prediction.

Nancy R. Cook (2007)

The c statistic, or area under the receiver operating characteristic (ROC) curve, achieved popularity in diagnostic testing, in which the test characteristics of sensitivity and specificity are relevant to discriminating diseased versus nondiseased patients. The c statistic, however, may not be optimal in assessing models that predict future risk or stratify individuals into risk categories. In this setting, calibration is as important to the accurate assessment of risk. For example, a biomarker with an odds ratio of 3 may have little effect on the c statistic, yet an increased level could shift estimated 10-year cardiovascular risk for an individual patient from 8% to 24%, which would lead to different treatment recommendations under current Adult Treatment Panel III guidelines. Accepted risk factors such as lipids, hypertension, and smoking have only marginal impact on the c statistic individually yet lead to more accurate reclassification of large proportions of patients into higher-risk or lower-risk categories. Perfectly calibrated models for complex disease can, in fact, only achieve values for the c statistic well below the theoretical maximum of 1. Use of the c statistic for model selection could thus naively eliminate established risk factors from cardiovascular risk prediction scores. As novel risk factors are discovered, sole reliance on the c statistic to evaluate their utility as risk predictors thus seems ill-advised.

0 comments Cited 514 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

Monitor alarm fatigue: standardizing use of physiological monitors and decreasing nuisance alarms.

Douglas K. Graham, Maria Cvach (2009)

Reliance on physiological monitors to continuously "watch" patients and to alert the nurse when a serious rhythm problem occurs is standard practice on monitored units. Alarms are intended to alert clinicians to deviations from a predetermined "normal" status. However, alarm fatigue may occur when the sheer number of monitor alarms overwhelms clinicians, possibly leading to alarms being disabled, silenced, or ignored. Excessive numbers of monitor alarms and fear that nurses have become desensitized to these alarms was the impetus for a unit-based quality improvement project. Small tests of change to improve alarm management were conducted on a medical progressive care unit. The types and frequency of monitor alarms in the unit were assessed. Nurses were trained to individualize patients' alarm parameter limits and levels. Monitor software was modified to promote audibility of critical alarms. Critical monitor alarms were reduced 43% from baseline data. The reduction of alarms could be attributed to adjustment of monitor alarm defaults, careful assessment and customization of monitor alarm parameter limits and levels, and implementation of an interdisciplinary monitor policy. Although alarms are important and sometimes life-saving, they can compromise patients' safety if ignored. This unit-based quality improvement initiative was beneficial as a starting point for revamping alarm management throughout the institution.

0 comments Cited 88 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record.

Gabriel Escobar, Juan Carlos LaGuardia, Benjamin Turk … (2016)

Ward patients who experience unplanned transfer to intensive care units have excess morbidity and mortality. To develop a predictive model for prediction of unplanned transfer from the medical-surgical ward to intensive care (or death on the ward in a patient who was "full code") using data from a comprehensive inpatient electronic medical record (EMR). Retrospective case-control study; unit of analysis was a 12-hour patient shift. Shifts where a patient experienced an unplanned transfer were event shifts; shifts without a transfer were comparison shifts. Hospitalization records were transformed into 12-hour shift records, with 10 randomly selected comparison shifts identified for each event shift. Analysis employed logistic regression and split validation. Integrated healthcare delivery system in Northern California. Hospitalized adults at 14 hospitals with comprehensive inpatient EMRs. Predictors included vital signs, laboratory test results, severity of illness scores, longitudinal chronic illness burden scores, transpired hospital length of stay, and care directives. Patients were also given a retrospective, electronically (not manually assigned) Modified Early Warning Score, or MEWS(re). Outcomes were transfer to the intensive care unit (ICU) from the ward or transitional care unit, or death outside the ICU among patients who were "full code". We identified 4,036 events and 39,782 comparison shifts from a cohort of 102,422 patients' hospitalizations. The MEWS(re) had a c-statistic of 0.709 in the derivation and 0.698 in the validation dataset; corresponding values for the EMR-based model were 0.845 and 0.775. Using these algorithms requires hospitals with comprehensive inpatient EMRs and longitudinal data. EMR-based detection of impending deterioration outside the ICU is feasible in integrated healthcare delivery systems. Copyright © 2012 Society of Hospital Medicine.

0 comments Cited 56 times – based on 0 reviews      Review now

Bookmark

All references

Author and article information

Contributors

Santiago Romero-Brufau:

ORCID: http://orcid.org/0000-0002-9922-0083

RomeroBrufau.Santiago@mayo.edu

Jeanne M. Huddleston: Huddleston.jeanne@mayo.edu

Gabriel J. Escobar: Gabriel.Escobar@kp.org

Mark Liebow: mliebow@mayo.edu

Journal

Journal ID (nlm-ta): Crit Care

Title: Critical Care

Publisher: BioMed Central (London )

ISSN (Print): 1364-8535

ISSN (Electronic): 1466-609X

Publication date (Electronic): 13 August 2015

Publication date PMC-release: 13 August 2015

Publication date (Print): 2015

Volume: 19

Issue: 1

Electronic Location Identifier: 285

Affiliations

[ ]Healthcare Systems Engineering Program, Mayo Clinic Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery, 200 First Street SW, Rochester, MN 55905 USA

[ ]Division of Health Care Policy and Research, Department of Health Sciences Research, Mayo Clinic, 200 First Street SW, Rochester, MN 55905 USA

[ ]Division of Hospital Internal Medicine, Mayo Clinic, 200 First Street SW, Rochester, MN 55905 USA

[ ]Kaiser Permanente Division of Research, 2000 Broadway Avenue, 032 R01, Oakland, CA 94612 USA

[ ]Division of General Internal Medicine, Mayo Clinic College of Medicine, 200 First Street SW, Rochester, MN 55905 USA

Author information

Santiago Romero-Brufau http://orcid.org/0000-0002-9922-0083

Article

Publisher ID: 999

DOI: 10.1186/s13054-015-0999-1

PMC ID: 4535737

PubMed ID: 26268570

SO-VID: 031166ef-04f9-44d2-be39-372eaf68e42e

License:

This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.