Helping everyone do better: a call for validation studies of routinely recorded health data

Abstract

There has been a surge of availability and use for research of routinely collected electronic health data, such as electronic health records, health administrative data, and disease registries. Symptomatic of this surge, in 2012, Pharmacoepidemiology and Drug Safety (PDS) published a supplemental issue containing several reviews of validated methods for identifying health outcomes using routine health data,1 focusing on databases feeding the US Mini-Sentinel Program.2 In one of the review papers of the PDS Supplement, Carnahan3 acknowledged that while ample validated algorithms exist for major health events, for example, cardiovascular events, validated methods of identifying many health outcomes are lacking. Furthermore, the referenced studies focused on algorithms based on coding sets used in the United States (eg, ICD-9) to identify events from US databases, set within the US health care system. This leaves out an entire segment of routine databases, most notably, Nordic national registries or other European databases such as Clinical Practice Research Datalink (CPRD), The Heatlh Improvement Network (THIN) Hospital Episode Statistics (HES), or PHARMO, all of which are set in health care systems that are differently run and financed than those in the United States. Since other systems function differently, and the databases contain different variables, validation of health status in US data may not always be generalizable.5–9 Many validation studies have been done among these various resources,10–12 but the work is far from complete, as shown in a systematic review of validation studies of the UK-based Clinical Practice Research Datalink, published in 2010.13 Some algorithms may become outdated because of changes in coding or medical practices; new diseases, without clear representation in classification systems, may emerge. Furthermore, in October 2015, the United States adopted ICD-10,14 while ICD-11 is looming on the horizon.15 Clinical Epidemiology has published and continues to publish studies that describe the validity of algorithms in routinely recorded health data, such as validation of medication use in hospitals,16,17 cancer characteristics and complications,18–20 or events related to reproductive and fetal medicine,21,22 to name just a few examples. An “algorithm” in the present context refers to a combination of values of routinely collected variables that allow identification of cases of a given disease or other health event without having to contact or examine the patient. For example, an algorithm based on a combination of diagnostic ICD-10 codes E10-E11 and medication ATC codes A10 may identify patients with diabetes. The commonly evaluated aspects of an algorithm’s validity are positive predictive value (proportion of algorithm-positive patients who truly have the disease of interest) and sensitivity (proportion of patients with the disease of interest who are algorithm-positive), and their counterparts negative predictive value (proportion of algorithm-negative persons without the disease of interest) and specificity (proportion of persons without the disease who are algorithm-negative). Validity of entire data sources is commonly measured by their completeness (proportion of true cases of a disease captured by a data source). A comprehensive review of methods for validating algorithms to identify disease cohorts from health administrative data, with accompanying reporting guidelines for such work, was published by the Journal of Clinical Epidemiology in 2011.23 Clinical Epidemiology is hereby issuing a targeted call for papers that report on results of validation studies. We are interested in publishing both original validation studies and systematic reviews, using various types of reference (“gold”) standards, such as review of medical charts or comparison with other data sources. Several resources are available to guide reporting, including the 2011 guidelines mentioned above,23 as well as the STARD Checklist,24 and the RECORD Checklist.25,26 Please take advantage of these resources in preparing your high-quality submissions. Some may think of validation work as mundane, a mere poor relative of the “real” original research. We subscribe to a different viewpoint. First, misclassification of study variables threatens the validity of research findings.27 Since epidemiologic research is “an exercise in measurement”,28 high-quality original research is unthinkable without accurate or accurately calibrated instruments. In our editorial experience, evidence of data validity is routinely requested by article referees. Second, following from above, results of validation studies allow epidemiologists to assess the extent of misclassification and estimate its impact on the study results. Third, shining the spotlight on validation studies may activate a feedback loop: physicians may become even more motivated to use systematic coding schemes keeping in mind that the data they feed into the routine databases will be used for research that will ultimately benefit their patients. Last, but not least, validation studies are frequently cited. For example, systematic reviews by Khan et al29 and Herrett et al,13 published in 2010, have already received more than 240 and 350 citations, respectively. We hope you find our arguments compelling and look forward to receiving your validation study submissions.