Establishing a risk prediction model for acute kidney injury: methodology is important

Abstract

To the Editor: By a retrospective study including 1124 hospitalized patients diagnosed with acute myocardial infarction (AMI), Wang et al [1] showed that the independent risk factors for acute kidney injury (AKI) were age >60 years, hypertension, chronic kidney disease (CKD), Killip class ≥3, extensive anterior myocardial infarction, use of furosemide, non-use of angiotensin-converting enzyme inhibitors (ACEI)/angiotensin receptor blocker (ARB), and these factors could provide a prediction model with good discriminative ability for the development of AKI. Given that AKI has been significantly associated with morbidity and mortality of patients with AMI,[2] their findings have potential clinical implications. Other than the limitations described by authors in the discussion; however, we noted some methodological issues in their study that needed further clarifications. First, this study showed that CKD was an independent risk factor for the occurrence of AKI. This study excluded the patients with end-stage renal diseases, but the readers were not provided with the evaluation method of estimated glomerular filtration rate (eGFR) and the diagnostic criteria of CKD used in this study. Based on diagnostic criteria of the Chronic Kidney Disease Epidemiology Collaboration,[3] normal renal function is defined as eGFR ≥90 mL·min−1·1.73 m−2, CKD stage 1 as 90 mL·min−1·1.73 m−2 > eGFR ≥75 mL·min−1·1.73 m−2, CKD stage 2 as 75 mL·min−1·1.73 m−2 > eGFR ≥60 mL·min−1·1.73 m−2, CKD stage 3A as 60 mL·min−1·1.73 m−2 > eGFR ≥45 mL·min−1·1.73 m−2, CKD stage 3B as 45 mL·min−1·1.73 m−2 > eGFR ≥30 mL·min−1·1.73 m−2, and CKD stage 4 as 30 mL·min−1·1.73 m−2 > eGFR ≥15 mL·min−1·1.73 m−2. As the severity of baseline CKD has been significantly associated with the risk of AKI in patients with AMI,[4] we are concerned that the lack of these data would have confused the interpretation of their results. Second, the multivariate regression analysis was used for the identification of risk factors of AKI. It is well known that emergent percutaneous coronary intervention (PCI) is one of the treatments used mostly common for AMI and the contrast-induced AKI is a recognized concern.[2] As the use of PCI was not included in the cardiac-associated data of patients, it was unclear how much patients received this treatment in this study. It must be emphasized that multivariate regression analysis is based on the assumption that there is a particular mathematical relation between the intervention and measured outcome. To obtain the true inferences of multivariate regression analysis for the adjusted odds ratio of the measured outcome, all of the known risk factors affecting measured outcome must be taken into the model. If an important risk factor is missed, the multivariate adjustment for the odd ratio of the measured outcome can be biased and even a spurious association between the intervention and measured outcome may be obtained. Thus, we argue that not taking emergent PCI into the model would have tampered with the inferences of multivariate regression analysis for risk factors of AKI and their adjusted odds ratios. Third, the authors determined the discriminative ability of the model by only providing the area under the receiving operator characteristic curve. This was incomplete. Discrimination refers to the ability of model distinguishing patients who experience an outcome from those who do not. The discriminative ability of a model is often quantified by the C-statistic, which represents the probability that a patient experiencing a measured outcome would have a higher predicted probability than a randomly selected patient not experiencing measured outcome. Generally speaking, C-statistics can be interpreted as excellent (0.90–1.00), good (0.80–0.89), fair (0.70–0.79), poor (0.60–0.69), or fail/no discriminative ability (0.50–0.59).[5] Finally, an important ignore by the authors was that the statistical validation of their model was not performed. Because the predictive model was developed by multivariate regression analysis using demographic, clinical, and other variables to generate outcome estimates, overfitting is a common issue, especially when the number of predictors and interaction terms are large, and the number of events is small. To protect against overfitting, an investigator often needs to split their data into a learning sample for model development and a test sample for model validation. If an investigator holds some of the same datasets aside for testing, it is called as an internal validation. If an investigator tests the model with an entirely different data source, it is called as external validation. Compared to internal validation, external validation can provide more rigorous protection against overfitting and evaluate the generalizability of the model to new contexts and populations.[6] Due to the lack of statistical validation for the model, it cannot exclude the possibility that the model leads to a less-reliable prediction for a new patient. We believe that addressing the above issues would improve the interpretation of the findings from this study. Author's Reply: AKI is a serious and fatal complication of AMI. It has high short-and long-term mortality rates and a poor prognosis but is potentially preventable. However, the current incidence, risk factors, and outcomes of AKI in the Chinese population are not well understood and would serve the first step to identify high-risk patients who should receive preventative care. Wang et al [1] presented a retrospective study including clinical data from 1145 consecutive hospitalized patients diagnosed with AMI in the Peking University People's Hospital and Beijing Jishuitan Hospital between October 2013 and September 2015. The results showed that approximately 26.0% of patients undergoing AMI developed AKI, and the development of AKI was strongly correlated with in-hospital mortality. Based on the linear regression analyses, a risk score for AKI among patients with AMI was created. The score considered age, hypertension, CKD, Killip class ≥3, extensive anterior myocardial infarction, use of furosemide, and non-use of an ACEI/ARB. The derived risk score for AKI had a good correlation when tested with the Hosmer-Lemeshow method (χ 2  = 12.848, P = 0.117) and a discrimination capacity (area under receiving operator characteristics [AUROC]) of 0.907 (0.887–0.926). There are several issues about the study mentioned in the letter, we addressed as follows: first, eGFR was estimated using the modification of diet in renal disease (MDRD) equation (eGFRMDRD): eGFRMDRD = 186 × serum creatinine−1.154 × Age−0.203 × 0.742 (if female) × 1.210 (if African American). In our study, we analyzed patients with stage 3 and 4 CKD (15 mL·min−1·1.73 m−2 ≤ eGFR <60 mL·min−1·1.73 m−2). Second, we had provided PCI-associated data in Table 2 of Wang et al.[1] The 734 (65.1%) patients underwent PCI. The 156 patients developed AKI during hospitalization, and the statistical analysis suggested that PCI was a risk factor for developing AKI post-AMI (P  < 0.001). Third, as we mentioned in the article, the AUROC is 0.907 (0.887–0.926) in the last part of “Result” in the article of Wang et al.[1] Fourth, that is a good question and we have not ignored it. We are working on it and hopefully, you can keep up with our next article. Conflicts of interest None.