Evidence Based Emergency Medicine; Part 5 Receiver Operating Curve and Area under the Curve

Abstract

1. Abstract Multiple diagnostic tools are used by emergency physicians, every day. In addition, new tools are evaluated to obtain more accurate methods and reduce time or cost of conventional ones. In the previous parts of this educational series, we described diagnostic performance characteristics of diagnostic tests including sensitivity, specificity, positive and negative predictive values, and likelihood ratios. The receiver operating characteristics (ROC) curve is a graphical presentation of screening characteristics. ROC curve is used to determine the best cutoff point and compare two or more tests or observers by measuring the area under the curve (AUC). In this part of our educational series, we explain ROC curve and two methods to determine the best cutoff value. 2. Definition ROC curve is a graphical presentation, which is created by plotting the true positive rate (sensitivity) against the false positive rate (1-specificity) at various threshold settings (1,2). 3. The area under ROC curve It is known that when sensitivity increases, false positive rate also increases. Therefore, ROC curve allows us to check sensitivity and false positive rate (1- specificity) at any point on the curve. The area under the curve is a measurement of the overall quality of the diagnostic test. Tests with the same AUC show the same overall diagnostic performance, but not the same sensitivity and specificity. AUC is interpreted as the probability that a random person with the disease has a higher test measurement than a random person who is healthy. Accuracy of a test is measured by the AUC, which can be calculated by many statistical software (1,2). Based on a rough classifying system, AUC can be interpreted as follows: 90 -100 = excellent; 80 - 90 = good; 70 - 80 = fair; 60 - 70 = poor; 50 - 60 = fail. In figure 1, the line (A) represents the ROC for an ideal diagnostic test. This curve represents a sensitivity and specificity of 100%. The AUC for this line is equal to 100, which is not common in practice. On the other hand, the diagonal line (B) represents ROC for a useless diagnostic test. This can be explained by noting that the true positive rate of the diagnostic test is equal to the false positive rate at any cutoff point. The AUC for this line is equal to 45, which is why such test is of no real value. Figure 1 ROC curve for a test with a sensitivity and specificity of 100% (A). ROC curve for a useless test (B). 4. Determining the best cutoff point on ROC curve There are two methods to choose the best cutoff point. 5. The first method It is known that the upper left corner of the graph represents the point with 100% sensitivity and 100% specificity. Therefore, the point on the curve that gives the highest sensitivity and the highest specificity will be the nearest point to the upper left corner of the graph. To determine the nearest point, we should measure the distance between each point and the upper left corner and choose the nearest point as the best cutoff point. To measure the distance of each point from the upper left corner, we can use the following equation: Distance = [ 1 - sensitivity 2 + 1 - specificity 2 ] 2 The point with the shortest distance should be chosen as the best cutoff point. 6. The second method The second method is called Youden index where we aim at maximizing the gap between true positive rate (represented as: sensitivity) and false positive rate (represented as 1- specificity). In this method, we calculate (sensitivity + specificity) for all points and Youden index is the point where the sum of sensitivity and specificity is the highest. 7. Example In the recently published study by Mazandarani and colleagues (4), they compared the ability of trauma injury severity score (TRISS) and acute physiology and chronic health evaluation (APACHE) III to predict mortality of intensive care unit admitted trauma patients. ROC curves for the methods are shown in figure 2 and 3. Figure 2 shows the ROC curve for TRISS; we notice that the curve is nearest to the upper left corner in the cutoff point (13.2) and therefore, the maximum sensitivity and specificity are achieved at this point (sensitivity = 76.52%; specificity = 95.65%). Figure 2 ROC curve of TRISS system Figure 3 ROC curve of APACHE III system Figure 3 shows the ROC curve for APACHE III; we notice that the curve is nearest to the upper left corner in the cutoff point (51) and therefore, maximum sensitivity and specificity are achieved at this point (sensitivity = 73.91%; specificity = 78.26%). 8. Notes This method might yield more than one cutoff point and therefore choosing between them is challenging. This method gives sensitivity and specificity equal priority in the diagnostic test; sometimes the cost of screening is high and the disease is rare, in these conditions, sensitivity and specificity are not given the same priority when determining the best cutoff point. The method of integrating the cost of screening and prevalence of the disease with sensitivity and specificity while determining the best cutoff is rarely used in medical literature.