Optimum binary cut-off threshold of a diagnostic test: comparison of different methods using Monte Carlo technique

Abstract

Background

Using Monte Carlo simulations, we compare different methods (maximizing Youden index, maximizing mutual information, and logistic regression) for their ability to determine optimum binary cut-off thresholds for a ratio-scaled diagnostic test variable. Special attention is given to the stability and precision of the results in dependence on the distributional characteristics as well as the pre-test probabilities of the diagnostic categories in the test population.

Methods

Fictitious data sets of a ratio-scaled diagnostic test with different distributional characteristics are generated for 50, 100 and 200 fictitious “individuals” with systematic variation of pre-test probabilities of two diagnostic categories. For each data set, optimum binary cut-off limits are determined employing different methods. Based on these optimum cut-off thresholds, sensitivities and specificities are calculated for the respective data sets. Mean values and SD of these variables are computed for 1000 repetitions each.

Results

Optimizations of cut-off limits using Youden index and logistic regression-derived likelihood ratio functions with correct adaption for pre-test probabilities both yield reasonably stable results, being nearly independent from pre-test probabilities actually used. Maximizing mutual information yields cut-off levels decreasing with increasing pre-test probability of disease. The most precise results (in terms of the smallest SD) are usually seen for the likelihood ratio method. With this parametric method, however, cut-off values show a significant positive bias and, hence, specificities are usually slightly higher, and sensitivities are consequently slightly lower than with the two non-parametric methods.

Conclusions

In terms of stability and bias, Youden index is best suited for determining optimal cut-off limits of a diagnostic variable. The results of Youden method and likelihood ratio method are surprisingly insensitive against distributional differences as well as pre-test probabilities of the two diagnostic categories. As an additional bonus of the parametric procedure, transfer of the likelihood ratio functions, obtained from logistic regression analysis, to other diagnostic scenarios with different pre-test probabilities is straightforward.

Electronic supplementary material

The online version of this article (doi:10.1186/s12911-014-0099-1) contains supplementary material, which is available to authorized users.