Comparing Machine Learning and Nurse Predictions for Hospital Admissions in a Multisite Emergency Care System

ABSTRACT

Background

Emergency department (ED) crowding strains patient care and drives up costs. Early decisions on the need for patient hospital admissions can allow for better planning and potentially improve throughput and alleviate crowding. We sought to prospectively compare nurse predictions with a machine learning (ML) model for hospital admissions and to evaluate whether adding the nurse prediction to ML outputs enhances predictive performance.

Methods

In this prospective, observational study at six hospitals in a large mixed quarternary/community ED system (annual ED census ∼500,000), triage nurses recorded a binary admission prediction for adult patients. These predictions were compared with an ensemble ML model (XGBoost + Bio-Clinical BERT) trained on structured data (demographics, vital signs, medical history) and triage text. Nurse predictions were similarly analyzed and then integrated with the ML output to assess for improvement in predictive accuracy.

Results

The ensemble ML model (XGBoost + Bio-Clinical BERT) was trained on 1.8 million historical ED visits (January 2019–December 2023). It was then tested on 46,912 prospective ED visits with recorded nurse predictions (September to October 2024). In the prospective arm, nurse predictions yielded an accuracy of 81.6% (95% CI, 81.3–81.9), sensitivity of 64.8% (63.7–65.8), and specificity of 85.7% (85.3–86.0). At a 0.30 probability threshold, the ML model attained an accuracy of 85.4% (85.0–85.7) and sensitivity of 70.8% (69.8–71.7). Combining nurse predictions with the ML output did not improve accuracy beyond the model alone.

Conclusions

Machine learning–based predictions outperformed triage nurse estimates for hospital admissions. These findings suggest that an admission prediction system anchored by ML can perform reliably using data available at triage.