Ultra-short screening tools involving fewer than five questions have been recommended as a simple method of detecting distress, anxiety, or depression in cancer settings. Such methods have practical appeal, but their diagnostic accuracy is unclear. A literature search limited to diagnostic validity studies of ultra-short screening in cancer settings identified 38 analyses, including 19 assessing the Distress Thermometer alone, involving a total of 6,414 unique patients. The pooled ability of ultra-short methods to detect depression was given by a sensitivity of 78.4%, a specificity of 66.8%, a positive predictive value (PPV) of 34.2%, and a negative predictive value (NPV) of 93.4%. Thus these tools were very good at excluding possible cases of depression but poor at confirming a suspected diagnosis. The pooled ability of ultra-short methods to detect anxiety was given by a sensitivity of 77.3% and a specificity of 56.6% (PPV, 55.2%; NPV, 80.25%) and for distress a sensitivity of 78.3% and a specificity of 66.5% (PPV, 59.7%; NPV, of 82.8%). Results using the Distress Thermometer alone were similar. Scores of integrated accuracy, using the Youden index and diagnostic odds ratio, suggested modest overall accuracy with least success in diagnosing anxiety disorders. Ultra-short methods were modestly effective in screening for mood disorders. Their rule-in ability was poorer than their rule-out ability. Ultra-short methods cannot be used alone to diagnose depression, anxiety, or distress in cancer patients but they may be considered as a first-stage screen to rule out cases of depression.