Open Access

Julia OhseORCiD, Bakir HadžićORCiD, Nicolina PeperkornORCiD, Maxi PloenORCiD, Janosch FoxORCiD, Johanna NeukirchORCiD, Matthias RätschORCiD, Youssef ShibanORCiD

• Background Despite high prevalence, depression often remains undetected. Artificial Intelligence (AI) screening methods may help address this gap. AI models’ proficiency in rating depression interviews in comparison with human raters has been demonstrated for clinical populations. Objectives The capacity of three Large Language Models (LLMs) for rating depression interviews was evaluated. The test dataset contained interviews from the general population to evaluate model capacity for screening. LLM ratings were compared with assessments provided by trained psychology students. Methods GPT-4, GPT-4o and GPT-4o-mini were prompted to rate interview transcripts from N = 55 participants from a convenience sample of the German population on a depression scale. Models were used without task-specific fine-tuning. LLM performance was evaluated via intraclass correlations with trained psychology students’ assessments leveraging a scale explicitly designed to yield reliable ratings when applied by first-time raters. Results GPT-4 and GPT-4o achieved moderate to good interrater reliability with student raters (ICC 0.76 and 0.75) while ICC for GPT-4o-mini was moderate (0.55), with 98.88% CIs [0.38 to 0.70] indicating greater uncertainty. F1 accuracy scores ranged from 0.56 for GPT-4o-mini to 0.75 for GPT-4. Discussion The results suggest GPT-4's and GPT-4o's potential for scoring depression interviews from the general population, with GPT-4o meeting sensitivity and specificity values for case finding instruments. Our results require further validation with clinician raters in a larger and more diverse sample. Future research should explore fine-tuning and potential model bias.