A new mental health benchmark reveals that while today’s state-of-the-art AI models may sound convincing, they remain inadequate when emotional insight, careful diagnosis, and real-world clinical judgment are most important.
Research: PsyEval: A comprehensive large-scale language model evaluation benchmark for mental health. Image credit: ahmetmapush / Shutterstock
In a recent study published in the journal npj mental health researchIn , researchers describe the development of ‘PsyEval’, a novel and comprehensive benchmark designed to assess the performance of large-scale language models (LLMs) in the context-sensitive and subjective domain of human mental health.
This benchmark was then used to evaluate 11 advanced LLMs across three key dimensions: 1. Knowledge Understanding, 2. Diagnosis and Assessment, and 3. Emotional Support. PsyEval’s findings reveal that while some large-scale language models can mirror human fluency and demonstrate strong fact recall, they still lack the deeper emotional insight and clinical nuance needed to perform complex mental health tasks reliably.
Specifically, PsyEval shows that the evaluated LLMs, while exhibiting a high degree of immediate sensitivity, exhibit an “empathy gap” compared to human counselors. Additionally, our findings suggest that while safety-related rejections may reduce the raw diagnostic classification scores of some large underlying models, less safe models may classify patients more aggressively, risking overdiagnosis.
background
Mental illness is one of the most major chronic diseases in human society today, with World Health Organization (WHO) data showing that depression alone affects 3.8% of the world’s population.
Compounding this problem, treatment rates remain extremely low: 13.7% in lower middle income countries, 22.0% in upper middle income countries, and 36.8% in high income countries. The paper notes that these numbers may be underestimates due to social bias and limited public awareness.
Against this background, researchers are increasingly investigating artificial intelligence (AI) and LLM as potential tools for mental health applications. Although early applications in basic sentiment analysis looked promising, it has been difficult to comprehensively evaluate these systems.
Unlike tasks with clearly verifiable outcomes, psychiatric assessments rely heavily on interpreting subtle, highly ambiguous, and often subjective verbal cues, in addition to establishing a strong therapeutic bond between those seeking support and mental health professionals.
Unfortunately, until now there has been a lack of a unified framework to stress test whether AI can safely balance knowledge and diagnostic performance with empathic communication.
About research
The present study aimed to address this assessment gap by developing PsyEval, a task suite specifically designed to approximate key elements of real-world psychiatric examinations across three dimensions. This benchmark consisted of separate assessment datasets covering knowledge, diagnostic classification, and emotional support tasks.
First, the “Knowledge Tasks” dataset consists of 5,531 multiple-choice questions extracted from the United States (US) and Mainland China medical licensing exams (USMLE and MCMLE) to test core medical facts and crisis response scenarios.
The “Diagnostic Task” dataset was then collated from 1,000 user-level test instances derived from selected social media posts in the Self-Reported Mental Health Detection (SMHD) dataset to assess single-condition classification and detection of multiple conditions or comorbidities. At the same time, 1,339 clinical interactions from the Chinese D4 dataset were used to assess depression and suicide risk levels.
Finally, 1,000 user inquiries from the Chinese PsyQA platform and 939 English-language counseling interactions from Counsel-Chat were incorporated as a dataset for emotional support assessment.
The researchers then applied PsyEval to compare 11 general-purpose LLMs and domain-specific LLMs across three prompting strategies: 1. Formal commands, 2. Step-by-step reasoning, and 3. Scenario simulations on mental health-related benchmark tasks such as GPT-4, Qwen2.5-72B, and SoulChat.
Emotional support responses were assessed using automated metrics and a hybrid human-AI voting process that included human annotators and DeepSeek-V3.
Research results
The results showed that larger generic models generally performed better on factual knowledge tasks, although performance was also highly dependent on language tuning. In the general knowledge task, Qwen2.5-72B achieved 91.0% accuracy on the Chinese MCMLE mental dataset. Similarly, GPT-4-turbo led the English task with approximately 76.0% accuracy.
Despite this factual mastery, the model’s performance generally declined during questioning in emergency psychiatric scenarios. For example, on the USMLE crisis response dataset, the accuracy of GPT-4-turbo decreased to approximately 73.0%.
This study further identified an “inverse scaling” phenomenon in the model’s raw diagnostic classification accuracy. Small-scale models like LLaMa-3-8B achieved near-perfect accuracy in classifying conditions such as 100.0% for attention-deficit/hyperactivity disorder (ADHD) and 96.0% for anxiety, significantly outperforming GPT-4, which achieved around 25% for ADHD.
The authors hypothesized that these discrepancies reflect a safety-practicality trade-off between benchmark accuracy and prudent model behavior. Flagship foundational models can enforce guardrails that impede patient diagnosis and medical care, and can lead to rejections and ambiguous responses that count as incorrect answers in PsyEval. Conversely, smaller models may allow diagnostic labels to be assigned more easily, potentially increasing the risk of overdiagnosis.
The study compared the emotional support performance of LLMs to human counselors and found that while some leading models matched or exceeded the latter in verbal fluency and consistency, humans consistently outperformed the evaluated AI models in asking deep questions to uncover deeper concerns, achieving “exploration scores” of 1.85 on PsyQA and 1.93 on Counsel-Chat.
Finally, prompting style was found to have a significant impact on LLM performance. For scenario simulation prompts, instructing the model to adopt the persona of a mental health professional generally produced the highest empathy scores, whereas step-by-step prompts often produced more mechanical responses.
conclusion
Although PsyEval demonstrated that several large-scale LLMs achieved high accuracy on selected medical knowledge questions, this study did not directly compare their knowledge or diagnostic performance to that of mental health professionals. Nevertheless, the models evaluated showed large differences in performance between English and Chinese tasks, remained sensitive to prompt wording, and were inconsistent in addressing comorbidities and grading clinical risk.
The SoulChat findings also suggest that fine-tuning of highly specialized conversations may be associated with poorer general knowledge performance in some models, but this observation should not be generalized to all specialized mental health models.
As a benchmark study, this study did not prospectively test the model in patients, measure treatment outcomes, assess diagnostic safety in real-world settings, or evaluate autonomous crisis intervention.
These findings highlight that future model development should focus on incorporating culturally diverse alignment data and designing smarter safety mechanisms that protect users without unduly reducing the potential clinical utility of the model.
Reference magazines:
- Jin, H., Chen, S., Dilixiati, D., Jiang, Y., Zhu, K.Q., and Wu, M. (2026). PsyEval: A comprehensive large-scale language model evaluation benchmark for mental health. npj mental health research, press articles. Doi: 10.1038/s44184-026-00227-0, https://www.nature.com/articles/s44184-026-00227-0
