Preference Learning Unlocks LLMs' Psycho-Counseling Skills
Mian Zhang, Shaun M. Eack, Zhiyu Zoey Chen
2025-03-03
Summary
This paper talks about a new way to make AI language models better at providing psychological counseling. The researchers created a special dataset and training method to teach AI how to respond more like professional therapists.
What's the problem?
There's a big need for mental health support, but not enough therapists to go around. AI language models could help, but they're not very good at counseling yet. This is because they haven't been trained on real therapy sessions, which are usually kept private. Also, it's hard to tell which therapist responses are actually good and helpful.
What's the solution?
The researchers came up with a set of rules to judge how good a therapist's response is. They used these rules to create a dataset called PsychoCounsel-Preference, with 36,000 examples of good counseling responses. They then used this dataset to train an AI model called PsychoCounsel-Llama3-8B. This model learned to give responses that professional therapists would prefer.
Why it matters?
This matters because it could help make mental health support more available to people who need it. The AI model they created is really good at giving counseling-like responses, even outperforming some of the most advanced AI systems. By sharing their dataset and model, the researchers are helping other scientists improve AI counseling too. This could lead to better online mental health support and help more people get the help they need when human therapists aren't available.
Abstract
Applying large language models (LLMs) to assist in psycho-counseling is an emerging and meaningful approach, driven by the significant gap between patient needs and the availability of mental health support. However, current LLMs struggle to consistently provide effective responses to client speeches, largely due to the lack of supervision from high-quality real psycho-counseling data, whose content is typically inaccessible due to client privacy concerns. Furthermore, the quality of therapists' responses in available sessions can vary significantly based on their professional training and experience. Assessing the quality of therapists' responses remains an open challenge. In this work, we address these challenges by first proposing a set of professional and comprehensive principles to evaluate therapists' responses to client speeches. Using these principles, we create a preference dataset, PsychoCounsel-Preference, which contains 36k high-quality preference comparison pairs. This dataset aligns with the preferences of professional psychotherapists, providing a robust foundation for evaluating and improving LLMs in psycho-counseling. Experiments on reward modeling and preference learning demonstrate that PsychoCounsel-Preference is an excellent resource for LLMs to acquire essential skills for responding to clients in a counseling session. Our best-aligned model, PsychoCounsel-Llama3-8B, achieves an impressive win rate of 87% against GPT-4o. We release PsychoCounsel-Preference, PsychoCounsel-Llama3-8B and the reward model PsychoCounsel Llama3-8B-Reward to facilitate the research of psycho-counseling with LLMs at: https://hf.co/Psychotherapy-LLM.