---
format: "aidr-story-markdown/v1"
id: "d432c0616da323aafd3391d290f006dc5fffc5ad660904be918796509c135933"
canonical_url: "https://aidr.today/d432c061?lang=en"
title: "CrisisBench: an evaluation suite for crisis counseling AI"
lang: "en"
requested_lang: "en"
available_langs: ["en","vi"]
translation_fallback: null
fallback_fields: []
published_at: "2026-11-18T05:00:00.000Z"
category: "Research"
topics: ["openai","llm"]
source_urls: ["https://arxiv.org/abs/2611.21622"]
summary: "Background: Crisis helplines are essential to mental health systems but are often understaffed and underfunded. LLMs could support counselors, but existing benchmarks do not capture the escalation, risk assessment, and de-escalation central to crisis conversations, while privacy restrictions limit use of real transcripts. Methods: We present CrisisBench, a privacy-preserving suite for single- and multi-turn crisis counseling evaluation. It includes 1,085 single-turn tasks from 200 de-identified conversations and 1,185 patient LLMs derived from hotline metadata. Patient models and a 23-item LLM judge were calibrated against expert counselors. Results: We evaluated five GPT models plus draft-then-edit and retrieval-augmented generation. All systems underperformed human counselors on excessive empathy and questioning in single-turn evaluation. The rubric achieved human-human Krippendorff's $\\alpha$ = 0.51 and LLM-human $\\alpha$ = 0.54 (Pearson r = 0.80). In multi-turn evaluation, retrieval-augmented generation scored highest overall (0.961), with all systems within 0.058."
---

# CrisisBench: an evaluation suite for crisis counseling AI

> [Open the canonical story](<https://aidr.today/d432c061?lang=en>)

**Published:** 2026-11-18T05:00:00.000Z
**Category:** Research
**Topics:** openai, llm

## Summary

Background: Crisis helplines are essential to mental health systems but are often understaffed and underfunded\. LLMs could support counselors, but existing benchmarks do not capture the escalation, risk assessment, and de\-escalation central to crisis conversations, while privacy restrictions limit use of real transcripts\. Methods: We present CrisisBench, a privacy\-preserving suite for single\- and multi\-turn crisis counseling evaluation\. It includes 1,085 single\-turn tasks from 200 de\-identified conversations and 1,185 patient LLMs derived from hotline metadata\. Patient models and a 23\-item LLM judge were calibrated against expert counselors\. Results: We evaluated five GPT models plus draft\-then\-edit and retrieval\-augmented generation\. All systems underperformed human counselors on excessive empathy and questioning in single\-turn evaluation\. The rubric achieved human\-human Krippendorff's $\\alpha$ \= 0\.51 and LLM\-human $\\alpha$ \= 0\.54 \(Pearson r \= 0\.80\)\. In multi\-turn evaluation, retrieval\-augmented generation scored highest overall \(0\.961\), with all systems within 0\.058\.

## Sources

- [Story source](<https://arxiv.org/abs/2611.21622>)

