---
format: "aidr-story-markdown/v1"
id: "f7cdc4ecbf75e3a327c8a12f18dbd9c31c8983173d6653f114a53fc138bd6de4"
canonical_url: "https://aidr.today/f7cdc4ec?lang=en"
title: "RAW: Refusal Aversion in the Wild, A Causal Measurement Method for Deployed LLMs"
lang: "en"
requested_lang: "en"
available_langs: ["en","vi"]
translation_fallback: null
fallback_fields: []
published_at: "2026-11-18T05:00:00.000Z"
category: "Research"
topics: ["llm"]
source_urls: ["https://arxiv.org/abs/2611.21641"]
summary: "Deployed LLMs field hundreds of millions of conversations a day and refuse a non-trivial fraction of user requests, yet the user-side cost of refusal, whether users disengage and by how much, is unmeasured on any production model: a blind spot for the consumer-facing platforms built on them. A/B-testing refusal is ethically and operationally constrained on frontier-lab products and unavailable to third-party platforms deploying these models through a fixed API. We introduce RAW, a method that uses LLM stochasticity as a natural experiment: at near-identical prompts in any existing log, the same model sometimes refuses and sometimes does not, and that variation identifies the causal user-side effect of refusal. Applying the method to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, with user-clustered inference, we find that for about half of the models we can measure precisely, a refusal costs 3 to 17 pp of subsequent re-engagement; the rest show no distinguishable effect, among them one deployment whose positive point estimate we trace to persistent users retrying the refused request."
---

# RAW: Refusal Aversion in the Wild, A Causal Measurement Method for Deployed LLMs

> [Open the canonical story](<https://aidr.today/f7cdc4ec?lang=en>)

**Published:** 2026-11-18T05:00:00.000Z
**Category:** Research
**Topics:** llm

## Summary

Deployed LLMs field hundreds of millions of conversations a day and refuse a non\-trivial fraction of user requests, yet the user\-side cost of refusal, whether users disengage and by how much, is unmeasured on any production model: a blind spot for the consumer\-facing platforms built on them\. A/B\-testing refusal is ethically and operationally constrained on frontier\-lab products and unavailable to third\-party platforms deploying these models through a fixed API\. We introduce RAW, a method that uses LLM stochasticity as a natural experiment: at near\-identical prompts in any existing log, the same model sometimes refuses and sometimes does not, and that variation identifies the causal user\-side effect of refusal\. Applying the method to 15 deployed LLMs in LMSYS\-Chat\-1M and WildChat, with user\-clustered inference, we find that for about half of the models we can measure precisely, a refusal costs 3 to 17 pp of subsequent re\-engagement; the rest show no distinguishable effect, among them one deployment whose positive point estimate we trace to persistent users retrying the refused request\.

## Sources

- [Story source](<https://arxiv.org/abs/2611.21641>)

