---
format: "aidr-story-markdown/v1"
id: "26add373fef7eefaef9817dae64d0f347538248ce1e0afdecbd252ec8ab2dacc"
canonical_url: "https://aidr.today/26add373?lang=en"
title: "Same Output, Different Gold: Measuring How Reference Choice Moves a Multilingual Benchmark Score"
lang: "en"
requested_lang: "en"
available_langs: ["en","vi"]
translation_fallback: null
fallback_fields: []
published_at: "2026-10-06T04:00:00.000Z"
category: "Research"
topics: ["llm"]
source_urls: ["https://arxiv.org/abs/2610.03825"]
summary: "A six-language benchmark study found that changing the reference annotation while keeping system output fixed can shift scores by up to 4.95 F1 points, with the effect attributed to an undocumented aggregation default that made the shipped reference partially match one of the outputs being scored."
---

# Same Output, Different Gold: Measuring How Reference Choice Moves a Multilingual Benchmark Score

> [Open the canonical story](<https://aidr.today/26add373?lang=en>)

**Published:** 2026-10-06T04:00:00.000Z
**Category:** Research
**Topics:** llm

## Summary

A six\-language benchmark study found that changing the reference annotation while keeping system output fixed can shift scores by up to 4\.95 F1 points, with the effect attributed to an undocumented aggregation default that made the shipped reference partially match one of the outputs being scored\.

## Sources

- [Story source](<https://arxiv.org/abs/2610.03825>)