---
format: "aidr-story-markdown/v1"
id: "e664233dc8d06845133dfbad0ed4072c1d92276f9f6a7a1b3bb908b3c1102137"
canonical_url: "https://aidr.today/e664233d?lang=en"
title: "Quantize, Qualify, Rerank: A Recipe for Compressing LLMs Without Losing Quality"
lang: "en"
requested_lang: "en"
available_langs: ["en","vi"]
translation_fallback: null
fallback_fields: []
published_at: "2026-11-17T05:00:00.000Z"
category: "Research"
topics: ["google","llm"]
source_urls: ["https://arxiv.org/abs/2611.18587"]
summary: "Compressing large language models lowers serving cost, but a single automatic metric may miss quality loss. For the WMT26 Model Compression constrained track, we compress google/gemma-3-12b-it (24.4 GB in bf16) for three translation directions on one H100. Paired bootstrap analysis exposes metric disagreement on 4-bit quantization; guided by WMT24 human meta-evaluation, we use MetricX24-XXL as primary and CometKiwi-XXL as a cross-check. Removing the unused vision stack and non-task vocabulary and using int8 weights or fp8 activations causes no resolved loss, whereas int4 leaves a small residual. QLoRA healing overfits its calibration domain. A Cometoid best-of-4 reranker at $T=0.3$ improves both the 8-bit and 4-bit systems under WMT25 test set. On post-submission WMT26, both remain competitive with bf16 under reference-free metrics; the small difference between the reranked variants is treated as noise."
---

# Quantize, Qualify, Rerank: A Recipe for Compressing LLMs Without Losing Quality

> [Open the canonical story](<https://aidr.today/e664233d?lang=en>)

**Published:** 2026-11-17T05:00:00.000Z
**Category:** Research
**Topics:** google, llm

## Summary

Compressing large language models lowers serving cost, but a single automatic metric may miss quality loss\. For the WMT26 Model Compression constrained track, we compress google/gemma\-3\-12b\-it \(24\.4 GB in bf16\) for three translation directions on one H100\. Paired bootstrap analysis exposes metric disagreement on 4\-bit quantization; guided by WMT24 human meta\-evaluation, we use MetricX24\-XXL as primary and CometKiwi\-XXL as a cross\-check\. Removing the unused vision stack and non\-task vocabulary and using int8 weights or fp8 activations causes no resolved loss, whereas int4 leaves a small residual\. QLoRA healing overfits its calibration domain\. A Cometoid best\-of\-4 reranker at $T\=0\.3$ improves both the 8\-bit and 4\-bit systems under WMT25 test set\. On post\-submission WMT26, both remain competitive with bf16 under reference\-free metrics; the small difference between the reranked variants is treated as noise\.

## Sources

- [Story source](<https://arxiv.org/abs/2611.18587>)

