ResearchNov 17, 2026 · 12:001 source

Quantize, Qualify, Rerank: A Recipe for Compressing LLMs Without Losing Quality

From arxiv.org

Image: arxiv.org
AI;DR

Compressing large language models lowers serving cost, but a single automatic metric may miss quality loss. For the WMT26 Model Compression constrained track, we compress google/gemma-3-12b-it (24.4 GB in bf16) for three translation directions on one H100. Paired bootstrap analysis exposes metric disagreement on 4-bit quantization; guided by WMT24 human meta-evaluation, we use MetricX24-XXL as primary and CometKiwi-XXL as a cross-check. Removing the unused vision stack and non-task vocabulary and using int8 weights or fp8 activations causes no resolved loss, whereas int4 leaves a small residual. QLoRA healing overfits its calibration domain. A Cometoid best-of-4 reranker at $T=0.3$ improves both the 8-bit and 4-bit systems under WMT25 test set. On post-submission WMT26, both remain competitive with bf16 under reference-free metrics; the small difference between the reranked variants is treated as noise.

Read the sourceSign in to suggest

Sources

#google#llmMarkdown
Content history (1)
  • 2026-11-17 · Summary · vi · Replaced stored text
    Nhóm tác giả đề xuất một quy trình gồm quantize, qualify và rerank để nén các LLMs mà vẫn…
    Nén LLMs giúp giảm chi phí vận hành, nhưng một chỉ số tự động duy nhất có thể bỏ sót việc…

One ranked edition, every morning

The stories that matter today. Free, cancel anytime.

Telegram @aidr_today