ResearchNov 17, 2026 · 12:001 source
Quantize, Qualify, Rerank: A Recipe for Compressing LLMs Without Losing Quality
From arxiv.org
Compressing large language models lowers serving cost, but a single automatic metric may miss quality loss. For the WMT26 Model Compression constrained track, we compress google/gemma-3-12b-it (24.4 GB in bf16) for three translation directions on one H100. Paired bootstrap analysis exposes metric disagreement on 4-bit quantization; guided by WMT24 human meta-evaluation, we use MetricX24-XXL as primary and CometKiwi-XXL as a cross-check. Removing the unused vision stack and non-task vocabulary and using int8 weights or fp8 activations causes no resolved loss, whereas int4 leaves a small residual. QLoRA healing overfits its calibration domain. A Cometoid best-of-4 reranker at $T=0.3$ improves both the 8-bit and 4-bit systems under WMT25 test set. On post-submission WMT26, both remain competitive with bf16 under reference-free metrics; the small difference between the reranked variants is treated as noise.
Sources
Content history (1)
- 2026-11-17 · Summary · vi · Replaced stored textNhóm tác giả đề xuất một quy trình gồm quantize, qualify và rerank để nén các LLMs mà vẫn…Nén LLMs giúp giảm chi phí vận hành, nhưng một chỉ số tự động duy nhất có thể bỏ sót việc…
One ranked edition, every morning
The stories that matter today. Free, cancel anytime.
Telegram @aidr_today