ResearchSep 2, 2026 · 15:572 sources

SMELT MoE Model Cuts Training FLOPs 18% at 54B Parameter Scale

From huggingnews.com

Image: huggingnews.com
AI;DR

The Sparse MoE Transformer, or SMELT, loops the middle half of its layers twice to reduce training FLOPs by 6.8% to 18.0% on the compute-optimal frontier. Researchers scaled the architecture to 54B non-embedding parameters using Chinchilla-style scaling laws. This block-level looping approach maintains parity for per-token FLOPs, total non-embedding parameters, and KV cache compared to unlooped baselines.

Read the sourceSign in to suggest

Sources

+ 1 more
  1. SUPPORT@teortaxestex“Loopies repeats individual layers; SMELT repeats on block level”x.com
#llm#training#moeMarkdown
Content history (1)
  • 2026-10-04 · Summary · vi · Wording fix
    SMELT (Sparse MoE Transformer) hoạt động bằng cách lặp lại hai lần nửa số lớp ở giữa mạng…
    SMELT hoạt động bằng cách lặp lại hai lần nửa số lớp ở giữa mạng để đẩy nhanh tốc độ giảm…

One ranked edition, every morning

The stories that matter today. Free, cancel anytime.

Telegram @aidr_today