ResearchSep 2, 2026 · 15:572 sources
SMELT MoE Model Cuts Training FLOPs 18% at 54B Parameter Scale
From huggingnews.com
The Sparse MoE Transformer, or SMELT, loops the middle half of its layers twice to reduce training FLOPs by 6.8% to 18.0% on the compute-optimal frontier. Researchers scaled the architecture to 54B non-embedding parameters using Chinchilla-style scaling laws. This block-level looping approach maintains parity for per-token FLOPs, total non-embedding parameters, and KV cache compared to unlooped baselines.
Read the sourceSign in to suggest
Sources
+ 1 more
- SUPPORT@teortaxestex“Loopies repeats individual layers; SMELT repeats on block level”x.com
#llm#training#moeMarkdown
Content history (1)
- 2026-10-04 · Summary · vi · Wording fixSMELT (Sparse MoE Transformer) hoạt động bằng cách lặp lại hai lần nửa số lớp ở giữa mạng…SMELT hoạt động bằng cách lặp lại hai lần nửa số lớp ở giữa mạng để đẩy nhanh tốc độ giảm…
One ranked edition, every morning
The stories that matter today. Free, cancel anytime.
Telegram @aidr_today