← Back to live feed1 story
GPU optimizations merged into the llama.cpp library increase speculative decoding speed on M3 Ultra hardware from 32.1 tok/s to 110 tok/s. Metal kernels were also extended to cover all 26 weight formats supported by the Mac GPU, allowing matrix multiplies to run up to 4.4x faster.
Sign in to suggest edits
Key sources
- SOURCE@qvac“On an M3 Ultra it now runs up to 3.4x faster than plain decoding, 110 tok/s against 32.1”x.com
- SOURCE@qvac“the fast Metal kernels from Monday now cover all 26 weight formats the Mac GPU runs”x.com
- SUPPORT@seraandroid“Merged to main in a few minutes!”x.com
- SOURCEmarketbrief.now