Q Labs Pretrains Transformers 4,000x Faster Without Backpropagation

Q Labs researchers developed a zeroth-order optimization method called the Noise speedrun that trains AI transformers 4,000x faster than backpropagation, reducing training time from 1.76 hours to 1.6 seconds. The Dust algorithm outperforms traditional backpropagation and Adam on validation loss and is up to 10,000x more compute-efficient than previous zeroth-order methods. The team is testing scalability by expanding the training dataset from 10M to 30M tokens.