Dust: Pretraining Transformers Without Backpropagation

A paper titled "Dust: Pretraining Transformers Without Backpropagation" proposes an alternative to backpropagation for training transformer-based language models, suggesting that more generic and brute-force learning algorithms based on search may eventually scale better with compute.