OpenESM: Energy-Steered Models Are Scalable Foundation Learners

Researchers introduce energy-steered models (ESMs) that integrate energy steering and token generation in a single architecture for foundational language modeling. Pre-trained on large-scale corpora, ESMs are evaluated across 26 downstream tasks and reportedly outperform GPTs on most, demonstrating promising scaling behavior with increasing model size and training data.