Researchers Reveal New Scaling Laws for Quadrillion Parameter AI Models

A team of computer scientists led by Ethan Baron has published a study detailing how large language models could be scaled to reach a capacity of one quadrillion parameters. The researchers developed a new set of data constrained scaling laws and learning algorithms to address the current inefficiency of AI models compared to human learning, proposing that current architectures are underparameterized by several orders of magnitude. The study finds that LLMs are roughly 1 million times less sample efficient than humans, noting that a human brain employs 100T synapses to learn from about 200M tokens of language. In contrast, the Chinchilla scaling law prescribes a 1T parameter model to be trained on 20T tokens, creating a gap of seven orders of magnitude in the ratio of parameters to tokens.