---
format: "aidr-story-markdown/v1"
id: "f7c372bd59cea80a2d4e9bd6260a716bbf90747f6402c0947f06032bd0395517"
canonical_url: "https://aidr.today/f7c372bd?lang=en"
title: "Weight-Space Programming Doesn't Need Transformers"
lang: "en"
requested_lang: "en"
available_langs: ["en","vi"]
translation_fallback: null
fallback_fields: []
published_at: "2026-11-18T00:00:00.000Z"
category: "Research"
topics: ["llm"]
source_urls: ["https://huggingface.co/papers/2611.22055"]
summary: "Standard neural architectures such as the Transformer were chosen because they train well with gradient descent and run fast on GPUs. In weight-space programming, a large language model (LLM) writes target weights by hand without gradient-based training, so those reasons no longer apply. Should we keep the Transformer? Given minimal guidance, ten labeled examples, and a sandbox, mainly on multiple-choice questions, we task three frontier LLMs with either generating the weights of a fixed architecture or designing the architecture themselves. When free to design, all three compile plain-text facts into a sparse table queried by a fixed procedure, reaching up to 54.4% test accuracy. An LLM-designed model achieves higher test accuracy than the most accurate released Transformer that the same LLM wrote while using about 1,800x fewer parameters. Without gradients, they choose a table, in which the LLM can state what each weight means and can find and rewrite a wrong one. Each wrong answer points to one row, and removing a row changes about one other answer. From this, we propose an associative-memory-based model up to 17x faster on GPUs than a same-sized Transformer."
---

# Weight\-Space Programming Doesn't Need Transformers

> [Open the canonical story](<https://aidr.today/f7c372bd?lang=en>)

**Published:** 2026-11-18T00:00:00.000Z
**Category:** Research
**Topics:** llm

## Summary

Standard neural architectures such as the Transformer were chosen because they train well with gradient descent and run fast on GPUs\. In weight\-space programming, a large language model \(LLM\) writes target weights by hand without gradient\-based training, so those reasons no longer apply\. Should we keep the Transformer? Given minimal guidance, ten labeled examples, and a sandbox, mainly on multiple\-choice questions, we task three frontier LLMs with either generating the weights of a fixed architecture or designing the architecture themselves\. When free to design, all three compile plain\-text facts into a sparse table queried by a fixed procedure, reaching up to 54\.4% test accuracy\. An LLM\-designed model achieves higher test accuracy than the most accurate released Transformer that the same LLM wrote while using about 1,800x fewer parameters\. Without gradients, they choose a table, in which the LLM can state what each weight means and can find and rewrite a wrong one\. Each wrong answer points to one row, and removing a row changes about one other answer\. From this, we propose an associative\-memory\-based model up to 17x faster on GPUs than a same\-sized Transformer\.

## Sources

- [Story source](<https://huggingface.co/papers/2611.22055>)

