On the Design of Hyper-Connections: Simplification, Stability, and Scalability

Hyper-Connections (HC) architectures are adopted in frontier language models but their components are insufficiently understood and introduce training overhead. Learned read/write gating without mixing and unconstrained mixing with fixed gates form two effective interaction groups. SimpleHC simplifies the architecture by removing residual mixing and uses shared memory access.