Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE

Researchers proposed Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE, finding that inter-layer shared Meta Experts reduce parameter redundancy and improve cross-entropy loss and downstream task accuracy compared to hierarchical Local Experts. Their model matches Vanilla MoE performance using only 70-80% of the experts when activated quantity and expert granularity are held fixed.