Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation

Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation (Road MOPD) introduces non-routed teachers as auxiliary cross-domain critics on student trajectories and coordinates multi-teacher supervision through token-level credit assignment and gradient updates to address gradient conflicts and capability degradation during joint optimization.