Theoretical Limits of Language Model Alignment

A study titled Theoretical Limits of Language Model Alignment derives the maximum achievable expected reward gain for a fixed KL-divergence budget using a Jeffreys divergence term, and shows that reward ensembling mitigates reward hacking in proxy reward settings.