Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation

Researchers propose Factor-Aware Predictive Learning (FAPL), a method that uses multimodal large language model latents to improve cinematic video generation by enabling temporal inference from sparse inputs and factor-aware control of character, scene, action, and camera. The approach combines latent-only generator grounding with factor-span replacement training to strengthen a Video Diffusion Transformer's use of MLLM latents for generation.