Process then Retrieve: Investigating the Origins of the Modality Gap in Vision-Language Models

A study investigates the modality gap in vision-language models by examining differences in information density between text and visual tokens at the embedding layer. The research finds that text tokens have lower semantic density than visual tokens, causing early layers to prioritize text processing before combining modalities. Mechanistic analysis shows that replacing vision token positions with semantically closest text tokens increases attention to those positions in early layers.