---
format: "aidr-story-markdown/v1"
id: "979ef42f6f753f3301bccab0f91d2228e1743d95a2ed031e1d1bf7bcfe4337e2"
canonical_url: "https://aidr.today/979ef42f?lang=en"
title: "Process then Retrieve: Investigating the Origins of the Modality Gap in Vision-Language Models"
lang: "en"
requested_lang: "en"
available_langs: ["en","vi"]
translation_fallback: null
fallback_fields: []
published_at: "2026-11-10T05:00:00.000Z"
category: "Research"
topics: ["llm"]
source_urls: ["https://arxiv.org/abs/2611.09897"]
summary: "A study investigates the modality gap in vision-language models by examining differences in information density between text and visual tokens at the embedding layer. The research finds that text tokens have lower semantic density than visual tokens, causing early layers to prioritize text processing before combining modalities. Mechanistic analysis shows that replacing vision token positions with semantically closest text tokens increases attention to those positions in early layers."
---

# Process then Retrieve: Investigating the Origins of the Modality Gap in Vision\-Language Models

> [Open the canonical story](<https://aidr.today/979ef42f?lang=en>)

**Published:** 2026-11-10T05:00:00.000Z
**Category:** Research
**Topics:** llm

## Summary

A study investigates the modality gap in vision\-language models by examining differences in information density between text and visual tokens at the embedding layer\. The research finds that text tokens have lower semantic density than visual tokens, causing early layers to prioritize text processing before combining modalities\. Mechanistic analysis shows that replacing vision token positions with semantically closest text tokens increases attention to those positions in early layers\.

## Sources

- [Story source](<https://arxiv.org/abs/2611.09897>)