Law of Vision Representation in MLLMs

🌈 Abstract

The article presents the "Law of Vision Representation" in multimodal large language models (MLLMs), which reveals a strong correlation between the combination of cross-modal alignment, correspondence in vision representation, and MLLM performance. The authors quantify these two factors using the cross-modal Alignment and Correspondence (AC) score, and find that the AC score is linearly correlated to model performance. By leveraging this relationship, they are able to identify and train the optimal vision representation without requiring finetuning the language model every time, resulting in a 99.7% reduction in computational cost.

🙋 Q&A

[01] Law of Vision Representation in MLLMs

1. What are the key factors that impact MLLM benchmark performance according to the "Law of Vision Representation"? The "Law of Vision Representation" states that the performance of a MLLM can be estimated by two factors: cross-modal alignment (A) and correspondence (C) of the vision representation.

2. How do high cross-modal alignment and accurate correspondence in vision representation lead to improved MLLM performance?

• High cross-modal alignment allows the pretrained language model to require less computational effort to bridge the gap between different modalities during finetuning, as the vision embedding distribution is closely aligned with the text embedding distribution.
• Accurate correspondence in the vision representation ensures precise attention within the image embeddings, enabling the MLLM to retrieve more visual details that may not be directly attended to by the text tokens.

3. How is the AC score defined and calculated to quantify the cross-modal alignment and correspondence?

• The alignment score (A) is calculated as the maximum cosine similarity between each pair of embedding vectors from the CLIP embedding and the target vision representation embedding.
• The correspondence score (C) is calculated as the Percentage of Correct Keypoints (PCK) using ground truth keypoints on the SPair-71k dataset.
• The AC score is a second-degree polynomial transformation of the A and C scores.

[02] AC Policy for Efficient Optimal Vision Representation Selection

1. What is the problem formulation and goal of the AC policy? The goal is to efficiently estimate the optimal vision representation from a search space of vision representations, without the need to finetune the language model for each representation.

2. How does the AC policy work?

• The policy fits a linear regression model using the AC scores of a subsampled set of vision representations as input, and the downstream performance as the target.
• The sampling strategy ensures diverse selection of data points by dividing the normalized A and C score space into regions and sampling from the remaining regions.

3. How effective is the AC policy compared to random selection? The AC policy consistently predicts the optimal vision representation with minimal resources, achieving over 89% Recall@3 while only requiring 3.88 full training runs on average, compared to needing 12 out of 13 runs for random selection to reach a similar performance.

[03] Limitations and Future Work

1. How can vision representations with high AC scores be identified? Two strategies are suggested:

• Increasing the resolution of well-aligned features to enhance correspondence
• Combining features with high A and C scores along the channel dimension to leverage the strengths of both

2. What are the limitations in the current AC score design?

• The A score calculation can be affected by differences in input resolution between the target encoder and the CLIP reference.
• The C score computed using the SPair-71k dataset may not accurately capture correspondence for images containing text, which is important for OCR-based benchmarks.

3. What future work is suggested to address the limitations?

• Exploring alternative reference models for cross-modal alignment beyond CLIP
• Developing OCR-specific correspondence datasets to better evaluate vision representations for text-heavy images

</output_format>