Building and better understanding vision-language models: insights and future directions

🌈 Abstract

The paper provides a comprehensive overview of the current state-of-the-art approaches in the field of vision-language models (VLMs), which take images and texts as inputs and output texts. It highlights the key aspects of the VLM development pipeline, including data, architecture, and training methods, and suggests promising research directions. The paper then details the practical steps taken to build Idefics3-8B, a powerful VLM that significantly outperforms its predecessor Idefics2-8B, while being trained efficiently, exclusively on open datasets, and using a straightforward pipeline. This includes the creation of the Docmatix dataset, a large-scale document understanding dataset, to boost the model's capabilities in this area.

🙋 Q&A

[01] Architectural Choices in VLMs

1. What are the two main types of architectures used to connect pre-trained language models with vision encoders? The two main types of architectures are:

• Cross-attention architecture: The image hidden states are used to condition the frozen language model using freshly initialized cross-attention layers.
• Self-attention architecture: The output of the vision encoder is treated as tokens and concatenated to the sequence of text tokens, which are then passed as input to the language model.

2. Which architecture performs better, and under what conditions? The cross-attention architecture significantly outperforms the self-attention architecture when the backbones are kept frozen during training. However, when parts of the vision encoder and language model are trained with LoRA, the self-attention architecture performs better despite having fewer trainable parameters.

3. How do the pre-trained backbones impact the performance of the resulting VLM? The performance of the standalone unimodal pre-trained backbones (language model and vision encoder) correlates with the performance of the resulting VLM. Replacing the backbones with higher-performing models can lead to substantial improvements across benchmarks.

4. What are the challenges with the image-splitting strategy, and what are the potential improvements? The image-splitting strategy, which divides the original image into multiple sub-images, can lead to a loss of global context. A potential improvement is to develop a vision encoder that can natively process images of varying resolutions, including very large ones, without changing the original aspect ratios, and efficiently handle long-context.

[02] Training Methods and Datasets for VLMs

1. What are the key stages in the typical VLM training process? The training process typically involves multiple stages:

• Pre-training: Aligning the backbone models and training the newly initialized parameters using large-scale datasets.
• Supervised fine-tuning (SFT): Training on curated datasets covering a wide range of tasks.
• Alignment phase: Aligning the model's output with human preferences, reducing hallucinations, and enhancing safety.

2. What are the main types of datasets used in the VLM training process, and what are their strengths and weaknesses?

• Image-text pairs: Easily collected but often noisy and ungrammatical. Recent approaches use synthetic re-captioning to improve quality.
• Interleaved image-text documents: Enhance in-context learning and expose the model to a wider distribution of texts, but limited in scale.
• PDF documents: Provide high-quality text transcriptions, but limited in diversity.
• Synthetic data: Can be tailored to include examples aligned with user tasks, but generating high-quality data for complex tasks like document understanding remains challenging.

3. How did the authors create the Docmatix dataset to enhance document understanding capabilities? The authors used text transcriptions from the PDFA dataset and employed an LLM to generate QA pairs, filtering out low-quality outputs. This resulted in a dataset of 1.3M documents and 9.5M QA pairs, a 240-fold increase in scale compared to previous open datasets.

[03] Challenges in Evaluating VLMs

1. What are the challenges with open-ended and multiple-choice benchmarks for evaluating VLMs? Open-ended benchmarks tend to favor models that produce answers closely aligned with the benchmark's expected format or writing style, rather than measuring true understanding. Multiple-choice benchmarks can reduce this bias, but still face challenges with potential contamination and overoptimization.

2. Why is there a discrepancy between the performance of VLMs during pre-training and after fine-tuning? During pre-training, the model only starts learning the specific task of visual question answering (beyond just image captioning or text transcription). The impact of development choices in the VLM may only become evident after the fine-tuning stage, leading to a delayed feedback loop.

[04] Building Idefics3

1. What are the key improvements in Idefics3 compared to its predecessor, Idefics2? Idefics3 benefits from:

• More visual tokens per image, using a simple pixel shuffle strategy instead of the perceiver resampler
• A third stage of pre-training on large high-quality synthetic datasets
• An improved language model backbone (Llama 3.1)

These enhancements led to significant improvements across various tasks, particularly in document understanding tasks, with a 13.7-point boost on the DocVQA benchmark.

2. How did the authors create the Docmatix dataset, and how did it impact the performance of VLMs? The authors used text transcriptions from the PDFA dataset and employed an LLM to generate QA pairs, filtering out low-quality outputs. This resulted in a dataset of 1.3M documents and 9.5M QA pairs, a 240-fold increase in scale compared to previous open datasets. Experiments showed that training on a small portion of Docmatix led to a nearly 20% relative improvement on the DocVQA benchmark for the Florence-2 model.