Cambrian

🌈 Abstract

The paper introduces Cambrian-1, a family of multimodal large language models (MLLMs) designed with a vision-centric approach. The key contributions are:

• Evaluating various visual representations, including self-supervised, strongly supervised, and combinations, through MLLM instruction tuning as an interface.
• Introducing the Cambrian Vision-Centric Benchmark (CV-Bench) by repurposing standard vision benchmarks into VQA format.
• Designing a new Spatial Vision Aggregator (SVA) connector that integrates high-resolution vision features with LLMs while reducing the number of tokens.
• Curating high-quality visual instruction-tuning data from publicly available sources, emphasizing data balancing and distribution ratio.
• Achieving state-of-the-art performance across diverse benchmarks and excelling in visual-centric tasks.

🙋 Q&A

[01] Multimodal LLMs: Preliminaries and Related Work

1. What are the key components of MLLM research? The key components of MLLM research include:

• Large Language Model
• Visual Encoder
• Multimodal Connector
• Data Curation Pipeline
• Instruction Tuning Strategy
• Evaluation & Benchmarking

2. What are the challenges in understanding the interactions between these components? Understanding the interactions between these components presents significant challenges, as each component has its own intricacies.

3. How does the paper investigate these aspects from a vision-centric perspective? The paper investigates these aspects from a vision-centric perspective, using MLLM instruction tuning as an interface to evaluate various visual representations.

[02] Evaluating Visual Representations through MLLMs

1. What are the limitations of existing vision benchmarks? The paper finds that most benchmarks do not properly measure vision-centric capabilities, and the ones that do have very few samples.

2. How does the paper address these limitations? The paper introduces the Cambrian Vision-Centric Benchmark (CV-Bench) by repurposing standard vision benchmarks into VQA format, providing significantly more examples than other vision-centric MLLM benchmarks.

3. What are the key findings from using MLLMs to evaluate visual representations? Key findings include:

• High-resolution encoders greatly enhance performance on chart and vision-centric benchmarks, and ConvNet-based architectures are well-suited for such tasks.
• Combining multiple vision encoders, including vision SSL models, enhances MLLM performance across various benchmarks, particularly in vision-centric tasks.

[03] Spatial Vision Aggregator (SVA): A New Connector Design

1. What are the two key design principles of the Spatial Vision Aggregator (SVA)? The two key design principles are:

1. Introducing spatial inductive bias by explicitly defining the aggregation space for each token in the query.
2. Aggregating vision features multiple times across the LLM layers, enabling the model to repeatedly access and integrate necessary visual information.

2. How does the SVA module perform compared to other aggregation approaches? The SVA module consistently outperforms other baselines, such as concatenation-based and Resampler approaches, and excels in aggregating high-resolution vision information.

[04] Instruction Tuning Data for Training MLLMs

1. What are the challenges in collecting multimodal (visual) instruction-tuning data? Unlike language data, multimodal (visual) instruction-tuning data is much rarer and harder to collect.

2. How does the paper address the data scarcity issue? The paper introduces a data engine to create large-scale, reliable, high-quality knowledge-based instruction tuning data from the internet, significantly increasing the diversity in the data pool.

3. What is the "answer machine phenomenon" and how does the paper address it? The "answer machine phenomenon" refers to a well-trained MLLM excelling at VQA benchmarks but lacking basic conversational abilities. The paper addresses this by incorporating additional system prompts during training, which improves the model's conversational ability while maintaining its benchmark performance.

[05] State of the Art Performance

1. How does Cambrian-1 compare to other leading MLLM frameworks? Cambrian-1 surpasses open-source models like LLaVA-NeXT and Mini-Gemini, and achieves competitive performance on a number of benchmarks compared to proprietary models such as GPT-4V, Gemini, and Grok-1.5.

2. What are the key advantages of Cambrian-1 compared to other models? Despite using only 576 visual tokens, Cambrian-1 performs better on OCR & Chart and Vision-Centric benchmarks compared to models like Mini-Gemini-HD and LLaVA-NeXT, which use 2880 tokens.