MoVA: Adapting Mixture of Vision Experts to Multimodal Context

🌈 Abstract

The paper proposes MoVA, a powerful multimodal large language model (MLLM) that adaptively routes and fuses task-specific vision experts based on multimodal context and model expertise. The key contributions are:

1. Analyzing the performance of individual vision encoders versus the plain fusion of multiple encoders, revealing that the inherent bias of each vision encoder can diminish its generalization ability across other irrelevant domains.

2. Proposing MoVA, which consists of a coarse-grained context-aware expert routing and a fine-grained expert fusion with a Mixture-of-Vision-Expert Adapter (MoV-Adapter). This allows MoVA to fully leverage representations from multiple context-relevant vision encoder experts while avoiding biased information from irrelevant experts.

3. Demonstrating the effectiveness of each component in MoVA through ablation studies, and showing that MoVA can achieve significant performance gains over current state-of-the-art methods in a wide range of challenging benchmarks.

🙋 Q&A

[01] Overview

1. What is the key idea behind multimodal large language models (MLLMs)? The key idea behind MLLMs is projecting the vision encoder representation into a large language model (LLM) through a projector, facilitating a general-purpose multimodal understanding.

2. What are the limitations of using a single vision encoder like CLIP in MLLMs? The CLIP vision encoder, while widely used, exhibits inconsistent performance across tasks and scenarios due to its training data and optimization target. MLLMs with a single CLIP vision encoder usually perform poorly on fine-grained tasks such as grounding and optical character recognition (OCR).

3. What is the proposed solution to address the limitations of using a single vision encoder? The paper proposes MoVA, a MLLM that adaptively routes and fuses task-specific vision experts based on multimodal context and model expertise, in order to fully leverage representations from multiple context-relevant vision encoder experts while avoiding biased information from irrelevant experts.

[02] Coarse-grained Context-aware Expert Routing

1. How does the context-aware expert routing strategy work in MoVA? The context-aware expert routing strategy employs the reasoning capacity of the LLM to select vision experts with strong relevance to the user's image and instruction from a pool of expert models. This is done in three steps: 1) converting the input image, user questions, and expert model descriptions into prompts for the LLM to perform expert selection, 2) feeding the prompts to the LLM to generate the output text indicating the selected experts, and 3) parsing the output to determine the relevant experts for the fine-grained expert fusion stage.

2. How are the routing annotations constructed for training the expert routing component? The routing annotations are constructed by computing the language modeling loss of each sample on models with different vision experts, and selecting the experts that result in the lowest loss as the relevant ones for that sample.

[03] Fine-grained Expert Fusion with MoV-Adapter

1. What is the structure of the MoV-Adapter module? The MoV-Adapter consists of adapter blocks and a text encoder. Each adapter block contains an expert knowledge extractor, a dynamic gating network, and a transformer block. The expert knowledge extractor uses cross-attention layers to extract task-specific knowledge from the selected expert features, and the dynamic gating network computes expert-wise soft weights to integrate the extracted knowledge.

2. How does the MoV-Adapter improve the visual representation? The MoV-Adapter enhances the visual representation by extracting and integrating task-specific knowledge from the selected experts based on the multimodal context. The cross-attention layers in the expert knowledge extractor and the dynamic gating network allow for a fine-grained fusion of the expert representations.

[04] Training and Evaluation

1. What are the key stages in the training process of MoVA? The training of MoVA consists of three stages: 1) MoV-Adapter pretraining on diverse visual instruction samples, 2) supervised finetuning on high-quality visual instruction data, and 3) expert-routing LoRA training to improve the efficiency and effectiveness of the expert routing component.

2. How does MoVA perform on the evaluated benchmarks compared to other MLLM models? MoVA achieves significant performance gains over current state-of-the-art MLLM methods across a wide range of challenging benchmarks, including MLLM benchmarks, visual question answering, visual grounding, image segmentation, and biomedical understanding.