UnifiedMLLM: Enabling Unified Representation for Multi-modal Multi-tasks With Large Language Model

🌈 Abstract

The paper proposes UnifiedMLLM, a comprehensive multi-modal large language model (MLLM) that can handle various multi-modal tasks using a unified representation. The key aspects are:

• Unified Representation: The model generates task tokens and grounding tokens to represent different tasks and regions, enabling seamless integration of multiple tasks.
• Task Router and Expert Integration: The task tokens and grounding tokens are used to activate corresponding expert models to execute the specified tasks.
• Dataset Construction: The authors construct task-specific datasets and a 100k multi-task dataset with complex scenarios to train the model.
• Three-stage Training Strategy: The model is trained in three stages - modality-perception pretraining, task adaptation tuning, and multi-task LoRAMoE tuning to improve its reasoning and task processing capabilities.

The experiments demonstrate the model's impressive performance across a wide range of multi-modal tasks, including referring segmentation, reasoning editing, layout-based image generation, and multi-modality generation.

🙋 Q&A

[01] Unified Representation

1. What is the key aspect of the unified representation proposed in the paper? The key aspect of the unified representation is the introduction of task tokens and grounding tokens. The task tokens indicate the type of task to be executed, while the grounding tokens contain region-relative coordinates expressed in text format. This unified representation enables the model to handle different multi-modal tasks seamlessly.

2. How does the unified representation facilitate the integration of different tasks? The task tokens and grounding tokens allow the model to understand the implicit intent behind user instructions and output not only the textual response but also the special tokens indicating the task type and specific region to be processed. These tokens are then routed through a task router, which activates the corresponding expert model for task execution. This decoupling of the language model and expert models enables seamless integration of various tasks.

[02] Task Router and Expert Integration

1. How does the task router component work in the proposed model? The task router component utilizes the task tokens and grounding tokens generated by the language model to determine the type and region of the task to be executed. Based on this information, the task router activates the corresponding expert model to perform the specific task.

2. What are the different expert models integrated into the proposed system? The paper mentions the integration of various expert models, including:

• Stable Diffusion and GLIGEN for text-to-image generation and layout-based generation tasks
• Instructpix2pix and GLIGEN for image global editing and reasoning editing tasks
• SEEM for image and video segmentation tasks
• FRESCO for video editing tasks
• ModelScopeT2V and I2vgen-xl for text-based and image-based video generation, respectively
• Auffusion for audio generation

[03] Dataset Construction

1. What are the key datasets used for training the proposed model? The authors constructed two main datasets:

1. Task-specific datasets: These datasets were created by transforming task-relevant datasets into a conversation format, where the model's output includes task tokens and grounding tokens.
2. Multi-turn Multi-task Dataset: The authors generated a 100k multi-turn, multi-task dialogue dataset using advanced grounding models and GPT-3.5 to further enhance the model's understanding of human intent and reasoning abilities.

2. How did the authors leverage existing datasets to create the task-specific datasets? The authors selected task-relevant datasets, such as the reasoning segmentation dataset from LISA, the reasoning editing dataset from SmartEdit, and the layout-based image generation dataset from LayoutGPT, and transformed them into a conversation format with task tokens and grounding tokens.

[04] Training Strategy

1. What are the three stages of the training strategy employed by the authors? The three-stage training strategy is as follows:

1. Modality-perception Pretraining: The model is trained to acquire the ability to perceive and understand different modal inputs.
2. Task Adaptation Tuning: The model is trained on task-specific datasets to develop its capability to understand human intent and complete different tasks.
3. Multi-task LoRAMoE Tuning: The model is further optimized using the multi-turn, multi-task dataset to enhance its reasoning ability and enable it to complete a variety of tasks in complex scenarios.

2. How does the LoRAMoE training approach help in maintaining the model's knowledge and capabilities? The authors employ the LoRAMoE training approach, where the backbone of the model is frozen to preserve its capabilities, and multiple expert models are introduced to handle various downstream tasks. The experts are trained using a low-rank format (LoRA), which significantly reduces training costs, improves training speed, and avoids degradation of the model's knowledge and capabilities during the training process.

[05] Experimental Results

1. What are the key multi-modal tasks evaluated in the experiments? The paper evaluates the proposed model's performance on a variety of multi-modal tasks, including:

• Referring segmentation
• Reasoning editing
• Layout-based image generation
• Text-to-image generation
• Text-to-video generation
• Text-to-audio generation

2. How does the proposed model perform compared to other methods on these tasks? The experimental results show that the proposed UnifiedMLLM model outperforms existing methodologies across the evaluated multi-modal tasks. The model demonstrates strong capabilities in understanding human intent, performing reasoning, and effectively accomplishing a wide range of tasks.