\scalerel*X: Towards General World Model with Natural Language Actions and Video States

🌈 Abstract

The paper presents \scalerelX, a step towards building a general world model that can simulate world states by generating videos across different domains and control the video generation with natural language actions. The model uses a staged training strategy that integrates existing pretrained language and video models, requiring only additional lightweight finetuning. \scalerelX demonstrates capabilities such as domain generality, video consistency, on-the-fly controllability, and action controllability transfer across domains.

🙋 Q&A

[01] Introduction

1. What are the key capabilities that a general world model should have? A general world model should have the following key capabilities:

• Consistency: It should generate consistent videos to accurately describe the world state.
• Controllability: It should allow on-the-fly control by accepting natural language actions at any time during video generation.
• Generality: It should perform well across diverse domains with different scenes and actions.

2. What are the limitations of current large language models (LLMs) and video generation models in building a general world model?

• LLMs are constrained by their reliance on language modality and limited understanding of the physical world.
• Video generation models lack interactive action control over the world simulations.

3. How does the \scalerel*X model address these limitations? \scalerel*X is a hybrid autoregressive-diffusion model that:

• Simulates world states by generating videos across different domains.
• Allows real-time control with free-text actions.
• Integrates a pretrained LLM and a pretrained video model, requiring only additional lightweight finetuning.

[02] Methods

1. What are the two core components of the \scalerel*X model architecture? The two core components are:

1. The autoregressive backbone, which stems from a pretrained LLM.
2. The video generator, which is initialized with a pretrained video model.

2. How does the model stitch these two components together? The model adds other necessary components, including a vision encoder and two adapters, to connect the vision encoder to the LLM backbone, and the LLM backbone to the video generator.

3. What is the two-stage training strategy used for \scalerel*X?

1. Pretraining stage:
   • Aims to acquire consistent general video generation, general text understanding, and alignment between the text and video components.
   • Reuses existing pretrained LLMs and video generation models.
2. Instruction tuning stage:
   • Trains the model on a curated video dataset with high-quality instructions (actions) to enhance the model's ability to follow natural language instructions and accurately predict subsequent video states.

[03] Qualitative Results

1. What are the key capabilities demonstrated by \scalerel*X in the qualitative results?

• On-the-fly control across diverse domains (indoor/outdoor, robot/human, 2D/3D games)
• Action controllability transfer to unseen domains
• Ability to generate longer videos in an autoregressive manner

2. What are some of the limitations and failure cases observed for \scalerel*X?

• The model can struggle to generate videos with high quality and good controllability, especially in domains where the data quality (precision of dynamics descriptions) is lower.
• Increasing the training compute helps mitigate some of these issues, indicating the potential for further enhancement with larger-scale training.

[04] Related Works

1. How does \scalerel*X differ from previous world models?

• Previous world models are usually designed for specific domains, while \scalerel*X aims to be a more general world model.
• \scalerel*X allows on-the-fly control with free-text actions, which is a key difference from previous text-to-video models.

2. How does \scalerel*X build upon recent advancements in video generation models?

• \scalerel*X integrates a pretrained video generation model with an autoregressive LLM backbone, enabling longer video generation and better controllability.
• This hybrid approach differs from previous video generation models that are based solely on diffusion architectures.