Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioning

🌈 Abstract

The paper proposes an interaction-aware trajectory-conditioned long-term multi-agent human pose forecasting model, called Trajectory2Pose (T2P). The key ideas are:

• Decoupling the overall human motion into global trajectories and local poses, and using a coarse-to-fine strategy to first forecast multi-modal global trajectory proposals, then condition local pose predictions on each trajectory mode.
• Introducing a graph-based agent-wise interaction module to enable reciprocal forecasting of local motion-conditioned global trajectory and trajectory-conditioned local pose.
• Addressing the lack of long-term (6s+) multi-agent (5+) datasets by constructing a new dataset, JRDB-GlobMultiPose, from real-world images and 2D annotations.

The proposed T2P model achieves state-of-the-art performance on both the new JRDB-GlobMultiPose dataset and previous benchmark datasets, demonstrating its generalized effectiveness in complex long-term multi-agent environments.

🙋 Q&A

[01] Proposed Model

1. What are the key components of the proposed Trajectory2Pose (T2P) model?

• The T2P model decouples the overall human motion into global trajectories and local poses, and uses a coarse-to-fine strategy.
• It first forecasts multi-modal global trajectory proposals, then conditions local pose predictions on each trajectory mode.
• The model introduces a graph-based agent-wise interaction module to enable reciprocal forecasting of local motion-conditioned global trajectory and trajectory-conditioned local pose.

2. How does the T2P model handle the multi-modality of human motion and the complexity of long-term multi-agent interactions?

• By decoupling the overall motion into global trajectories and local poses, and using a coarse-to-fine approach, the T2P model is able to effectively handle the multi-modality of human motion.
• The graph-based agent-wise interaction module allows the model to capture the complex interactions between agents in long-term multi-agent environments.

3. What are the key advantages of the T2P model compared to previous methods?

• The T2P model outperforms previous state-of-the-art methods on both complex and simpler datasets, demonstrating its generalized effectiveness.
• It is able to handle longer prediction horizons (6s+) and more agents (5+) compared to previous methods.

[02] JRDB-GlobMultiPose Dataset

1. Why was the new JRDB-GlobMultiPose dataset created?

• Previous datasets lacked long-term (6s+) and multi-agent (5+) scenarios, which are essential for fully leveraging human pose forecasting in real-world applications.
• The authors constructed the JRDB-GlobMultiPose dataset from real-world images and 2D annotations to enable a comprehensive evaluation of their proposed T2P model in complex environments.

2. How was the 3D human pose data extracted for the JRDB-GlobMultiPose dataset?

• The authors used a state-of-the-art monocular 3D pose estimation method (BEV) to extract raw 3D joint positions from the image sequences.
• They then refined the 3D poses using the 2D pose and 3D bounding box annotations provided in the original JRDB dataset to ensure the accuracy of the 3D pose information.

3. What are the key characteristics of the JRDB-GlobMultiPose dataset?

• The dataset contains up to 24 agents and forecasts up to 5 seconds of motion, providing a more realistic and challenging environment for long-term multi-agent human pose forecasting.
• It includes diverse human motions and rich inter-agent interactions, which serve as valuable cues for the T2P model to learn complex spatio-temporal dynamics.