SoccerNet Game State Reconstruction: End-to-End Athlete Tracking and Identification on a Minimap

🌈 Abstract

The paper introduces the concept of Game State Reconstruction (GSR), a novel computer vision task for tracking and identifying all athletes on a minimap of the sports pitch. To support research on this task, the authors publicly release the SoccerNet-GSR dataset, the first dataset for Game State Reconstruction, consisting of 200 30-second fully annotated clips. The paper also introduces GS-HOTA, a new evaluation metric to benchmark GSR methods, and proposes GSR-Baseline, the first end-to-end and open-source pipeline for game state reconstruction.

🙋 Q&A

[01] Introduction

1. What is the motivation behind the Game State Reconstruction task?

• Tracking and identifying athletes on the pitch holds a central role in collecting essential insights from the game, such as estimating the total distance covered by players or understanding team tactics.
• Reconstructing the game state from videos captured by a single camera is challenging as it requires understanding the position of the athletes and the viewpoint of the camera to localize and identify players within the field.

2. What are the key limitations of existing approaches?

• Manual generation of athlete tracking and identification data by human annotators is time-consuming and costly.
• Sensor-based solutions require athletes to wear special, sometimes expensive, equipment.
• Automatic solutions based on optical tracking systems require sophisticated, well-calibrated static multi-camera setups in stadiums, which is costly and not scalable.
• Existing Multi-Object Tracking (MOT) methods offer only a partial solution as they lack critical identification information and grounding in a real-world coordinate system.

3. How does the proposed Game State Reconstruction task address these limitations?

• GSR aims to recognize the state of a sports game by identifying and tracking all athletes on the pitch based on input videos captured by a single camera.
• The game state data can be visualized in a minimap of the game, offering a concise representation of the ongoing gameplay dynamics.

[02] SoccerNet-GSR Dataset

1. What are the key components of the SoccerNet-GSR dataset?

• The dataset includes over 9.37 million line points for football pitch registration and camera calibration, and over 2.36 million athlete positions on the pitch with unique identification information, including their role, team, and jersey number.
• The videos are uncut broadcast sequences captured by a single moving camera, where only a portion of the football pitch is visible at any given time.

2. How does the SoccerNet-GSR dataset differ from the original SoccerNet-Tracking dataset?

• The original SoccerNet-Tracking dataset lacked information like pitch localization, camera calibration, and athlete positions on the pitch, which are critical for the Game State Reconstruction task.
• The new SoccerNet-GSR dataset provides these additional annotations to support research on the GSR task.

[03] GS-HOTA Evaluation Metric

1. Why are standard MOT evaluation metrics not suitable for the GSR task?

• Standard MOT metrics like MOTA and HOTA do not account for the additional attributes predicted on the tracked targets, such as team, role, and jersey numbers.
• They rely on an IoU score to match predicted and ground truth bounding boxes in the image space, while GSR operates on 2D points within the pitch coordinate system.

2. How does the proposed GS-HOTA metric address these limitations?

• GS-HOTA introduces a new similarity score that combines localization similarity in the pitch coordinate system and identification similarity based on the predicted attributes (role, team, jersey number).
• The metric also considers the impact of non-uniquely identifiable targets, which can occur when athletes share the same attributes.

[04] GSR Baseline

1. What is the overall architecture of the GSR-Baseline pipeline?

• The pipeline splits the GSR task into several sub-tasks, including athlete detection and tracking, pitch localization and camera calibration, and athlete identification (role, team, jersey number).
• It leverages state-of-the-art methods for each sub-task, such as YOLOv8 for detection, StrongSORT for tracking, PRTreID for re-identification, and MMOCR for jersey number recognition.

2. What are the key challenges and limitations of the GSR-Baseline?

• The ablation study reveals that the pipeline struggles the most with jersey number recognition, followed by pitch localization, team affiliation, and role classification.
• The complete GSR-Baseline pipeline achieves a GS-HOTA score of 22.26 on the test set, highlighting the complexity of the GSR task and the need for further research.

3. What is the inference time of the GSR-Baseline?

• The pipeline takes on average 6 minutes to process a 30-second video sequence, with the pitch localization, camera calibration, and jersey-number recognition modules being the most time-consuming.

[05] Conclusion

1. What are the key contributions of this work?

• Introduction of the Game State Reconstruction (GSR) task, a novel computer vision task for tracking and identifying all athletes on a minimap of the sports pitch.
• Release of the SoccerNet-GSR dataset, the first open-source dataset for the GSR task.
• Proposal of the GS-HOTA evaluation metric to benchmark GSR methods.
• Introduction of the GSR-Baseline, the first end-to-end and open-source pipeline for game state reconstruction.

2. What are the future research directions highlighted by this work?

• Enhancing specific modules of the GSR-Baseline to increase performance.
• Implementing real-time GSR pipelines.
• Developing end-to-end differentiable methods for tackling the GSR task in a single step.