Will Scaling Solve Robotics?

🌈 Abstract

The article discusses the debate around whether training large neural networks on large datasets is a feasible solution for solving robotics problems. It presents the key arguments on both sides of the debate, as well as some related points and potential ways forward.

🙋 Q&A

[01] Why Scaling Might Work

1. Questions related to the content of the section?

• The argument that scaling worked for computer vision and natural language processing, so it could also work for robotics
• Evidence that scaling is starting to show promising results in robotics, such as the RT-X, RT-2, and Diffusion Policies papers
• The idea that progress in data, compute, and foundation models are waves that robotics researchers should ride
• The hypothesis that robotics tasks lie on a relatively simple manifold that a large model could discover
• The argument that large models are the best approach to capture "commonsense" capabilities that are important for robotics

[02] Why Scaling Might Not Work

1. Questions related to the content of the section?

• The lack of large robotics datasets, and the difficulty in collecting them compared to vision and language data
• The challenge of dealing with the variety of robot embodiments and environments
• The high cost and energy-intensive nature of training large robotics models
• The "99.X problem" and long tails in robotics tasks, which may be difficult to overcome with scaling alone
• Evidence from self-driving car companies that large-scale ML approaches have not yet fully solved the 99.X problem

[03] Miscellaneous Related Arguments

1. Questions related to the content of the section?

• The argument that we can probably deploy learning-based approaches robustly, even without theoretical guarantees
• The idea of pursuing human-in-the-loop systems to address the 99.X problem
• Leveraging simulation and pre-training on general vision/language data to reduce the need for real-world robotics data
• Combining classical and learning-based approaches to get the best of both worlds

[04] What Can/Should We Take Away From All This?

1. Questions related to the content of the section?

• The consensus that scaling up learning with large datasets is a promising direction to pursue, even if it doesn't fully solve robotics
• The importance of also continuing to pursue other existing directions, like classical robotics techniques
• The need to focus more on real-world mobile manipulation and making robot learning systems easier to use
• The value of being more forthright about reporting negative results
• The potential for thinking outside the box and trying something totally new