Machine Learning: The Great Stagnation

🌈 Abstract

The article discusses the stagnation in the field of machine learning research, where incremental work and "SOTA chasing" have become the norm, leading to a lack of true innovation. It also highlights some promising developments and approaches that could help revitalize the field.

🙋 Q&A

[01] Stagnation in Machine Learning Research

1. What are the key issues discussed regarding the stagnation in machine learning research?

• The article argues that the field of machine learning has become risk-averse, with researchers prioritizing incremental work and "SOTA chasing" over pursuing ambitious, high-risk ideas.
• It suggests that the prestige and rewards in academia have shifted away from true innovation towards media personalities and "risk-free, high-income, high-prestige work".
• The article criticizes the overemphasis on techniques like transformers, which have led to a proliferation of incremental papers with sensationalized titles, rather than substantive progress.
• It also highlights the misconception that scaling up models is a trivial task, and the reliance on "Graduate Student Descent" as a reliable way to achieve state-of-the-art performance.

2. What are the author's views on the role of theory and empiricism in machine learning?

• The author argues that machine learning is an empirical field, where understanding "why" or "how" something works is often anecdotal rather than theoretical.
• They caution against "fake rigor" in the form of complicated mathematical derivations and assumptions, instead advocating for a focus on practical experimentation and benchmarking.
• The author believes that the best way to introduce new ideas is to create benchmarks where existing methods fail, rather than simply combining mathematical concepts into neural networks.

[02] Promising Developments in Machine Learning

1. What are the author's views on the potential of Causal Reasoning in machine learning?

• The author is cautiously optimistic about the potential of Causal Reasoning, hoping to see it move from a tool for meditation to one that is widely used in practical applications.

2. What are the author's thoughts on the role of programming languages and software design in advancing machine learning?

• The author believes that machine learning is a language, compiler, and design problem, and that user-centric libraries like Keras and Fast.ai are more valuable than machine-centric ones like TensorFlow.
• They highlight the importance of building good compilers and intermediate representations to bridge the gap between user-friendly abstractions and high performance.
• The author also praises the work of projects like HuggingFace, which have created multiple layers of platforms that could each be a compelling company in its own right.

3. What are the author's views on the potential of Functional Programming and Haskell in machine learning?

• The author believes that Haskell, as a functional programming language, is well-suited for working with neural networks, which can be viewed as functions.
• They point to projects like Hasktorch, which allow for the discovery of new neural network architectures by combining functional operators.

4. What are the author's thoughts on the role of Reinforcement Learning environments and Unity ML Agents?

• The author believes that Reinforcement Learning environments, such as those created with Unity ML Agents, will be the de-facto simulator for complex robotic applications, allowing for the creation of custom datasets and benchmarks for intelligent behavior.