Superhuman?

🌈 Abstract

The article discusses the progress of Artificial General Intelligence (AGI) and the current capabilities of AI systems compared to human performance. It examines the various benchmarks and tests used to measure AI abilities, the rapid improvement of AI across different domains, and the implications of AI surpassing human abilities in specific tasks.

🙋 Q&A

[01] The Explicit Goal of AI Labs

1. What is the explicit goal of many of the most important AI labs on the planet?

• The explicit goal of many of the most important AI labs is to achieve Artificial General Intelligence (AGI), which can mean anything from "superhuman machine god" to "a machine that can do any task better than a human."

2. What are the different views among computer scientists on when AGI might be achieved?

• Computer scientists are divided on the approaches and timelines for achieving AGI. The average expected date for achieving AGI in a 2023 survey was 2047, but the same survey also gave a 10% chance of AGI being achieved by 2027.

3. What is the current state of AI compared to human performance?

• While AI has not yet achieved a level that can do every task better than a human, it has already achieved superhuman performance in some complex and "human" tasks, such as persuading people in debates, providing better emotional support than humans, and generating startup ideas better than business school students.

[02] The Jagged Frontier of AI

1. What is the "Jagged Frontier of AI" and how does it describe the current capabilities of AI?

• The "Jagged Frontier of AI" refers to the uneven abilities of AI, where it can excel at some tasks but struggle with others, even within the same job. AI works best as a "co-intelligence," a tool that humans use to augment their own performance, especially once they understand the shape of the Jagged Frontier.

2. What are the major AI companies aiming to achieve with AI?

• The major AI companies want to push the Frontier further and further until AI is better than every human at every task, which raises a lot of issues and questions about how to measure and understand AI's capabilities.

[03] Testing for Superhumanity

1. How do AI companies test for superhuman abilities?

• One way to test AI's abilities is to give it human tests, such as the ones OpenAI showcased upon the release of GPT-4, where the AI outperformed a large percentage of human test-takers.

2. What are the issues with using human tests to measure AI capabilities?

• The use of human tests to measure AI capabilities is fraught with problems, such as the possibility of the AI "memorizing" the answers, and the limitations of the tests themselves in accurately measuring real-world skills.

3. What are the challenges with using AI benchmarks to understand the progress towards AGI?

• AI benchmarks, such as the MMLU, have their own set of issues, including the potential for AI to be trained on the test questions, the uncalibrated nature of the tests, and the unclear relationship between test scores and real-world abilities.

[04] The Rapid Improvement of AI

1. What do the various AI benchmarks and measures, such as the Arena Leaderboard, suggest about the overall progress of AI?

• Across a wide range of benchmarks, AI ability gains have been rapid, often exceeding human-level performance. As long as the scaling laws of training Large Language Models continue to hold, this rapid increase is likely to continue.

2. How should we interpret the superhuman abilities of AI in certain tasks?

• The increasing ability of AI to beat humans across a range of benchmarks is a sign of superhuman ability, but it also requires cautious interpretation. AI is more akin to an "alien intelligence" with a distinct set of capabilities and limitations, rather than a direct comparison to human abilities.

[05] Tiers of AGI Development

1. What are the different tiers of AGI development proposed in the article?

• The article proposes four tiers of AGI development:
  • Tier 1: AGI - "a machine that can do any task better than a human"
  • Tier 2: Weak AGI - a machine beats an average human expert at all the tasks in their job, but only for some jobs
  • Tier 3: Artificial Focused Intelligence - AIs beat an average human expert at a clearly defined, important, and intellectually challenging task
  • Tier 4: Co-Intelligence - Humans working with AI often exceed the best performance of either alone

2. What are the implications of the different tiers of AGI development?

• As AI continues to surpass human abilities in specific domains, it will likely lead to significant disruptions across industries, but the path to true AGI remains uncertain. The rise of Artificial Focused Intelligence and co-intelligence systems will likely lead to increased productivity and efficiency, but may also require a re-evaluation of the role of humans in decision-making.