The Coprophagic AI crisis

🌈 Abstract

The article discusses the challenges faced by science fiction (SF) writers in distinguishing between science fiction as a thought experiment and making predictions about the future. It highlights the tendency of some SF writers and tech billionaires to treat SF as prophecy, leading to the creation of technologies that may not be beneficial. The article also delves into the issue of "botshit" - the proliferation of inaccurate or fabricated content generated by AI systems, and how this can undermine the quality of information on the internet.

🙋 Q&A

[01] The distinction between science fiction and predictions

1. Questions related to the content of the section?

• The article emphasizes the importance for SF writers to distinguish between science fiction as a thought experiment and making actual predictions about the future.
• SF writers often "palm cards" or use narrative devices like faster-than-light drives or time machines as part of their stories, without these being scientifically grounded proposals.
• In the past, the inability to distinguish between SF and prophecy was not a major issue, as those who fell for the "SF-as-prophecy" delusion did not have the power to reshape society around their beliefs.
• However, with the rise of SF-obsessed tech billionaires, SF writers are becoming more vocal about separating their made-up stories from actual predictions.

2. What is the key point the author is making about the distinction between science fiction and predictions? The key point is that SF writers need to be clear about the difference between their fictional thought experiments and making actual predictions about the future, as the latter can have real-world consequences, especially with the influence of tech billionaires.

[02] The "add enough compute and the computer will wake up" trope

1. Questions related to the content of the section?

• The article discusses the common SF trope that once a computer matches the human brain in "complexity" or "power," it will become conscious.
• This idea is often used by those inflating the current AI hype bubble, who believe that simply increasing the training data and computational power of AI systems will solve their defects.
• The author argues that this belief is not supported by evidence and that the proliferation of "botshit" (inaccurate or fabricated content) generated by AI systems undermines the idea that more data and compute will linearly improve AI performance.

2. What is the key point the author is making about the "add enough compute" trope? The key point is that the belief that simply increasing the training data and computational power of AI systems will solve their problems is a flawed idea not supported by evidence, and the growing issue of "botshit" generated by AI systems further undermines this notion.

[03] The problem of "botshit" and its impact on the internet

1. Questions related to the content of the section?

• The article introduces the term "botshit," which refers to the inaccurate or fabricated content generated by AI systems at scale.
• As the internet becomes increasingly filled with botshit, the amount of human-generated "content" is dwindling to homeopathic levels, even in sources considered high-quality.
• The article discusses how AI companies are setting themselves up for this problem, as the push for "AI-powered search" could lead to a future where the open web becomes an AI "CAFO" (Concentrated Animal Feeding Operation) and search crawlers ingest the contents of its "manure lagoon."

2. What is the key point the author is making about the problem of "botshit"? The key point is that the proliferation of inaccurate or fabricated content generated by AI systems, known as "botshit," is undermining the quality of information on the internet, even in supposedly high-quality sources. This problem is exacerbated by the push for AI-powered search, which could lead to the open web becoming dominated by AI-generated content.

[04] The mathematical consequences of AI "coprophagia"

1. Questions related to the content of the section?

• The article discusses the concept of "Habsburg AI," where an AI model is trained on the output of another AI model, similar to feeding cows a slurry made of the diseased brains of other cows.
• A recent paper, "The Curse of Recursion: Training on Generated Data Makes Models Forget," delves into the mathematical consequences of this AI "coprophagia," finding that "using model-generated content in training causes irreversible defects."
• The author argues that even if one accepts the idea that more training data can solve AI problems, the training data itself is becoming increasingly elusive and contaminated with botshit, further undermining the potential for improvement.

2. What is the key point the author is making about the mathematical consequences of AI "coprophagia"? The key point is that training AI models on the output of other AI models, a practice known as "AI coprophagia," can lead to irreversible defects in the models, as demonstrated by the findings of the "The Curse of Recursion" paper. This undermines the belief that simply increasing the training data can linearly improve AI performance.