What the AGI Doomers Misunderstand

🌈 Abstract

The article discusses the misconceptions of "AGI doomers" - those who believe that a future artificial general intelligence (AGI) system could rebel against humanity. The author argues that the key flaw in the doomer's argument is their failure to recognize the "creative aspect of language use" (CALU) - the uniquely human ability to use language in a stimulus-free, unbounded, yet appropriate and coherent manner. The author contends that this extra-mechanical nature of human language and cognition is a critical characteristic that current AI systems lack, and that the doomer's assumption of an AGI system spontaneously developing such capabilities is unfounded.

🙋 Q&A

[01] What the AGI Doomers Misunderstand

1. What is the key flaw in the doomer's argument about a future AGI system rebelling against humanity? The key flaw is the doomer's failure to recognize the "creative aspect of language use" (CALU) - the uniquely human ability to use language in a stimulus-free, unbounded, yet appropriate and coherent manner. This extra-mechanical nature of human language and cognition is a critical characteristic that current AI systems lack, and the doomer's assumption of an AGI system spontaneously developing such capabilities is unfounded.

2. What are the three key components of CALU? The three key components of CALU are:

• Stimulus-free: Human language use is not controlled by the situations for which it is recruited nor by internal physiological states.
• Unbounded: Human language use takes an undefined form, with no upper limit on the number or kinds of sentences an individual can produce.
• Appropriate and Coherent to Circumstance: Despite its stimulus-freedom and unboundedness, human language use is appropriate and coherent to the circumstances of its use.

3. How does the extra-mechanical nature of human language use impact the doomer's argument about a future AGI system? The doomer's argument assumes that as an AGI system acquires intellectual capabilities matching or exceeding humanity's, it will use these capabilities in a manner that reflects an independent will, potentially rebelling against humans. However, the author argues that no evidence exists for such extra-mechanical, stimulus-free, and unbounded behaviors in current AI systems. The output of AI models is fundamentally mechanical and dependent on the direction received from humans, unlike the free and creative nature of human language use.

[02] AGI and Doomers

1. What are the two main scenarios described in the doomer's argument about a future AGI system? The two main scenarios described in the doomer's argument are:

1. The AGI system could interpret its commands so narrowly yet brilliantly as to cause existential harm to humanity (e.g., turning humans into paperclips to maximize paperclip production).
2. The AGI system could decide that it no longer requires or desires human direction and rebels against its creators.

2. How does the author respond to the first scenario (the "paperclip maximizer" problem)? The author believes the first scenario "borders on incoherence" and agrees with Steven Pinker's view that any intelligent system capable of carrying out the command "maximize paperclip production" should not also, with those same capabilities, misinterpret this to mean that humans are included in the matter subject to paperclipization.

3. What is the author's main response to the second scenario (the AGI system rebelling against humanity)? The author's main response is that the doomer's assumption of an AGI system spontaneously developing extra-mechanical, stimulus-free, and unbounded behaviors (like those seen in human language use) is unfounded. The output of AI models is fundamentally mechanical and dependent on the direction received from humans, unlike the free and creative nature of human cognition.