From sci-fi to state law: California’s plan to prevent AI catastrophe

🌈 Abstract

The article discusses the "Safe and Secure Innovation for Frontier Artificial Intelligence Models Act" (SB-1047) introduced in California, which aims to regulate large AI models to prevent potential catastrophic harms. The article explores the debate around the bill, including concerns that it may limit AI research and development, and the differing viewpoints of supporters and critics.

🙋 Q&A

[01] California's "Safe and Secure Innovation for Frontier Artificial Intelligence Models Act" (SB-1047)

1. What are the key provisions of the SB-1047 bill?

• The bill requires companies behind sufficiently large AI models (over $100 million in training costs) to implement testing procedures and systems to prevent and respond to "safety incidents"
• The bill defines "safety incidents" as harms that could lead to "mass casualties or at least $500 million of damage," such as the creation or use of weapons of mass destruction or instructions for conducting cyberattacks on critical infrastructure
• The bill requires model creators to have the capability to promptly shut down their models and have policies in place for when such a shutdown would be enacted

2. What are the concerns raised by critics of the bill?

• Critics argue that the bill's focus on existential threats by future AI models could severely limit research and development for more prosaic, non-threatening AI uses today
• Some experts believe the bill is based on "outlandish fears of future systems that resemble science fiction more than current technology"
• Critics argue that the bill is "fictional-led legislation" driven by "AI doomers" with "fictional fears" rather than a "sane, sound, 'light touch' safety bill"

3. What are the arguments made by supporters of the bill?

• Supporters, including AI luminaries Geoffrey Hinton and Yoshua Bengio, agree with the bill's co-founder Dan Hendrycks that the bill is a necessary step to prevent potential catastrophic harm from advanced AI systems
• Bengio wrote that "AI systems beyond a certain level of capability can pose meaningful risks to democracies and public safety" and that the bill offers a "practical approach" to addressing this

[02] Debate around the bill

1. What is the core disagreement between supporters and critics of the bill?

• Supporters believe the bill is a necessary precaution against the potential existential risks posed by advanced AI systems, while critics argue that the bill is based on "outlandish fears" and could limit valuable AI research and development

2. How do critics characterize the motivations behind the bill?

• Critics argue that the "power-seeking behavior" is not of AI systems, but of "AI doomers" who are trying to pass "fictional-led legislation" driven by their "fictional fears"

3. What are the potential consequences of the bill, according to critics?

• Critics believe the bill could "ruin California's and the US's technological advantage" if implemented, due to its potential to limit valuable AI research and development