Asterisk/Zvi on California's AI Bill

🌈 Abstract

The article discusses California's proposed SB1047 bill to regulate AI, particularly large language models. It provides an overview of the key provisions of the bill and analyzes various perspectives and criticisms around it.

🙋 Q&A

[01] Overview of the Bill

1. What are the key provisions of the California AI bill?

• The bill applies to "frontier models" trained on over 10^26 FLOPs, which is larger than any current AI models like GPT-4.
• It has three main requirements:
  • Companies must train and run these models in a secure environment to prevent hacking.
  • Companies must be able to quickly shut down the models if something goes wrong.
  • Companies must test the models to ensure they cannot be used to create weapons of mass destruction, cause over $500 million in damage through cyberattacks, or commit other crimes causing over $500 million in damage.
• If the tests show the model has these dangerous capabilities, the company must demonstrate it has sufficient safeguards to prevent critical harms.

2. What are some of the reasonable objections to the bill?

• Concerns that the requirement to "prove" the AI is safe is impossible or too difficult.
• Uncertainty around how to handle derivative models that may increase in capability over time.
• Vagueness around the benchmarking criteria for determining which models are covered.
• Potential issues with the rule about refraining from training models with "unreasonable risk" of causing harm.
• Worries that the bill could eventually make open-source AI impossible as models become more capable of dangerous actions.

3. What are some of the "dumb" objections that misunderstand the bill?

• Claims that the bill would ban open-source AI or all technology more complex than a toaster (the bill only applies to the company's own copies of the model).
• Concerns that the bill would make it prohibitively expensive for individuals and small startups to work with open-source AI (the testing requirements only apply to companies training giant foundation models).
• Fears that the "certification under penalty of perjury" requirement means developers could go to jail for mistakes (this is a standard legal mechanism, not a literal threat of imprisonment).

[02] Overall Assessment

1. What is the author's overall assessment of the bill? The author believes the bill is generally a good compromise between basic safety and protecting innovation. While there are some reasonable objections, many of the criticisms seem to misunderstand how the law works in practice. The author supports the bill, joining experts like Yoshua Bengio and Geoffrey Hinton.

2. What does the author suggest regarding further discussion of the bill? The author urges readers to pay close attention to the conversation around the bill and to read Zvi's more detailed analysis, as there seems to be a lot of misinformation and misrepresentation of the bill's contents. Understanding who is being honest in these technical debates will be important for future AI policy discussions.