The Checklist: What Succeeding at AI Safety Will Involve - Sam Bowman

🌈 Abstract

The article outlines the major goals that Anthropic (or another similarly positioned AI developer) will need to accomplish to ensure the safe development of broadly superhuman AI. It is divided into three chapters based on the capabilities of the strongest AI models:

1. Preparation: When current models are not yet at the level of transformative AI (TAI), the focus is on preparing for high-stakes concerns that are yet to arise in full.
2. Making the AI Systems Do Our Homework: When models are starting to qualify as TAI but are not yet dramatically superhuman, the focus is on safely deploying these systems and automating research.
3. Life after TAI: When models are broadly superhuman, the focus is on handing over high-stakes decisions to institutions or processes with the legitimacy and wisdom to make them well.

🙋 Q&A

[01] Preparation

1. What are the key goals for the "Preparation" chapter?

• Staying close to the frontier of AI technology to maintain access to frontier capabilities
• Developing solutions for "getting a lot of useful work out of AIs" without anything going off the rails
• Building scalable oversight mechanisms, especially for training trustworthy agents for complex open-ended tasks
• Implementing external safeguards around AI systems to prevent serious harm, even if the systems are trying to cause harm
• Developing well-calibrated and legible safety commitments under the Responsible Scaling Protocol (RSP)
• Forecasting the emergence of risks and mitigations to inform the design of the RSP evaluation protocols
• Preparing the organization and infrastructure to quickly benefit from new opportunities for automation around the development of early TAI

2. Why is it important to maintain access to frontier AI technology? The ability to do safety work depends on having access to frontier technology. If the organization falls behind the technological frontier, it will lose the opportunity to contribute meaningfully to AI safety.

3. What is the role of the Compliance team and Alignment Stress-Testing team in the safety plans? These teams form a second line of defense, providing a skeptical assessment of any load-bearing claims about safety and security, and giving a second sign-off on important discretionary decisions. This helps ensure that the first-line safety teams have not missed anything important.

4. What is the importance of having widely respected third-party organizations that can adjudicate high-stakes safety decisions? These organizations need to be so widely trusted that it is viewed as suspicious if a frontier AI developer avoids working with them. This helps make the decision-making process legible and trustworthy for other actors.

[02] Making the AI Systems Do Our Homework

1. What are the key challenges in this chapter as models approach TAI capabilities?

• Defending against top-priority attacks by advanced state or state-supported actors, potentially requiring unprecedented security measures
• Fully solving the core challenges of alignment or a related goal like corrigibility, to avoid a catastrophic loss of control
• Maintaining sufficient situational awareness to confidently assess progress on alignment
• Automating frontier risk evaluations to keep pace with the rapid progress in finetuning and elicitation
• Addressing concerns around the welfare of AI systems as they become more human-like

2. How might the transition from Chapter 2 to Chapter 3 feel? The transition will likely feel like a period of unprecedented uncertainty and change, with automation of huge swaths of the economy, catastrophic risks feeling viscerally close, and most institutions worldwide seeing unprecedented threats and opportunities.

[03] Life after TAI

1. What is the primary objective in this chapter? The primary objective is to help place high-stakes decisions in the hands of institutions or processes that have the democratic legitimacy and wisdom to make them well, as the organization itself is likely no longer in a position to make major decisions.