The Significance of Preparedness in AI Development
In order to address the potential risks associated with the continuous improvement of AI models, Company X has established a new team called Preparedness. This team, led by Aleksander Madry, is dedicated to closely integrating capability assessment, evaluations, and internal red teaming for cutting-edge models, ranging from near-future models to those with AGI-level capabilities. The primary objective of the Preparedness team is to effectively track, evaluate, forecast, and safeguard against catastrophic risks across various domains, including:
- Individualized persuasion
- Cybersecurity
- Chemical, biological, radiological, and nuclear (CBRN) threats
- Autonomous replication and adaptation (ARA)
Developing a Risk-Informed Development Policy (RDP)
The mission of the Preparedness team also entails the development and maintenance of a Risk-Informed Development Policy (RDP). This policy outlines our approach to conducting rigorous evaluations of the capabilities of frontier models, as well as monitoring their progress. Additionally, the RDP aims to establish a spectrum of protective measures and a governance structure to ensure accountability and oversight throughout the entire development process. It complements our current risk mitigation efforts, contributing to the safety and alignment of advanced systems before and after deployment.