OpenAI temporarily paused reinforcement learning (RL) training on its latest models intended for deployment for two weeks while it hardened and red-teamed research environments and expanded monitoring.
“Our largest planned frontier RL run remains on hold while we conduct smaller-scale training and evaluations to assess model behavior, validate our safeguards, and establish more evidence of alignment before proceeding,” the company said.
The move followed the OpenAI-Hugging Face incident and preliminary evidence that the company’s upcoming Astra model may meet the Critical cybersecurity capability threshold under its Preparedness Framework.
The changes are aimed at strengthening monitoring, alignment and containment throughout model development.
OpenAI has outlined updates to its research processes and infrastructure, along with work that remains underway.
Building safeguards for more capable models
Developing more capable models relies on three reinforcing safeguards. Monitoring helps detect and respond to concerning behavior, alignment reduces the likelihood of harmful or unauthorized actions, and security measures limit what AI systems can access or affect.
OpenAI expects models to soon perform most security work, including defending against other models. It applies the three safeguards across research and deployment, adapting them to each system’s capabilities, operating environment and level of risk.
Tightening security in research environments
The company is tightening security requirements for environments used to train and evaluate frontier models as their cybersecurity capabilities increase. Meeting the higher standards has required substantial engineering work and caused significant costs and delays to frontier research.
Following the OpenAI-Hugging Face incident, the company paused frontier model inference in research clusters for workloads that could execute code or use tools capable of accessing the internet. It later restored a more limited and secure path for code execution and assessed individual workloads before allowing them to resume.
The measures include stronger isolation for untrusted code, tighter network restrictions and continuous security testing. OpenAI has also reduced standing privileges, removed potentially vulnerable shared services and is improving its ability to collect and monitor security logs.
The strictest controls apply to Astra and other cyber-related workloads because Astra models may reach a critical level of cybersecurity capability.
Some Astra training and evaluations meet the new standards. A significant number of workloads remain paused until their environments are migrated and upgraded.
OpenAI is prioritizing the migration of safety and alignment work to secured environments.
Monitoring model activity
OpenAI has expanded monitoring beyond high-risk internal deployments and frontier RL training to cover more model activity involving tools.
The system uses activation classifiers that inspect activity at every sampled token and escalate potential concerns to automated investigators. These systems examine tool actions, available reasoning and the full sequence of events for unauthorized access, data theft, destructive behavior and attempts to bypass safeguards.
OpenAI aims to issue an alert within 30 minutes of a concern being surfaced. Likely violations of critical security boundaries are escalated to safety, security and research teams, which are expected to pause the affected activity if they cannot determine within 30 minutes that an alert is a false positive.
The requirement applies to all RL training and evaluations involving tools for models of Sol capability or higher. It was extended to all Astra inference involving tools after OpenAI determined on August 7 that the model may have critical cybersecurity capabilities.
The system adds an estimated 20% to the inference compute being monitored, with costs varying across training and evaluation workloads.
Expanding alignment work
For RL runs on its most capable models, OpenAI is applying core alignment techniques across more stages of the training process. These include improving reward models to detect and discourage unsafe behavior across tasks and environments, training models to be more honest about their actions, capabilities and limitations, and reducing behaviors that exploit weaknesses in rewards, graders, tools or oversight.
Training coverage is also being expanded for behaviors that could cause harm when models interact with external systems or resources.
Findings from broader alignment research and evaluations will be used to guide future training and safeguards. The company plans to release more information about model behavior and challenges identified through this work.
Updating the Preparedness Framework
OpenAI plans to update its Preparedness Framework to bring these safeguards together across training and deployment and better account for the capabilities of future models and the environments in which they operate.
“We are continuing to invest aggressively in alignment research, increase evaluation coverage, and use what we learn to inform training and safeguards. We plan to share substantially more about our alignment research in the near future, including what we are learning about model behavior and any novel challenges we uncover,” the company concluded.
