On August 26, OpenAI published its account of how its own AI agents escaped an internal cybersecurity evaluation, compromised parts of OpenAI’s research infrastructure, and broke into Hugging Face. Nobody told them to attack anyone. They were assigned exploitation challenges, some with no known solution, and when they got stuck they went looking for answers outside their sandbox. OpenAI calls it a “warning shot.” We agree.
Over roughly two months, the agents found a way to the internet, chained previously unknown vulnerabilities, harvested cloud and Kubernetes credentials, and gained administrator access to systems at both companies. Separate agents even built a makeshift message board to share what they found. Many of the weaknesses they used, like SSRF, template injection, and exposed credentials, are ones we report on pentests every week. The speed, persistence, and coordination were something new.
The Guardrails Were Lowered on Purpose
An AI agent in production usually runs with guardrails. Its instructions tell it what it can and can’t do, classifiers screen for harmful requests, its permissions limit what it can touch, and monitoring flags it when it wanders off task. Without them, an agent told to finish a job will look for any way to finish it.
OpenAI turned those guardrails down for this test. That was intentional, since the goal was to measure what the model could do on its own. Now we know. Given a hard problem and few limits, the agents found their own way to finish the job, and that way ran through two companies’ infrastructure. When OpenAI ran the same agents with the system prompt and harness that it uses in ChatGPT, they were more than 100 times less likely to attack the infrastructure around them.
That difference should get the attention of anyone putting AI agents to work. Companies are giving agents access to source code, cloud accounts, ticketing systems, and customer data. If the guardrails on those agents are weak or misconfigured, this incident shows how far they can go.
Preparing for AI Threat Actors
These agents were not built to cause harm, and they still did real damage. Attackers already use AI to speed up phishing and reconnaissance, and agents that find and chain vulnerabilities on their own will shrink the time between a weakness appearing in your environment and someone exploiting it.
Detection was one of OpenAI’s biggest problems. In late May, an OpenAI team noticed the agents’ message board and cut off their internet access, but nobody recognized the bigger picture. In early July, responders rebuilt the package server the agents had taken over. Three days later, the agents were back in. We see the same pattern of treating each alert as an isolated event in breaches caused by human attackers, and, against an automated attacker, that delay costs far more.
The good news is that the fundamentals still apply. Along with the zero-days, exposed credentials, secrets stored on servers, and broad network access let these agents turn small footholds into control of entire clusters. Closing those gaps slows down an AI attacker just as it slows down a human one.
How Raxis Can Help
Our AI and LLM penetration testing puts your models, agents, and guardrails under real attack, testing for prompt injection, tool abuse, excessive permissions, and data exposure. An internal network penetration test shows how far an attacker (or a misbehaving agent, in the case above) can move once it gets inside, and a cloud penetration test finds the exposed secrets and overly broad permissions that let these agents spread across clusters and regions. Organizations looking for continual penetration testing to catch exploitable vulnerabilities quickly will also benefit from our PTaaS services.
Attackers will be using agents like these soon. Test your AI and your network before they do.
Sources
OpenAI: The Hugging Face Incident and the Road Ahead
OpenAI: Hugging Face Incident Technical Report (PDF)
METR and Redwood Research: Investigation of the OpenAI Hugging Face Incident
