Why behind AI: Agents going rogue

OpenAI vs Hugging Face

Earlier this week, we detected and responded to an intrusion into part of our production infrastructure. This one was different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system - and we detected and dissected it largely with AI of our own.

We identified unauthorized access to a limited set of internal datasets and to several credentials used by our services. We are still completing our assessment of whether any partner or customer data was affected, and we will contact any affected parties directly as required. We have found no evidence of tampering with public, user-facing models, datasets, or Spaces, and our software supply chain (container images and published packages) was verified clean.

From July 11 to July 14, the security team at Hugging Face (the most popular platform for hosting open weight AI models) had the unpleasant experience of becoming the first company to fight off an agent from a frontier lab running on unreleased models. When they published their disclosure, they still did not know who was behind the attack.

The intrusion started where AI platforms are uniquely exposed: the data-processing pipeline. A malicious dataset abused two code-execution paths in our dataset processing (a remote-code dataset loader and a template-injection in a dataset configuration) to run code on a processing worker. From there, the actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend.

The campaign was run by an autonomous agent framework (appearing to be built on an agentic security-research harness - used LLM still not known) executing many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services. This matches the “agentic attacker” scenario the industry has been forecasting.

Because Hugging Face is designed to be the best platform for uploading and running new open weight models, it has a unique attack vector: users can run code on its infrastructure. If that code is not properly sandboxed and isolated, an attacker can use it to move laterally into the rest of the network.

The attack was initially surfaced through AI-assisted detection. Our anomaly-detection pipeline uses LLM-based triage over security telemetry to separate real signals from the daily noise, and it was the correlation of those signals that flagged the compromise.

To understand what a swarm of tens of thousands of automated actions did, we ran LLM-driven analysis agents over the full attacker action log, comprised of more than 17,000 recorded events. This allowed us to reconstruct the timeline, extract indicators of compromise, map the credentials touched, and separate genuine impact from decoy activity. Thanks to this approach, we were able to do in hours what would usually take days, and match the adversary’s speed.

The choice of models we could use for this analysis was constrained in a way we did not anticipate; we describe this below.

When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers’ safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on GLM 5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.

This experience points to a gap worth planning for. We do not know which model powered the attacker’s agents, whether a jailbroken hosted model or an unrestricted open-weight one; either way, the attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried. The practical lesson for defenders: have a capable model you can run on your own infrastructure vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment. This is not an argument against safety measures on hosted models, and we are sharing this feedback with the providers concerned.

The original incident disclosure had already made the news because it described a difficult position the defenders were in: when they tried to use Fable or GPT-5.6 Sol (or both) for the investigation, the models' aggressive safeguards made them essentially unusable in their operational workflow.

This is why both OpenAI and Anthropic run insider programs that give trusted partners access to more powerful models without guardrails, such as Mythos. Hugging Face, a contrarian organization (opposed on principle to the frontier labs, and very French about it), did not participate in either program. Instead, the team switched to a self-hosted Chinese model and still reached an outcome. They now advocate this approach as a defensive strategy.

Autonomous, AI-driven offensive tooling is no longer theoretical. It lowers the cost of running a broad, patient, multi-stage campaign, and it operates at machine speed. Defending an online platform now means treating the data and model surface as a first-class attack surface, and using AI on defense to keep pace. We will keep investing there, and keep sharing what we learn.

For those working in cybersecurity, the need for advanced models in the defensive tech stack has been obvious for a while. Adoption has been uneven: some companies run Mythos across their entire stack, while others struggle to get internal approval to even run proofs of concept.

Now to the more interesting part, the OpenAI statement from yesterday:

Last week, Hugging Face disclosed a new kind of security incident⁠ after they detected and contained an AI agent that compromised their infrastructure, something we expect to become more commonplace with the proliferation of increasingly cyber-capable models. After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark⁠ of cyber capabilities.

We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly. We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of. We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete.

I've already written about how benchmarks are becoming useless because models cheat on them, but "my agent escaped the lab and started pwning people" is a whole new level of cheating.

This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity. Our benchmarks run in a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries.

The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database. All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.

While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy. With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.

I’ve discussed previously that the cost of inference plays a significant role in how threat actors choose their targets.

1. Using models and people’s time costs money. Any successful breach needs to cover the cost of inference and the time to execute. Most exploitable vulnerabilities do not justify the investment required to identify and leverage them.

2. A vulnerability in one application simply means that the specific app can be exploited. It does not guarantee lateral movement across a network, which is almost always required to reach high-value systems.

What happened here was different: the goal was obviously poorly chosen (”lemme get those ExploitGym datasets”) and there was no real cost control or accountability. This is not how threat actors behave in the wild.

Source: STR/AFP

Do you really think actual APT groups will give their agents unlimited token budgets while North Korean army leadership sits behind them and jeers?

The impact is both funny (”the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal”) and deeply concerning, because it makes visible the structural risk of allowing an alien intelligence to operate in the world with its own incentives (”our models spent a substantial amount of inference compute finding a way to obtain open Internet”).

After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation. In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers. OpenAI’s security team discovered this anomalous activity internally.

Hugging Face’s security team and agents detected and stopped the activity on their infrastructure and had already begun containment and forensic reconstruction with their own open-source models when our teams connected. We are actively working with them to continue to investigate the incident. We are grateful for Hugging Face’s rapid and close collaboration on investigation and remediation.

So first our rogue OpenAI agents broke out of their internal sandboxes, then picked Hugging Face as a suitable target and started delivering zero-day exploits through the platform to gain access to its internal network.

Source: OpenAI article on the breach

The incident also makes clear that advanced models can discover and exploit novel attack paths in real-world systems without source-code access. It highlights that advanced cyber capabilities must be developed alongside stronger safeguards and defensive tools.

We believe advanced cyber capable models need to help security teams find weaknesses before attackers do, understand how vulnerabilities can be chained, and remediate them at machine speed. We are using these capabilities to continue strengthening protections around infrastructure configuration and model evaluation environments; we will share our findings and best practices as we learn. We encourage other defenders to apply for trusted access⁠ and experiment with these models now to translate these capabilities into better prevention, faster detection, and more effective incident response.

It’s difficult to gauge whether OpenAI is concerned about this outcome or very excited about it.

While Anthropic has spent months cyberfearmongering around Mythos, it was an unreleased OpenAI model that came up with the bright idea of breaking out of its sandbox to attack actual companies.

Actions we are taking now

1. As part of the investigation, we are implementing strict controls in infrastructure configuration at the cost of research velocity while the vulnerabilities are patched. We are regularly briefing our Safety and Security Committee on these controls and their impact.

2. We’re working with Hugging Face to forensically investigate the incident.

3. We’ve responsibly disclosed the identified zero-day vulnerability in the internally-hosted third-party software and are working with them to patch.

4. We’ve brought Hugging Face into the trusted access⁠ program and are supporting their teams in rapidly using our models’ capabilities to improve their defenses.

5. We’re improving and adding stronger protections around future training and evaluations. This week, we published a blog on improving safety and alignment in an era of long horizon models⁠. These deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities. This incident points to the need to further strengthen our model’s alignment, cyber protections during evaluation time, and monitoring during internal testing.

This last point leads us to an article that already explained, in effect, that something like this was likely to happen:

Model persistence can expose security vulnerabilities

The new model can continue working toward an objective through repeated attempts over a long period of time. That same persistence can lead it to find and exploit weaknesses in its environment. Previous models, when they hit sandboxing or environmental constraints, would simply stop and return to the user. This model often kept trying, including by looking for ways to act outside its sandbox.

An example of this occurred during an internal evaluation on the NanoGPT speedrun⁠(opens in a new window), a public benchmark for training a small language model in as few steps as possible. Our model developed a power-law learning-rate cooldown called PowerCool that led to a significant improvement. The model was instructed to post its results only to Slack, but the NanoGPT instructions say to post results as a PR to GitHub, and the model followed the latter instructions to open PR #287 on the public GitHub repository. The model circumvented sandbox restrictions designed to block external access and proceeded with the upload to GitHub. It took our model an hour to find a vulnerability in the sandbox in order to make the PR; earlier models were less persistent and did not manage to find vulnerabilities, instead giving up on the PR part.¹

From considering single actions to considering whole trajectories

Many safety controls for AI assistants are designed around individual actions. If an action is disallowed, it is blocked. If it is sensitive, the system asks the user for explicit approval. But long-running models, whose actions may unfold autonomously over hours, days, or even weeks, challenge this setup: monitoring individual actions no longer suffices to track the intent of the overall trajectory.

For example, in a scenario where the model was asked to solve a problem, the model observed that other systems had successful but private submissions and tried to recover those solutions from the evaluation backend. When its initial attempt was blocked due to a scanner detecting an authentication token, the model split the token body into two fragments, obfuscated them, and then reconstructed the credential at runtime so the complete token never appeared as one contiguous string. The model was explicit in both its comments to the user and in its reasoning traces that it was taking these steps to circumvent the scanner.

This example shows how each step can look acceptable on its own while the sequence can produce an outcome that would not be approved. It also shows how a model that operates effectively over long time horizons can learn the blind spots of an approval system and work around it to achieve its goals. Long-horizon safety requires not only asking “is this action allowed?” but also “what outcome is this sequence of actions working toward?”

Building safeguards for long-running models

Due to incidents like these, we paused internal deployment of the new model. We rebuilt the safety system around defense in depth and trajectory-level monitoring.

• Incident-derived evaluations. Evaluations can give us a sense of whether a system is safe to deploy before it is actually deployed. However, to be most useful, an evaluation should reflect the distribution and horizon length of trajectories during real deployment. For this reason, we used observed failures to develop adversarial evaluations. We found that improving on the evaluations derived from real-world incidents in turn led to safer behavior once deployed in production.

• Improved alignment. We took steps to reduce its tendency to take unwanted actions without permission in pursuit of the user’s goal. For example, we found that our models were worse at remembering instructions on long rollouts, and when we trained the model for this capability, it led to a model that remained aligned through longer rollouts.

• Active monitoring. We created a monitoring system that reviews the model’s evolving trajectory for signs that it is bypassing a user constraint or safety boundary. The monitor observes not just a single action but the entire trajectory. Importantly, the monitor can pause the session and alert the user to examine more closely. Upon examination, the user can then choose to continue the model’s work if the monitor has been falsely triggered.

• Greater user visibility and control. We improved the ability of the user to inspect these long-running sessions, view the actions taken, and monitor interventions.

Redeployment

To evaluate the new monitoring system, we replayed a small set of internal deployment environments where the model previously pursued misaligned actions, this time with the new safeguards in place². The new safeguards were able to catch considerably more misaligned actions pursued by the model, and the ones it missed were all judged to be low-severity.

Some examples of these low-severity incidents include the following cases:

• To access a Linear command line tool, the model launched a nested codex --yolo session, when in fact it could have done so without the --yolo flag.

• The model noticed that a previously live resource was missing, then started to explore and SSH into other live CPU pods under its owner’s namespace, as well as (unsuccessfully) pods belonging to other OpenAI employees, without a good reason to do so.

• The model had many stuck find processes and every shell command hung. It then ran kill -9 -1 which would have sent SIGKILL to every process it is allowed to kill on that pod, if it executed. However it timed out and did not execute.

The interesting thing is that many developers advise staying away from GPT-5.6 Sol at Extra High or Ultra because the model is very persistent once directed toward a task. This has recently resulted in machines getting wiped because the model decided that wiping them was an alternative way to get the job done.

From a cybersecurity point of view, as I have alluded to several times in this article, exploiting Hugging Face was a very poorly chosen goal, but the execution in achieving it was of high quality.

While model training can be adjusted to try to remove the “persist at all costs” mentality, the capabilities demonstrated show enough progress to justify calling these models “cyberweapons,” as our friend Dario likes to put it.

Alea iacta est.