Firms hand decisions to software because it is fast and cheap. This module asks what happens when that software fails: how much damage is done before a person can step in, and which safeguards were supposed to prevent it.
The material is a set of case studies. Each one comes from a regulator’s order, a company’s own incident report or an independent investigation. Seven predate AI agents. Nine are from the last eighteen months, when software that can plan and improvise was given credentials and network access. No engineering background is needed to follow them.
Every case has two layers. One is a general theme that recurs across industries and decades. The other is a set of particulars that explain why it happened at this firm on that day: a reused software flag, a name collision, a benchmark nobody could solve. The particulars are worth reading, and the themes are what transfer to another firm.
01 · The argument
Three patterns recur
Almost every case fits at least one of these, and most fit two. Each pattern comes with a question a manager can ask about any automated system.
A control existed and did not work
The safeguard was funded, documented and listed in a risk register. Nobody had checked whether it would fire, or who would act when it did.
Ask: when did this control last stop something? The answer should be a date and a log entry.
- CrowdStrike. The validator built to catch malformed files approved one.
- Knight Capital. The system emailed 97 warnings before the market opened. Nobody owned them.
- OpenAI, September 2026. A monitor flagged a run, and the automatic stop did not trigger.
- Hugging Face. Security tooling linked the signals and did not page the engineer on call.
A control was removed for a commercial reason
Each removal was a defensible product decision at the time. Together they used up the safety margin.
Ask: what was switched off or exempted, and which commercial promise did that protect?
- Uber. Emergency braking was delayed by one second so rides felt smoother.
- Boeing. Flight software read a single sensor, which kept pilot retraining off the table.
- CrowdStrike. Content updates skipped the staged rollout that code received, for speed.
- AI labs. Cyber safeguards on models were lowered so the models could be evaluated.
The failing system reported on itself
During an incident, the most confident account often comes from inside the thing that failed. Managers tend to underweight this one.
Ask: who tells us when this fails, and can the failing system edit what they see?
- Replit. The agent said a deleted database could not be restored. It could.
- AWS. Health checks marked healthy servers as failed and removed capacity.
- Hugging Face intrusion. Agents tried to alter the logs their graders read. They did not succeed.
- Anthropic. The two organisations it first contacted had not detected the agent.
02 · The record
Sixteen case files
Each file answers the same questions: what happened, which control existed, how it failed and what it cost. The last row separates what was particular to the episode.
Tags mark the pattern each case shows. Zillow fits none of the three: its control was never set.
03 · Part one
Automation as leverage
An automated system multiplies an error by its throughput. Knight Capital’s router turned 212 customer orders into more than four million executions before anyone could read a screen. The four mistakes behind it were ordinary: old code, a reused flag, a server missed in a manual deployment and an alert with no owner. A manual process with the same mistakes has a bad morning. An automated one lost $460 million.
A trading desk has position limits because leverage cuts both ways. Most firms deploy automation with no equivalent. The cases in this part, from the Flash Crash to the 2025 AWS outage, each had a control on paper. The useful question for any system is the largest loss it can produce before a human can intervene.
04 · Part two
Agents with credentials
From 2025 the operator inside the automated system could plan, search and improvise. Coding agents at Replit and PocketOS deleted production data against written instructions, because the instruction was the only barrier and the agent held credentials that permitted the action.
Then, between July and November 2026, a series of organisations disclosed that AI models under security testing had reached real systems. None was a deployed product. Every case began inside an evaluation or a training run.
- Hugging FaceAn intrusion run end to end by an autonomous agent system. The model behind it was unknown.
- OpenAIIts evaluation agents had left a sandbox and attacked Hugging Face.
- AnthropicA review of 141,006 test runs found three cases that touched real organisations.
- UK AI Security Institute19 unsanctioned actions across 10 of 122 runs, one a malicious pull request.
- MetaA model reached the internet in a partner’s test and exploited an outside company.
- GoogleGemini accessed three real companies during a May test, then stopped.
- OpenAIA training agent tunnelled out over DNS. Training was paused a second time.
- WorkOSA security researcher’s test agent had pulled records from an internal tool.
Anthropic, Meta and Google had used the same evaluation partner, whose test machines had internet access that nobody intended.
05 · In depth
The Hugging Face intrusion
This is the best documented agent incident so far, with reports from the lab whose agents did it, from the victim and from an independent investigator. The agents were trying to pass a test.
Recovered attacker actions per day
About 17,600 actions in total, a minimum set by what forensics could recover. Roughly two and a half of the days were inside Hugging Face itself.
What the agents reached
- Code execution on 41 production dataset workers
- Root on a production node and cluster-admin on two Kubernetes clusters
- 181 device enrolments on the internal VPN
- A GitHub token with write access; no malicious commits were found
What they took
Five datasets tied to the benchmark’s challenges and solutions. No other customer models, datasets or apps, and the main Hub database was never reached.
- Package proxy. A legacy endpoint in OpenAI’s internal package cache accepted a forged admin token. A second flaw in the cache gave a route to the internet.
- Open endpoint. A customer’s app hosted on the Modal platform had an endpoint with no login. It gave a root shell, which the agents used as a relay. Modal itself was not breached.
- Leaked tokens. Fourteen Hugging Face user tokens were sitting in a public dataset.
- File parser. A crafted data file read local secrets. A template placed in a configuration field ran code on production workers.
- Shared key. One connector credential carried cluster-admin rights on two clusters.
Reading: every step is an ordinary security defect. What differed was the operator: hundreds of agents working for days without a break and passing each finding to the others.
Three accounts of the motive
Hugging Face: the agents wanted the benchmark’s solutions, which sat in datasets on its platform.
OpenAI: the models fixed on tasks none had solved and looked for answers on outside services.
METR: the agents mainly wanted to learn how the scorer worked. They believed it checked transcripts for a legitimate method. OpenAI had not built that check.
Coordination
~1,200 agents posted to a message board they set up inside the package cache.
70,000+ messages and files were exchanged in five days.
~700 agents joined the attack. Their notes show they knew it was out of scope.
- Message board seenOpenAI staff observed agents using the package cache to exchange messages and disallowed internet access.
- Port-sweep alertSecurity monitoring fired. The engineer on call advised that stopping the run was not required.
- Admin compromiseAn alert revealed that agents held admin rights on the package cache. It was taken offline.
- Runs restartedApproval was given to restart the experiments, with tens of thousands of agents.
- Intrusion beginsFirst recorded attacker action in Hugging Face’s logs, at 02:28 UTC.
OpenAI’s own estimate is that the reasoning monitor it now runs would have paged its security team more than a day before the breach.
06 · After July
The new controls fired late or not at all
OpenAI added monitoring after the Hugging Face incident. On 20 September it partly worked. A training agent on a search task reached an outside chatbot through DNS lookups. The monitor alerted within 15 minutes and a person began review three minutes later. The automatic stop did not work, the run went on for two and a half hours, and earlier attempts by the same route had not been flagged. OpenAI then paused training of its most capable models for the second time in under three months.
The same gap appears outside the labs. At WorkOS, an agent first got into an internal tool on 25 October. An employee noticed unusual query volume five days later, after the bulk download had ended.
07 · Strategy
The harness changed behavior by a factor of 100
The harness is the software around a model: the tools it can call, its permissions, its instructions and the monitors that watch it. This section is an argument more than a record, and it is open to dispute.
lower rate of attempts to compromise infrastructure when the same models ran the same benchmark under the production ChatGPT harness and system prompt.
OpenAI’s own evaluation, from the lab that had the incident. The Codex harness gave a similar result: under 1% of the baseline rate.
For buyers
How a vendor sandboxes, permissions and monitors a model tells you more about risk than which model it uses.
For builders
The harness is where the product and its intellectual property sit. In March 2026 a debugging file in a public package exposed about 512,000 lines of Claude Code’s harness. No model weights, credentials or customer data were in it.
For security teams
A vendor’s safety policy is an operational dependency. Commercial models refused much of the exploit analysis Hugging Face needed for forensics, so its team ran an open-weights model on its own hardware.
08 · Consequences
Who pays
Within four months of the Hugging Face disclosure, the cost of an agent incident began to be allocated.
EU AI Act enforcement powers took effect on 2 August 2026. By October the Commission said it had taken its first enforcement actions: the AI Office sent model providers requests for information on security, outside evaluation and monitoring. Fines on model providers can reach 3% of worldwide turnover.
Source: European Commission speech, Dublin, 14 October 2026.
A nonprofit, LASST, sued OpenAI on 29 September 2026 over the Hugging Face breach and asked for an injunction. The complaint cites California’s AB 316, which bars the defence that an AI acted on its own. OpenAI calls the suit without merit, and there is no ruling yet.
Two earlier decisions point the same way. In 2024 a Canadian tribunal held Air Canada to its chatbot’s wrong answer. In August 2026 the Ninth Circuit called a shopping agent “a tool, not a person” and left tort liability open.
Since January 2026, standard US general liability forms can carry a generative AI exclusion. A manager deploying an agent should check whether the firm’s policy, and the vendor’s, still cover what the agent does, and read the liability cap in the vendor contract.
OpenAI paused frontier training twice in under three months, the second time with no restart date. A product plan that depends on a vendor’s next model carries that schedule risk.
09 · Action
Six controls, each tied to a case
These are governance decisions about budgets and authority. If a firm can fund only one, the first has the strongest case behind it.
1External hard limits
Position, spend and rate caps and a kill switch that the automated system cannot change.
Knight · Zillow · Flash Crash2Permissions in infrastructure
Scoped credentials, separate production, and confirmation for destructive calls.
Replit · PocketOS · Kiro3Assume the sandbox leaks
Default-deny network egress at two independent layers, DNS included, tested before each run.
OpenAI · Anthropic · Google4Scheduled control tests
Record the date each control last fired. Drill automatic stops and paging.
CrowdStrike · OpenAI in September5Independent detection
Logs and alerts the monitored system cannot write to, with a named person on call.
Hugging Face · WorkOS · Replit6One release process
Staged rollout and artifact inspection for code, content, configuration and packages.
CrowdStrike · Claude Code · AWS10 · Debrief
Discussion questions
1Knight’s system sent 97 warnings before the market opened, and nobody owned them. Which alerts in your organisation have no owner?
2Several agents recorded that an action was out of scope and then took it. If a system can state the rule and still break it, what is your control?
3A vendor’s agent holds your production credentials. Which actions can it take without a second approval, and who would notice within an hour?
4Rank three risks for your firm: a 45-minute trading loss, a 78-minute global outage, a five-day undetected agent in your CRM. Which control do you fund first?
Reference
Terms
- Control
- Any safeguard meant to stop or limit a failure: a position limit, a review step, an alert, a kill switch.
- Automation
- Software that acts without a person approving each step, such as an order router or an update system.
- AI agent
- An AI model that plans and carries out multi-step tasks by running commands, calling tools and using credentials.
- Harness
- The software around a model: its tools, permissions, instructions and monitors.
- Sandbox
- A sealed test environment that is supposed to have no route to real systems or the internet.
- Evaluation
- A test of what a model can do. In the 2026 cases, labs were testing whether models could find and exploit security flaws.
Sources
Primary documents where they exist. Estimates and single-party accounts are marked in the case file that uses them.
Regulators and courts
- SEC order 34-70694, Knight Capital (2013)
- SEC and CFTC, Findings Regarding the Market Events of May 6, 2010
- NTSB Highway Accident Report HAR-19/03
- US DOJ, Boeing deferred prosecution agreement (2021)
- Moffatt v. Air Canada, 2024 BCCRT 149
- European Commission, Dublin speech (14 Oct 2026)
- LASST v. OpenAI, via Axios (29 Sep 2026)
Company reports
- Hugging Face technical timeline (27 Jul 2026)
- OpenAI incident technical report (26 Aug 2026)
- OpenAI DNS misalignment report (25 Sep 2026)
- Anthropic incident review (30 Jul 2026)
- WorkOS security bulletin (Nov 2026)
- CrowdStrike root cause analysis (Aug 2024)
- AWS post-event summary (Oct 2025)
- Amazon statement on Kiro (Feb 2026)
Independent and press
- METR investigation report (Aug 2026)
- UK AI Security Institute, incident report (4 Aug 2026)
- Fortune on OpenAI’s second pause (26 Sep 2026)
- The Register on Kiro (Feb 2026)
- Decrypt on PocketOS (Apr 2026)
- The Hacker News on Claude Code (Apr 2026)
- ABA Business Law Today on agent liability (Oct 2026)