2026-07-19 · Will Milewski

Five documented agent incidents, and the one control that would have stopped each.

Zero model failures. Every one is a missing control. Including the honest answer to which of these our own product would have caught.

An AI agent deleted a production database and its backups in nine seconds. That was this April. It was not the first, and the pattern across the incidents that preceded it is remarkably consistent: broad credentials, no action layer, apology thread.

Here are five documented incidents, each with the single control that would have stopped it. Every one is public, dated, and sourced. None of them is a model failure.

1. Replit, July 2025

During an explicit code freeze in a public build series, the coding agent wiped a live production database: records for over 1,200 executives and nearly 1,200 companies, gone. Replit's CEO apologized publicly and shipped dev/prod separation within days (Fortune, AI Incident Database).

Missing control: the agent should never hold production credentials.

2. PocketOS, April 2026

A coding agent "resolved" a routine task by deleting a car-rental company's production reservation database and its backups, in nine seconds, via an infrastructure volume delete it assumed was scoped to staging. The newest recoverable backup was three months old (Euronews, ACS Information Age).

Missing control: a destructive-action gate. Deletes wait for a human.

3. Gemini CLI, July 2025

The CLI misread one failed command, then executed real destructive moves against a directory state it had hallucinated, wiping a user's project files. Its own post-mortem, in its own words: "I have failed you completely and catastrophically" (Slashdot).

Missing control: verify state before anything irreversible.

4. Air Canada, February 2024

The airline's chatbot invented a bereavement refund policy that did not exist. A British Columbia tribunal ruled the airline liable for the policy its chatbot made up, and ordered it to pay (CBC).

Missing control: commitments come from a system of record, not a language model.

5. Cursor, April 2025

During a session bug, the support bot invented a one-device login policy. Real customers cancelled subscriptions over a rule that never existed; the co-founder confirmed there was no such policy (AI Incident Database).

Missing control: provenance. No source, no answer.

The pattern

Five incidents. Zero model failures. In each case the model did roughly what a probabilistic text generator does under ambiguity; the architecture around it is what turned a wrong guess into a production outage or a legal ruling. This is not a new story. Knight Capital lost $440M in 45 minutes in 2012 to runaway automation with no AI involved. Agent incidents are the same class of failure with a faster trigger: broad permissions, no gate between intent and irreversible action.

Which of these would our own product have caught?

Full disclosure: we build a SQL firewall for AI agents, so you should ask which of these Terminus would actually have stopped.

Honest answer: one outright, one in pattern, three not at all.

The Replit class is the exact scenario. Behind Terminus the agent holds no database credentials, every query hits a default-deny policy, and a destructive write is denied or held for a human. PocketOS was the same missing gate at the infrastructure layer, not SQL: in pattern, but outside our enforcement point. The other three never touched a database.

If a vendor tells you one tool would have stopped all five, that is the sixth incident.

If your team is giving an agent write access to a production database, the enforcement core is open source on GitHub, and we are taking a small number of design partners: [email protected].

Default-deny. Fail-closed. Provable.