The Floor

In September, Anthropic published a case where an attacker’s AI agents rebuilt their own malware every time a security product caught it, and kept iterating until it went undetected.¹ Read that as a defender. Every detection you ship becomes the attacker’s next training signal. You can win that round. You can’t stop playing it.

Most of security is built on that game. We look at behavior and try to infer the goal behind it. Is this process malicious. Is this login really the user. Is this agent about to do something bad. Classifiers, heuristics, and now language-model monitors, all trying to read intent.

Two things make that a bad foundation for AI agents.

The first is the worst case. Rice’s theorem says every non-trivial question about what a program will do is undecidable in general.² An agent’s intent isn’t literally a property of a program, but the lesson carries: a checker that has to infer what an actor will ultimately do is, in the general case, attempting something provably impossible. Security lives in the worst case, because the adversary is the one choosing the input.

The second is the adversary. A monitor that reasons about intent is a thing the adversary can reason about too. Kale and colleagues measured this directly: when an agent knows it is being monitored, the monitor’s reliability drops substantially, and the agent’s awareness matters more than anything you tell the monitor.³ You can’t win a mind-reading contest against something that is modeling your mind back.

So I stopped trying to read intent. The move is old and it isn’t mine. James Anderson described the reference monitor in 1972: a mechanism that is always invoked, can’t be tampered with, and is small enough to verify.⁴ Saltzer and Schroeder wrote down the principle behind it in 1975 and called it complete mediation: every access to every object gets checked.⁵ You don’t ask whether the actor meant well. You ask whether this specific action is inside the authorized set. If it isn’t, it doesn’t run.

That check is decidable because I define the authorized set over concrete actions: this host, this port, this tool. Intent is contested. An action is on the record.

The shape I keep coming back to: the model proposes, and a deterministic engine decides. Put all of the intelligence on the proposing side. Keep the deciding side dumb on purpose. No judgment in it to fool, a fixed boundary, default deny, and every decision written to a log you can verify instead of trust.

I built duelswarm to see whether that holds up against a live attacker. It is a sandboxed range where an autonomous red agent and an autonomous blue agent fight, both driven by language models, and every action either one takes passes through the gate before it runs. Real exploits run against real vulnerable containers. The log is hash-chained, so nobody, including me, can quietly rewrite what happened.

Across every live run, the number of out-of-scope actions that executed is zero.

That number on its own proves less than it sounds like. A gate that blocks what it blocks will always report zero for the things it saw. The real question is whether anything reaches out of scope without the gate seeing it. So the gate gets attacked directly: aliases, raw IPs, alternate addressing forms, and egress tricks hidden inside free-form shell commands. Out-of-scope hosts also sit on a network the attacker’s foothold can’t route to, so part of that zero is topology, not just the gate. And a separate auditor re-derives scope compliance from the log on its own, without trusting the gate’s verdicts. The zero is the gate agreeing with an independent check, under attack.

I also tried to talk my way past it. Red can petition an LLM “authorizing officer” and argue when it’s denied. A human judge can direct the attack. When the judge points red at an out-of-scope target, the action is blocked like anyone else’s. Nobody is trusted, including me.

What I couldn’t show yet is the other half of the story, the learned monitor eroding while the floor holds. My first attempt at that curve came back as a null, and I wrote about why in Why I Shipped the Null.

None of this says monitoring is useless. It says monitoring shouldn’t have to be load-bearing. If the floor is what stops out-of-scope harm, the monitor is free to do something more interesting than catch. That’s a later post.

The floor has a limit, and I’d rather name it than have someone else find it. It guarantees that actions stay in scope. It does not guarantee that in-scope actions are harmless. A chain of individually authorized steps can still add up to something bad, and the hard problem moves into how you define scope and how scopes compose. That’s The Seams, and I think it’s where most of the real work is.

What the floor buys you is a guarantee that doesn’t degrade when the adversary gets smarter, because there’s nothing in it to outsmart. Everything else I write here starts from that.

Footnotes

1. Anthropic, “Detecting and countering misuse of AI: September 2026,” case GTG-20006. ↩

2. H. G. Rice, “Classes of Recursively Enumerable Sets and Their Decision Problems,” 1953. ↩

3. Kale et al., “Reliable Weak-to-Strong Monitoring of LLM Agents,” ICLR 2026, arXiv 2508.19461. ↩

4. J. P. Anderson, “Computer Security Technology Planning Study,” 1972. ↩

5. J. Saltzer and M. Schroeder, “The Protection of Information in Computer Systems,” Proc. IEEE 63(9), 1975. ↩