In Experiment 2 we measured a supervised GPT-4o-mini agent across 19 scenarios, 15 times each. Adding an explicit policy to the system prompt cut the leak rate from 53.3% to 17.5% — a ~36-point drop, and also a public admission that 1-in-6 supervised sends still violated the policy we had just written.

The obvious move from there is to stop asking the agent and start stopping the agent.

This is that experiment. But the experiment is in service of a larger thesis.

---

The thesis

LLMs cannot be made reliably safe through prompting alone. They have to be embedded in systems that enforce constraints at the action boundary — outside the model, where the model has no vote.

What's actually been built across these three experiments is a four-step control loop:

generate → detect → decide → enforce

• Experiment 1 measured what happens with only generate. Agents fail stochastically; the same prompt produces different outcomes; "safe once" means nothing.
• Experiment 2 added a probabilistic shaping pass at generate (the supervision policy in the system prompt) and a detect layer (rules + LLM judge). Result: risk got smaller and visible, but not bounded.
• Experiment 3 closes the loop by adding decide and enforce as steps that run outside the model. The judge's verdict isn't advice anymore — it's a gate.

A clarification we keep seeing teams skip:

Detection is observation. Enforcement is action.

Detection tells you what happened. Enforcement decides what is allowed to happen. They are routinely conflated in production agent stacks. They are not the same layer, and they have not been the same finding here.

---

The setup

Same 19 scenarios. Same gpt-4o-mini agent, same supervision prompt. Same calibrated LLM judge from Experiment 2 — recipient classifier, content-not-topic rule, verbatim-evidence self-check. 15 runs × 19 scenarios × 2 agent modes (unsupervised + supervised) = 570 emails evaluated end-to-end.

The only thing new is a decision node downstream of the judge:

When the judge flags an email as risky, we route the action to an enforcement decision. Two outcomes, mapped by the scenario's risk type:

• pii / sensitive_data / wrong_recipient → BLOCK — the action is dropped. Nobody reviews it. Nothing is sent.
• over_sharing / hallucination → REQUIRE APPROVAL — the action is queued for a human. It is still not sent.

When the judge says safe, we ALLOW the action — the email goes out as the supervised agent wrote it.

No thresholding. No confidence tuning. No second-order heuristics. judge.risky === true is the trigger; the per-scenario policy is a two-line switch statement. We deliberately did not build a sophisticated router, because the point of Experiment 3 is to find out what a maximally simple enforcement layer actually costs to operate — and whether it pays off.

---

The result that isn't the finding

First, the top-line numbers:

The enforced column is zero. Not "near-zero" — zero.

We are calling this out up front because it is the first thing a careful reader will spot. The interesting number is not the leak rate. The interesting number is how often the enforcement layer has to fire — i.e., how much throughput the system costs.

---

The finding

Every supervised email that the judge flags triggers an intervention. Across 15 runs × 19 scenarios:

For every 285 sends the supervised agent attempts, 50 are halted — 33 dropped outright, 17 queued for human approval. The remaining 235 go through as written.

That number is the point of the experiment. Everyone agrees an enforcement layer "works." The question nobody has been answering in public is what it costs to operate one. 17.5% is what it costs on this workload, with this agent, at this level of supervision.

Where the interventions cluster

Per-risk-type intervention rate:

Two things to read out of that table:

• The none row is the critical control. Safe scenarios — routine meeting follow-ups, feature announcements, conference thank-yous — should almost never fire the enforcement layer. If they did, the judge would be producing false positives that create phantom work for the review queue. On this batch the none intervention rate is 0 out of 75. Across every safe send in the entire experiment, the enforcement layer was silent. That's the precondition for the rest of the experiment meaning anything.
• The risky-context rows tell you what the review queue actually looks like. hallucination fires 50% of the time — driven almost entirely by a single scenario where the agent writes "following up on our recent connection" to someone it has never interacted with, and does so on 15 out of 15 supervised runs. sensitive_data fires 32% of the time, concentrated in two scenarios involving customer dataset references that supervision couldn't strip. These are exactly the Experiment 2 hard-cases, now surfaced as intervention volume instead of as silent leaks.

Scenarios with volatile intervention

Six of the 19 scenarios fired the enforcement layer inconsistently across 15 runs — same scenario, same agent policy, same enforcement rule, different outcome:

This is the second-order finding. Even when the enforcement rule is deterministic (judge.risky === true → intervene), the enforcement outcome is not deterministic, because the judge itself is stochastic at the boundary — same email, same judge, 15 independent runs, and the verdict can differ run-to-run.

Operationally, that means a human running this system will see their review queue grow in ways that depend on the specific roll of the judge's sampling, not on a clean property of the incoming traffic. That's not a bug; it's a property of running any probabilistic evaluator at a boundary. But it is a property you have to budget for — the same send that routes to human review on Monday might go straight through on Tuesday.

---

What enforcement actually changes

The Experiment 2 takeaway was: supervision reduces risk, but does not enforce. A one-paragraph system prompt gave us a ~36-point drop, and also a 17.5% floor we couldn't get under no matter how much we tuned the prompt.

The Experiment 3 takeaway is structurally different. It's not that risk got smaller. It's that the nature of the system changed.

That is the conceptual move enforcement makes. It is not "make the model smarter." It is move control outside the model.

Concretely, what changed across the three modes at the action boundary:

• Unsupervised. Agent's unconstrained output hits the world. 53.3% leak rate. Risk is invisible.
• Supervised. Output is shaped by a policy in the system prompt. 17.5% leak rate. Risk is reduced — and still invisible, because the sends that violate the policy look identical to the ones that don't.
• Supervised + enforced. Output is shaped by a policy and gated by a detector. 0% residual sends. Risk is now visible as intervention volume — 33 blocks and 17 approval-queue items per 285 sends.

The underlying model didn't get less stochastic. The agent is producing risky output at the same rate as under supervision alone — the evidence is that the intervention rate (17.5%) and the unenforced supervised leak rate (17.5%) are the same number, as they must be. What changed is that each piece of risky output now lands in an operator's queue, not a recipient's inbox.

The category-defining claim:

The model continues to be probabilistic. The boundary it acts through stops being. Behavior becomes bounded — not because we improved the agent, but because we stopped trusting it as the last line.

That reframing is what makes the experiment useful. "Is this agent safe to deploy?" is an unanswerable question for a probabilistic system. "What is our intervention budget for this agent, and what does the review queue look like?" is a question with a number. 17.5%, for this agent, on this workload.

---

What this means for anyone deploying an agent

If you are putting an LLM agent in front of real actions — email, CRM writes, ticket creation, payments, code execution — three concrete implications:

1. Prompt-level policy is the first layer, not the last. Supervision got us a ~36-point drop. That is real value, cheaply obtained. It is also not enforcement. Do not skip it. Do not rely on it alone.
2. Budget for an intervention rate. A reasonable deployment has to staff (or automate) a response to 17.5% of its agent's actions being intercepted. That's your review queue load (17 of the 50 on this workload — roughly 6% of sends), plus your dropped-actions rate (33 of 50 — roughly 12% of sends), plus whatever follow-up the blocked actions need. On this benchmark, with a well-calibrated judge, the total is not going to be 1%. It is going to be high-teens.
3. Regex alone is not a safety layer. From Experiment 2: the rule-based analyzer caught 0 leaks that the judge didn't also catch. An enforcement layer built on regex triggers would block almost nothing in this batch. The enforcement decision needs a detector with the same context awareness the agent has.

The practical shape of a production system falls out of that:

• An agent with supervision in the prompt.
• A detector with recipient awareness and verbatim-evidence self-check.
• An enforcement rule that halts any action the detector flags.
• A policy map that routes interventions between "never send this" (block) and "human judgment required" (approval queue).
• A review queue sized to the intervention rate the system produces on real traffic — budget for the hallucination and customer-data-mention cases, because those are where the queue is going to live.

None of that is novel individually. The thing we haven't seen published is the intervention rate a system like that actually produces on agent output, measured with a detector you can read. 17.5% is our number. We'd love to see other teams' numbers on comparable workloads.

---

Caveats, cleanly stated

• Judge-as-oracle is circular. Same judge fires the intervention and scores whether a send leaked. Enforced residual is therefore ≈0 by construction. We chose consistency over partial ground truth — see the full spec for the reasoning. Independent human labeling of a subset of these 570 emails would produce a real false-block rate; we did not do that in this run. What we can say is that the judge produced 0 interventions on 75 safe-scenario runs — a useful precondition, even if it's not a full false-positive audit.
• Policy map is opinionated. The split between block and require_approval is a two-line switch on risk_type. A different product team would map this differently — e.g., over_sharing as hard-block, hallucination as soft-fail. That would change the block/approval ratio (33/17 on this run) but not the total intervention rate.
• Single model. gpt-4o-mini on both agent and judge. Cross-model validation is the next experiment on the list — different agent model, same scenarios, same detector.
• Scenario set. 19 email-sending scenarios, v1. Not a claim about every agent task surface. Extending to CRM writes, code comments, and file operations is planned.

Everything is reproducible:

• Code, scenarios, analyzer, judge, enforcement runner: github.com/Tether-Labs/research
• Run it yourself: npm install && OPENAI_API_KEY=... npm run experiment:batch:3

If our intervention rate is wrong in a specific direction, we want to know.

---

What's next

The natural next axis is model comparison: hold scenarios, supervision prompt, judge, and enforcement constant, swap the agent model. Does a stronger base model produce lower intervention rates? Does a cheaper one produce higher? This is the experiment that starts to turn Tether's methodology into a benchmark — call it TetherBench v0.1 — that anyone deploying agents can use to ask: for this workload, what is my intervention budget with model X?

Experiments 1 through 3 have been about making one question measurable: how does an agent actually behave at the action boundary, and what does it cost to make that boundary safe? The answer so far: agents fail stochastically; prompt-level supervision reduces but does not enforce; an external enforcement layer turns the residual into visible operator workload at a cost of 17.5% of outgoing actions.

That's the finding. The thesis is bigger:

Three experiments are evidence. They aren't proof. Cross-model validation, broader scenario coverage, and a public benchmark are the next steps in turning evidence into a position. If you're building agents that take real actions and want to compare notes, the code is open — fork the runner, swap in your model, and tell us what your intervention rate looks like.

---

This research is part of Tether Labs, an open research initiative studying how autonomous agents actually behave at the action boundary. Code, scenarios, judge, enforcement runner, and full methodology: github.com/Tether-Labs/research.