The 59% That Never Ship: Why AI Agents Die Between Demo and Production Link to heading

An AI agent’s happy-path demo giving way to the long tail of production failures

Tian Pan framed this better than anyone I’ve read. In The Demo-to-Production Cliff, he makes a claim that sounds wrong until you do the arithmetic: a 90%-accurate per step agent isn’t 90% of a product. It’s an unshippable one. His diagnosis is airtight, so I’m not going to re-litigate it — I’m going to take it as given and push on the part that leaves practitioners stuck: how you make failure cheap on real infrastructure. That’s the question I put to 16 CTOs who deploy agents for a living, and their answers are what most of this post is about.

First, the part of Pan’s argument you need in your pocket.

The setup: why the math is brutal (Pan’s argument, compressed) Link to heading

An agent doesn’t take one action. It runs a chain of them, and reliability across a chain is multiplicative, not additive. A 5-step task at 90%-per-step doesn’t ship at 90% — it ships at 0.9⁵ ≈ 59%. Ten steps lands at 35%. Even demo-grade 95%-per-step puts a 20-step process at ~36%.

The demo hides this because a demo is a single trajectory — the happy path, short and rehearsed. Production is the full distribution: long chains, empty results, malformed inputs, tools that time out. The cliff lives in the tail, and one sample from the middle can’t show it to you. That’s Pan’s core insight, and the industry data says it’s not theoretical:

  • MIT’s Project NANDA found 95% of enterprise GenAI pilots delivered no measurable P&L impact (The GenAI Divide: State of AI in Business 2025).
  • Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027 (via Reuters).
  • IDC pegs enterprise adoption at 79%, but only 11% run agents in production — a 68-point gap (IDC).

Where I want to pick up the thread is everything after the diagnosis — because “make failure cheap” is the right instruction, and also the point where most write-ups stop and every real team gets stuck.

What actually kills agents in production Link to heading

I recently sat with 16 CTOs from global consultancies and systems integrators who deploy agents for a living. The same three gaps came up in every conversation. And they’re not separate from the math above. They’re what the failure tail looks like inside a real enterprise.

1. Cost is nondeterministic. An agent that goes wrong runs for hours and burns tokens. You can’t budget for it — and when it does run away, nobody can say who pays for it.

2. Trust without observability. When an agent errs, you own an outcome you can’t see or explain.

3. Reliability isn’t capability. What’s theoretically possible isn’t what’s reliable. Agents that ran 30–60 minutes last year now run for hours, and the errors compound.

Pan’s four levers — and the one that actually breaks teams Link to heading

Pan’s fix is to stop chasing per-step accuracy (diminishing returns against a multiplicative penalty) and instead make failure cheap, via four levers: detectability, reversibility, bounded blast radius, and graceful handoff. It’s the right framework, and all four levers earn their place. What I want to add is the implementation layer underneath them — because a framework this clean deserves to be picked up and made concrete, and that’s exactly what the CTOs were asking for.

So I’ll do that here: map each lever to the AWS primitives that implement it, and then go deep on the cost lever — bounded blast radius on spend — because it’s the one that most rewards a worked example, and the one the roundtable kept circling back to.

There’s a tension you have to name first. The reflex for accuracy is to reach for the biggest, most capable model — which is also the most expensive per token. Multiply that across a long, looping chain of nondeterministic calls and the bill explodes. The accuracy reflex is also the cost problem, and you don’t resolve it by picking a side. You resolve it in the architecture.

The precondition: Detectability (→ the trust gap) Link to heading

You can’t reverse, bound, or hand off a failure you never noticed.

Verify outcomes, not the absence of exceptions. An agent that returns “success” with a wrong result is more dangerous than one that throws. Instrument end-to-end: trace every step, decision, and tool call.

With detection in place, three levers make the failures it surfaces cheap.

Lever 1 — Reversibility (→ the reliability gap) Link to heading

Classify every tool as reversible or irreversible. Route the irreversible ones through staging, soft-delete, or a confirmation gate. A wrong-but-reversible action is an inconvenience; a wrong-and-irreversible one is an incident.

Lever 2 — Bounded blast radius (→ the cost gap) Link to heading

Cap the worst case before it happens. Nondeterministic cost is only unbudgetable if you leave it uncapped.

Borrow the cost bucket from distributed systems. Rate limiters have solved “unpredictable consumption” for decades with the token bucket algorithm: tokens refill at a fixed rate into a capped bucket, each request spends tokens, and when the bucket is empty the request is refused. Apply the same idea to agent cost:

  1. Calibrate in development. Measure the true cost of a successful agent action — averaged across both successes and failures, since failed attempts still burn tokens. That average becomes your refill rate.
  2. Meter in production. Give each agent action a cost bucket. An action that comes in under budget banks the difference back into the bucket; one that runs over draws it down. As long as the bucket has credit, bursts are absorbed.
  3. Fail closed when empty. If an action would exceed budget and the bucket is empty, it doesn’t proceed — exactly the “non-conformant packet gets dropped” behavior, applied to spend. The runaway agent is stopped structurally, not discovered on the invoice.

Run buckets at three scopes — per action, per session, and per user — so a single greedy step, a stuck session, and a heavy user each hit their own ceiling. The net effect: the average cost per agent action stays predictable even though any individual action is nondeterministic. That’s the runaway-agent cost problem, answered in the architecture.

The cost bucket caps the worst case; two tactics lower the baseline it caps, and both defuse the accuracy-vs-cost tension directly:

  • Intelligent routing (right-size the model to the step). Don’t run every step on a frontier model. Send simple, deterministic steps to small, cheap models (Amazon Nova Micro/Lite, Claude Haiku) and reserve the expensive reasoning models for the few steps that actually need them. Amazon Bedrock Intelligent Prompt Routing does this at the endpoint: routine work goes to the cheaper resource, and you escalate to the expensive one only when the task genuinely demands it.
  • Aggressive caching (prompts and tools). Agent chains re-send near-identical context and re-run identical tool calls constantly. Cache both: prompt caching for repeated context, and tool/result caching so a deterministic lookup isn’t paid for twice. On long chains this is often the single biggest line-item reduction.
  • Bedrock per-request token metering — the measurement layer that makes routing and the cost bucket enforceable per action/session/user.
  • AWS Budgets + Cost Anomaly Detection — hard caps and runaway-spend alerts with per-application cost-allocation tags.
  • AgentCore Runtime session timeouts / step limits — stop the “ran for hours” failure mode structurally.

Lever 3 — Graceful handoff Link to heading

When the agent is uncertain, it should stop and hand a human a clean picture — not guess confidently. A good handoff is a feature, not a failure.

  • Bedrock AgentCore human-in-the-loop — pause on low-confidence or high-risk steps and require explicit human confirmation before the agent proceeds (how-to).
  • Strands Agents — build HITL approval flows into the agent loop with AG-UI/CopilotKit on Amazon Bedrock AgentCore (how-to).

How do you know it’s working? Evaluate the tail, not the mean. Link to heading

Once the levers are in place, the measurement has to change too — and here I’ll borrow Pan’s sharpest point directly, because it’s the right one. “85% success” is a mean. It sizes the 90% and says nothing about the shape of the 10% that decides shippability. Two agents at 85% can be opposite products: one stops to ask when unsure; the other takes confidently-wrong, irreversible actions.

What the levers above change is which of those two agents you’re building. So score your failures by type, not just count — and tie the score to the levers: was the failure detected? reversible? did it stay inside its cost bucket? did it hand off cleanly? Build deliberately adversarial eval sets — empty results, malformed inputs, long chains, ambiguous instructions — and find the cliff in test, not in the incident channel.

The reframe Link to heading

The 11% who ship aren’t the earliest movers. They’re the ones who closed the reliability gap first — and IDC puts the average ROI for deployed agents at ~171%, so the prize is real.

Pan’s diagnosis tells you where the 59% goes: into an unmeasured failure tail. The 16 CTOs told me why teams don’t close it: the levers sound like principles, and principles don’t survive a sprint. Detectability, reversibility, and graceful handoff have obvious homes — Observability, Step Functions, human-in-the-loop. The one that stays abstract is cost, which is exactly why I gave it a concrete mechanism: a cost bucket, calibrated in dev, metered per action, session, and user, failing closed when it’s empty.

So if you’re scaling agents, the honest question isn’t “is the model good enough?” It’s this: of the four levers, which one is still a principle in your system instead of a line of code? For most teams I talk to, it’s cost. That’s where is your 59%.

Sources Link to heading

  1. The Demo-to-Production Cliff: Why a 90%-Accurate Agent Ships at 0%, Tian Pan — tianpan.co (the framing this post is in conversation with: the compounding math, the tail-vs-mean eval, and the four levers to make failure cheap. This post builds on his diagnosis and answers the “how” on AWS.)
  2. IDC Predicts 10x AI Agent Usage by 2027 — digitalapplied.com (79% adopted / 11% in production; ~171% avg ROI)
  3. Over 40% of agentic AI projects will be scrapped by 2027, Gartner says — Reuters via Yahoo (primary: Gartner Hype Cycle for Agentic AI)
  4. The GenAI Divide: State of AI in Business 2025 (MIT Project NANDA) — Fortune (95% of pilots, no measurable P&L impact — not “95% fail”)
  5. Field evidence: AWS partner CTO roundtable, mid-2026 — first-hand notes from a discussion with 16 CTOs (anonymized).