Agents rarely fail the way the demo suggested they might. They fail on the eleventh record, in a way nobody scripted.

The gap between a demo that works and a deployment that holds is not intelligence. It is exposure. A demo meets data that was chosen, users who know what to type, and a person watching every result. Production meets none of those, and the failure modes that follow are boringly consistent across orgs.

Failure one: data the agent was never designed to meet

The most common cause is not reasoning at all. It is a shape of data nobody anticipated.

A blank field where the prompt assumed a value. A picklist entry spelled three different ways across the same org. A record owned by a deactivated user. A description field containing an entire email thread including signatures and legal disclaimers. The model handles novel language well and handles novel data shapes badly, and production is mostly novel data shapes.

• Required fields that are only required on some record types.

• Free text where a picklist was intended, accumulated over years.

• Dates in the future that mean something different from dates in the past.

• Records created by an integration that follows none of the org conventions.

The fix is not a better model. It is running the agent against a sample of real records before launch, specifically the ugly ones, and deciding explicitly what should happen for each shape rather than hoping.

Failure two: permissions

An agent tested by an administrator passes every case, because an administrator can see everything.

The same agent running as a support representative silently returns partial results. Partial results are worse than an error, because nothing looks wrong. The response is fluent, well structured, and drawn from half the data. Nobody catches it until a decision is made on it.

This is the single most under tested dimension we encounter. Teams test what the agent does and almost never test who it does it as.

• Test as each persona that will use it, not as the person who built it.

• Assert on record counts, not just on whether a response was produced.

• Make the agent state explicitly when its view was restricted.

Failure three: scale of consequence

A demo writes to one record and a human watches it happen. Production writes to hundreds and nobody watches any of them.

The gap is not accuracy. A two percent error rate is genuinely impressive and completely invisible at demo scale. At a thousand records a week it is twenty wrong writes, landing in different accounts, discovered weeks apart by different people who each conclude the system cannot be trusted.

Volume also changes which errors matter. A wrong summary is an annoyance once and a pattern at scale, because people start correcting for it, and correcting for a system is how a system becomes shelfware.

Failure four: nobody can reconstruct what happened

When something does go wrong, the question is always the same: was the agent wrong, or was the data wrong.

Without a log that captures what the agent read, what it proposed, what it applied, and what a human changed afterwards, that question is unanswerable. Teams end up arguing from impressions, and the agent loses that argument regardless of whether it was at fault.

• Log the input, the proposal, the applied change, and the before value.

• Link every write back to the record that triggered it.

• Capture human corrections as a first class signal, not as an edit history.

What these have in common

None of these are model problems, so none of them are fixed by a better model. That is the useful conclusion, because it redirects effort somewhere it can actually land.

They are fixed by narrowing scope so the agent meets fewer shapes, enforcing confirmation on writes that are expensive to undo, testing as the running user rather than the builder, and logging every action with enough context to reconstruct what the agent believed at the time.

The short version

Demos fail on capability. Deployments fail on data, permissions, volume, and the inability to explain what happened.

Budget for the second list. It is less interesting and it is the entire difference between a pilot and something still running a year later.