Engineering
9 min read
The first serious incident with an agent is not a technical problem. It is a forensics problem, and most systems fail it.
Someone reports that a record is wrong. The question is immediate and simple: did the agent do this, and if so, why. If the system cannot answer within a few minutes, the answer defaults to distrust, and distrust is very hard to reverse once it has a specific example attached.
Traces are not logs
Most agent frameworks produce a trace: the sequence of reasoning steps and tool calls for one run. Traces are useful for debugging and close to useless for operations.
A trace is organised around the run. Operations questions are organised around the record. Nobody asks what happened in run 4f8a. They ask what happened to this opportunity, and answering that from traces means searching every run for a mention of an id.
The fix is to log from the record's point of view as well: one row per write, on the record that was written, queryable by the people who own that record.
Trace for debugging the agent.
Record level log for answering questions about data.
Both, because they serve different people.
The four fields that matter
For every write, four pieces of information determine whether an incident is answerable.
The value before the change, captured at write time rather than reconstructed from field history later. The value after. The agent's stated reason, in its own words. And a link back to the input that triggered the action, whether that was a user message, a scheduled run, or another record.
With those four, the question of whether the agent was wrong or the data was wrong has a definite answer. Without any one of them it does not, and the argument becomes about impressions.
Before value, captured at the moment of write.
After value.
Stated reason, stored even when it is thin or wrong.
Source link back to the triggering input.
What to measure continuously
Beyond per action logging, a few aggregate signals catch problems before anyone reports them.
Correction rate is the most valuable. How often does a human change what the agent proposed. A rising correction rate is the earliest warning that something upstream has shifted, usually a schema change or a new data source.
Abstention rate is the second. How often does the agent decline to act. A sudden drop means it has become overconfident. A sudden rise usually means retrieval broke and it is no longer finding what it needs.
Action mix is the third and least obvious. Which actions fire, in what proportion. When that distribution shifts without a deployment, something changed in the data.
Make it visible to the right people
Observability that lives in an engineering dashboard answers engineering questions. The people who notice agent problems first are the ones using the records.
In a Salesforce context this argues for putting the log where they already work: a related list on the record, a badge on the page layout, a list view they can filter. The moment an administrator can see what the agent did to a record without asking anyone, the support burden drops and the trust conversation changes character entirely.
Surface agent actions on the record page, not only in a console.
Give administrators a list view of everything the agent did this week.
Make the reason human readable rather than a serialised object.
The short version
Log per write, not per run. Capture before, after, reason, and source. Watch correction rate, abstention rate, and action mix.
Then put all of it where the people who own the data can see it without asking. An agent nobody can inspect is an agent nobody will keep.