There is no public benchmark for your Salesforce org, and there never will be. That is the whole difficulty of evaluating an agent that does real work.

Public benchmarks measure general capability, which is useful for choosing a model and useless for knowing whether your deployment is improving. The questions that matter are local: does it get this process right, for these users, on this data.

Score outcomes, not responses

The instinct is to score responses for quality, usually with a model as judge. This produces a number that feels rigorous and moves for reasons nobody can explain.

A better starting point is to score outcomes, because outcomes have correct answers. Did the field end up with the right value. Was the task assigned to the person who could actually do it. Did the article get drafted from the recap it claims to summarise, or from something else.

Outcome scoring is harder to set up and vastly easier to trust. When it moves, you know why.

• Field level accuracy against a known correct value.

• Assignment correctness, measured against who actually completed the work.

• Whether a human edited the output before accepting it.

• Time from trigger to resolved, compared against the manual baseline.

Build a frozen set from real cases

Take fifty real cases from the last quarter. Have someone who knows the process write down what should have happened for each. Freeze it.

That frozen set is worth more than any general benchmark, because it encodes the judgement of the people the agent is meant to help. It is also the only artifact in the whole programme that stays valid across model releases.

Fifty is enough to start. The temptation is to build five hundred and never finish, and a frozen fifty that runs on every change beats a perfect five hundred that runs never.

• Draw from real records, including the messy ones.

• Have a domain expert define correct, not the engineer.

• Include cases where the correct answer is to refuse.

• Freeze it and version it. A set that changes cannot show a trend.

The two numbers that predict trust

Then measure the two things that actually determine whether people keep using it.

The first is how often the agent declines when it should. An agent that recognises it lacks the information to act, and says so, builds confidence quickly even when it is unhelpful.

The second is how often it acts when it should have declined. This is the number that ends deployments. An agent that abstains too often is annoying and survives. An agent that acts confidently on a bad read is a liability, and one visible instance can cost more trust than a hundred correct actions earn.

The difference between those two numbers is entirely a threshold you chose, which means it is a product decision rather than a model limitation.

When to re-run

Re-run the frozen set on every prompt change, every model upgrade, and every schema migration.

Most regressions arrive through the third one, and almost nobody tests for it. Someone renames a picklist value, adds a required field, or changes a page layout, and an agent that worked yesterday now fails on a case nobody connected to the change. The schema migration is a normal piece of administrative work that nobody thinks of as an AI change.

• Wire the evaluation into the same pipeline that deploys metadata.

• Alert on a drop, not on an absolute score.

• Keep the last ten runs so a slow decline is visible.

What not to measure

Some numbers look like evaluation and are not.

Token cost per run is an operations metric, not a quality one, and optimising it early usually means cutting context that was doing useful work. Response length tells you nothing. User satisfaction surveys measure the interface more than the agent. And a model grading its own output is measuring agreement with itself, which is not the same as being right.

The short version

You will not get a leaderboard. You can get something better: fifty real cases with known correct outcomes, frozen, re-run on every change.

Measure whether it declines when it should and acts when it should not. Those two numbers decide whether anyone is still using the thing next quarter.