Trust and quality notes
- Last updated
- October 10, 2026
You set up an agent to prepare a weekly client report. It gathers the numbers, writes the summary, and puts the draft in front of a person for review.
The first run looks impressive. The second run saves some time. By week four, nobody can answer the question that matters: Is this helping the business, or are we just watching a clever demo repeat itself?
That gap is becoming easier to see. In its September 2026 State of Markets II, a16z describes AI adoption as broad but still shallow. Its full presentation says a small group of power users is pulling away while most use remains limited. The report also points to plenty of company stories about AI results, but far fewer teams tracking a useful operating measure.
The answer is not a bigger AI strategy. It is a small scoreboard for one repeated job.
Start with a finished piece of work
Do not begin with tokens, prompts, model names, or hours spent inside an AI tool. Begin with something a customer or teammate can recognize as complete.
For a weekly client report, the unit might be one report delivered on time and accepted by the account lead. For an appointment reminder workflow, it might be one correct reminder sent with the right approval. For lead research, it might be one prospect record that passes review and is ready for outreach.
A useful unit has a clear finish line. “The agent ran” is not a finish line. “The report was accepted and sent” is.
Write the unit in one sentence:
One completed result is ________.
If two people would fill that blank differently, settle the definition before measuring anything else.
Record the old way once
A result has no meaning without a comparison. Before changing the workflow, record a small baseline from the way the work happens today.
You do not need a six-month study. Use five to ten recent examples when the work is frequent enough. Record:
- how long the work took from start to finish;
- how many minutes a person spent actively working on it;
- how many corrections were needed;
- how often the work missed its deadline;
- what the work cost, including software and paid labor;
- whether the final result was accepted.
Keep the numbers honest. If you do not have a reliable old record, label the baseline as an estimate. Do not turn a guess into a savings claim.
Track four numbers for the agent
A simple scorecard can answer most early questions.
1. Accepted output rate
Count how many completed results pass review without being rebuilt.
Accepted output rate = accepted results divided by total attempted results.
This keeps the focus on usable work. An agent that produces ten drafts and only two survive review did not complete ten pieces of work.
2. Human review time
Track the active minutes a person spends checking, correcting, and approving each result.
Review is not failure. It is part of many responsible workflows. The problem appears when review takes as long as doing the work by hand, or when nobody planned for it.
3. Exception rate
Count the runs that leave the normal path. Examples include a missing file, an unclear instruction, a disconnected account, a request that needs approval, or a result that should be escalated to a person.
Then name the exceptions. A single percentage can hide the difference between a harmless missing field and a serious permission problem.
4. Cost per accepted result
Add the direct model, tool, and review costs for the test period. Divide that total by the number of accepted results.
This is more useful than cost per run. Cheap failed work is still failed work.
Run a small, bounded test
Choose one repeated workflow and run it long enough to see normal variation. Ten to twenty attempts is often enough to expose obvious problems, but the right number depends on how often the work changes and what happens when it goes wrong.
Set the rules before the test:
- which inputs the agent may use;
- which actions it may take;
- what always needs approval;
- what counts as accepted;
- when the agent must stop and ask;
- who reviews the result;
- what would cause the test to end early.
Keep the original process available while the test is running. A reversible test makes it easier to learn without turning every mistake into a client problem.
Compare outcomes, not activity
At the end of the test, compare the agent workflow with the baseline.
A good result might look like this: accepted output stays steady, review time falls, deadlines improve, and the cost per accepted result is reasonable. That is evidence to expand carefully.
A mixed result is still useful. Perhaps the agent is fast with routine reports but fails whenever source files arrive late. That tells you to improve the intake step, not buy a more expensive model.
A poor result is useful too. If corrections remain high and review time does not fall, pause. The least complicated fix may be a better form, a checklist, or ordinary software rather than another agent.
Scale the rule, not just the tool
When one workflow works, save the definition of done, permissions, approval points, exception rules, and scorecard. That operating rule is what makes the next client or team easier to support.
For AI service providers, this matters more than a flashy first demo. One client can be managed from memory. Ten clients need separate workspaces, clear approvals, and a record of what happened. The team needs to know which result is ready, which one needs a person, and which workflow is quietly creating more work than it removes.
The market may be early, as the full a16z presentation argues. That is not a reason to wait. It is a reason to get better at proving what works.
Pick one repeated job. Define the finished result. Measure accepted work, review time, exceptions, and cost. Then let the evidence decide what deserves to grow.