Stop Measuring AI by Token Price. Measure the Cost of a Completed Task.

A cheap AI model can still create an expensive workflow.

The model may cost almost nothing to run, but the task can still fail. A person may need to rewrite the instructions, check every answer, repair missing details, or finish the work by hand. The token bill looks small while the real cost stays hidden.

That is why teams should stop asking only, “How much does this model cost?” A better question is, “What does it cost us to get one useful task all the way to done?”

In a Y Combinator interview about open models, Jeffrey Morgan made the same distinction: low cost per token matters, but low cost per completed task is the more useful measure.[1]

Start with one repeatable task

Do not compare AI tools with a vague prompt such as “help with marketing.” Pick one piece of work that happens often and has a clear finish line.

Good test tasks include:

• turn a recorded meeting into assigned follow-ups;
• classify a support request and prepare a grounded draft reply;
• review a weekly report and flag missing numbers;
• research a prospect and update the correct customer record;
• collect approved source material for a social post.

The task should use the same kind of input each time. It should also produce an output that another person can inspect.

If the work changes completely on every run, you are not measuring a workflow. You are measuring a collection of unrelated experiments.

Define “done” before the first run

A fast answer is not the same as finished work.

Write a short completion checklist before testing. For a meeting follow-up workflow, “done” might mean:

1. every decision is captured;
2. every action has an owner and due date;
3. the tasks are created in the correct system;
4. uncertain details are marked instead of invented;
5. a person can review the result in two minutes.

This turns quality into something the team can inspect. It also stops a polished paragraph from receiving credit for work it did not complete.

The Agentic Workers Super Agent guide makes the same practical distinction between an answer and an outcome. Memory, tools, timing, and verification are part of the work, not optional extras around the model.

Measure the whole task cost

For each run, record five things:

1. AI usage

Record the credits, model charge, or other metered usage. The Agentic Workers pricing page explains how credits track model use and tool activity after a run.

2. Human setup time

Count the minutes spent finding files, cleaning inputs, rewriting instructions, or preparing the workspace.

3. Review and repair time

Count the minutes spent checking facts, correcting mistakes, filling gaps, and moving the result into its final system.

4. Failure cost

Record failed runs, duplicate actions, missing outputs, and work that had to be restarted. A run that costs little but fails often is not cheap.

5. Delay

Record the time from request to accepted result. Slow work can block another person even when the direct charge is low.

A simple comparison looks like this:

Cost of one completed task = AI usage + human setup + review and repair + failed-run cost

Keep delay beside that number as a separate service measure. Converting every minute into money can create false precision. The raw time is often enough to show which option creates less friction.

Compare systems, not model names

A model does not work alone. The same model can perform very differently depending on the context, tools, permissions, instructions, and checks around it.

That means a fair test should compare complete systems. Keep the task and input fixed, then change one important part at a time:

• the model;
• the instructions;
• the available business context;
• the connected tools;
• the review rule;
• the retry or stop condition.

Run the task several times before choosing a winner. One excellent result can be luck. One poor result can be a bad input. Repeated runs show whether the system is dependable enough for real work.

Use the smallest model that clears the bar

The best system does not need the smartest model for every step.

A routine classification may work well with a fast, lower-cost model. A difficult judgment, sensitive reply, or unusual exception may need a stronger model or a person. The useful design is often a route:

1. send routine work down the simple path;
2. check the result against clear rules;
3. escalate exceptions when confidence is low or risk is high;
4. keep a record of what happened.

This is better than sending every task to the most expensive option. It is also safer than forcing every task through the cheapest one.

Watch for cheap work that moves cost to people

A low usage bill can hide several bad patterns:

• the team reviews every line because the system provides no evidence;
• the output arrives in chat and someone must copy it elsewhere;
• failures are silent, so people keep checking whether the work ran;
• the system repeats actions because it cannot see prior work;
• unusual cases are guessed instead of escalated;
• no one can tell which input, tool, or rule produced the result.

These are workflow problems. Switching models may not fix them.

A managed agent should reduce this hidden work by keeping the right context, using approved tools, following a schedule, recording actions, and stopping safely when the task leaves its boundaries. That is the difference between buying cheap answers and building reliable capacity.

Run a one-week task-cost test

Choose one repeated task and track every run for a week. Use a small scorecard with these columns:

• accepted result: yes or no;
• AI usage;
• setup minutes;
• review and repair minutes;
• failed attempts;
• total delay;
• reason for escalation.

At the end of the week, compare the accepted results, not just the generated outputs. Keep the system that finishes more work with less human rescue. If none of the options clears the completion bar, fix the workflow before spending time on a new model.

The Y Combinator interview inspired this task-cost lens, but the useful lesson is broader than any one model provider: cheaper intelligence only creates value when the surrounding system can turn it into completed work.

If you want help finding the first workflow worth measuring, start with a free workflow audit. We will trace one recurring bottleneck and tell you whether it needs a process fix, ordinary software, or a managed agent.

Sources

[1] https://x.com/ycombinator/status/2095883627200659800