How to Test an AI System Before You Blame the Model

A useful AI task fails twice, and the first response is often, “This model is bad.” Then someone tries a different model, changes the prompt, switches tools, and runs the test at another time. The next result is better, but nobody knows why. The team has spent time and money without learning what to fix.

You can avoid that confusion with a small controlled experiment. The goal is not to find the “best AI” for every kind of work. It is to find the setup that completes one real business task well enough, within your limits.

Start with one fixed business task

Choose a task that matters, but will not cause harm if a test goes wrong. A good example is turning a saved customer call transcript into:

• a five-bullet summary;
• three follow-up actions with an owner;
• a draft email under 150 words;
• a short list of facts that need human review.

Do not start with “help with sales” or “research our market.” Those goals are too wide to score. The test needs a clear finish line.

Use the same input file in every trial. Keep the instructions, examples, output format, and success rules fixed. Remove names, email addresses, payment details, health data, and other private information unless the approved system is allowed to handle them. If you must use sensitive data, get permission first and follow your company’s access, storage, and deletion rules.

Create an answer key before testing. For the call example, list the facts that a correct summary must include, the actions that should appear, and details that must not be invented. This gives reviewers one shared standard.

Separate the three layers

An AI system is more than a model. It has three parts that can affect the result:

1. The harness: The app or workflow around the model. It supplies instructions, files, tools, memory, permissions, and checks.
2. The model: The trained system that reads the request and produces or chooses the next response.
3. Serving: The service that delivers the model, including its speed, limits, errors, and availability at that moment.

If you change all three at once, a better result teaches you almost nothing. Change one layer at a time.

For example, first keep the model and service fixed while comparing two harness setups. One setup may receive only the transcript. The other may also receive the approved email style guide and a required output checklist. Next, keep the winning harness and service fixed while testing a different model. Last, keep the harness and model fixed while repeating trials through the same approved service at planned times.

Run repeat trials, not one lucky demo

One result can be unusually good or bad. Run at least three trials for each setup. Start each trial from a clean session so earlier messages do not leak into later results. Use the same source file and exact instruction. Record the date, time, model name, app or workflow, service route if known, and settings.

Randomize the order when you can. If setup A always runs first thing in the morning and setup B always runs during a busy afternoon, time may affect the comparison. Do not keep rerunning only the setup you want to win.

Hide setup names from reviewers if practical. Label outputs A1, A2, and B1 instead of showing a provider or product name. This helps people score the work itself rather than their expectations.

Use a copyable scorecard

Score every trial against the same rules. Write a short reason beside each score. A number without evidence is easy to bend later.

• Test ID:
• Business task:
• Date and time:
• Harness, model, and service:
• Changed layer:
• Quality, 0 to 5: Are the required facts correct, clear, and useful?
• Completion, 0 to 5: Did it produce every required item in the right format?
• Speed: How many minutes passed from start to a review-ready result?
• Cost: What did this trial cost, using the bill or usage record when available?
• Recoverability, 0 to 5: After an error, could it retry safely, explain the problem, and continue without losing work?
• Unsupported claims found:
• Human corrections needed:
• Permission or privacy issue:
• Final result: Pass, fail, or needs another test.

Agree on the pass rule before seeing results. A team might require quality and completion scores of at least 4, zero invented customer facts, no privacy breach, and a review-ready result within ten minutes. Those are example rules, not universal targets. Choose limits that fit the actual task.

Add stop conditions before you begin

A test should stop at once if the system tries to send a real email, edit a live customer record, purchase something, expose private data, or request access beyond the test. Use read-only files and a sandbox when possible. Turn off outside actions unless they are the exact thing being tested.

Also set a budget and time limit. Stop a trial after the agreed number of retries, minutes, or cost. Record the failure instead of helping one setup until it passes. If an output could affect a customer, employee, contract, payment, or legal decision, require human approval before any action.

Stop the whole experiment when one setup clearly passes the agreed bar across repeat trials, when all options fail the same critical rule, or when new runs are no longer changing the decision. If every setup fails, inspect the task and instructions before buying a different model. The work may be too vague, the source may be incomplete, or the harness may lack a needed tool or check.

Make the decision from the pattern

Look for a pattern, not a single total score. If quality rises after adding the right reference file and checklist, improve the harness. If the same harness and service produce better answers with another model across repeated trials, the model may be the layer to change. If answer quality stays similar but delays and errors cluster by time or route, investigate serving.

Cost belongs beside quality, not above it. The cheapest failed task is still a failure. The fastest output is not useful if a person must rebuild it. Recoverability matters because real work includes missing files, tool errors, and unclear inputs.

This testing method builds on Roy’s useful distinction between the harness, model, and serving layers, while leaving out his unverified provider comparisons. It also matches primary guidance from OpenAI on environments, context, tools, and feedback loops and Anthropic on using the simplest sufficient system and weighing performance against speed and cost.

If you want to test a bounded business workflow with clear permissions and review steps, explore Agentic Workers agents.