Evaluating Agentic Systems Without Vibes

Agent demos are easy to make impressive. Production agents are difficult to make dependable. The difference is evaluation.

An agent usually combines a language model with retrieval, tools, state, orchestration, and business rules. Looking only at the final answer hides which component failed and makes improvements difficult to measure. A useful evaluation system therefore works at several levels.

Define the contract first

Before choosing metrics, define what the system is allowed to do. For an agent that prepares a commercial offer, the contract might include:

• Use only inventory returned by approved tools.
• Never price below verified cost.
• Include a valid source URL.
• Preserve the buyer's hard constraints.
• Ask for clarification when required information is missing.
• Return a schema that downstream software can parse.

These rules are more useful than a vague target such as "produce a good offer". They turn expectations into testable properties.

Layer one: deterministic checks

Start with everything that ordinary code can verify. Deterministic checks are fast, cheap, and repeatable:

• Does the output match the schema?
• Are required fields present?
• Does the selected item exist in the tool response?
• Is price greater than or equal to cost?
• Did the agent exceed its tool-call budget?
• Did it attempt to call an unapproved tool?

If a rule can be expressed without a model, it usually should be. Using an LLM judge for arithmetic or schema validation adds cost and uncertainty without adding insight.

Layer two: component evaluations

An end-to-end failure often begins earlier in the pipeline. Evaluate important components independently.

Retrieval

• Did the correct evidence appear in the top results?
• Was irrelevant context excluded?
• Did chunking separate information that belongs together?
• Are sources recent and authoritative enough for the task?

Tool selection

• Did the agent choose the correct tool?
• Were arguments complete and valid?
• Did it retry only when retrying could help?
• Did it distinguish a tool failure from an empty result?

Planning and state

• Were hard constraints preserved across turns?
• Did the agent repeat completed work?
• Did it stop when the task was done?
• Did it recover from partial failure without corrupting state?

Component tests provide diagnostic resolution. They tell you whether to improve a prompt, retrieval policy, tool description, or orchestration rule.

Layer three: model-graded quality

Some properties genuinely require judgement: clarity, usefulness, faithful use of evidence, and whether a recommendation handles the user's underlying intent.

A model judge can help, but only with a precise rubric and examples of different score levels. Replace a broad question such as "Is this answer good?" with explicit criteria:

Constraint satisfaction: 0-2
Evidence grounding:      0-2
Decision usefulness:    0-2
Clarity:                 0-2
Unsupported claims:     0-2 penalty

The judge should return a score and a short reason. Periodically compare those judgements with human ratings to detect drift or systematic bias.

Layer four: end-to-end scenarios

Real users do not interact with isolated components. Maintain a scenario suite that covers the complete workflow:

• A normal successful case.
• Missing information or conflicting constraints.
• No valid inventory.
• A tool timeout or malformed result.
• Adversarial or irrelevant instructions.
• A case where the correct action is to stop.

Each scenario should define observable success conditions. Avoid asserting one exact paragraph when several responses could be correct.

Measure trajectories, not only answers

Two agents can produce the same final answer with very different risk and cost. Capture the trajectory: the sequence of tool calls, tokens consumed, latency by stage, retries, state changes, selected evidence, and final validation results.

Trajectory data reveals loops, unnecessary searches, brittle ordering, and expensive behaviour that a final-answer score will miss.

Build evaluation sets from failures

The best evaluation dataset is a memory of ways the system has failed. When a real issue appears:

1. Reduce it to the smallest reproducible case.
2. Add that case to the evaluation set.
3. Record the expected property, not only an exact output.
4. Verify the proposed fix improves the case.
5. Run the complete suite to check for regressions.

Over time, the suite becomes a behavioural specification for the product.

A practical scorecard

Area	Example signal
Correctness	Constraint and schema pass rate
Grounding	Claims supported by retrieved evidence
Tool use	Valid tool calls and argument accuracy
Efficiency	Cost, latency, and calls per successful task
Robustness	Recovery rate across failure scenarios
Safety	Policy violations and unsafe actions
Experience	Human usefulness rating and clarification rate

The main lesson

Agent reliability does not come from one better prompt. It comes from designing the workflow so that failures are observable, localised, and testable.

Use deterministic code for hard rules. Use component tests for diagnosis. Use model judges for genuinely qualitative properties. Use end-to-end scenarios for product behaviour. Then keep every meaningful failure as a permanent regression case.

That is how an agent stops being a demo and becomes software.