Stop Writing Eval Harnesses. Start Directing Them.

Most enterprise agent programs do not stall on model quality. They stall on evidence.

I spend a lot of time with engineering leaders in contact centers, insurance, and financial services who have a voice agent that demos beautifully and a review board that will not sign off. The blocking question is never "is the model good enough." It is some version of: show me what happens when a caller is frustrated, references a policy number that was cancelled last month, and asks for a human three times.

Nobody answers that quickly. Answering it requires someone to build a test harness, and the person who knows exactly what a bad call sounds like is almost never the person who writes Python. Your conversational designer can describe the failure in forty seconds. Getting it into a repeatable test takes a sprint and a ticket.

That gap is the bottleneck. Not inference cost, not model selection, not the framework debate. The gap between knowing what to test and being able to test it.

Connecting Claude to Okareo closes it.

What the connection actually is

Two pieces, and the distinction matters.

The Okareo MCP server gives Claude the tools: the callable actions against Okareo. Save a scenario. Define a driver persona. Register a target. Run a simulation. Pull a transcript. Re-score a completed run against different checks. Query the analytics cubes. Roughly sixty tools covering scenarios, models, checks, simulations, voice monitoring, and dashboards.

The Okareo Skills give Claude the method: when to reach for which tool, and how to run a real workflow instead of a sequence of API calls. Tools without method produce a copilot that can technically call run_simulation and has no idea what a good simulation looks like.

Setup is one config block pointing at the hosted endpoint at tools.okareo.com/mcp. No install, no Python, no container. First connect opens a browser for OAuth sign-in and the client stores the token from there. There is no API key sitting in a checked-in config file, which is the detail your security reviewer will care about more than anything else in this post. Headless CI can fall back to a bearer header.

There is a wrinkle worth naming: the Claude entry points are not identical. Claude Code installs everything as one plugin. Claude Desktop and claude.ai take the MCP server as a connector and the skills separately, per user. The API takes skills through the container parameter and authenticates the MCP server with a bearer token. Skills do not sync between surfaces.

The thing that resolves most of the resulting confusion is architectural: the state lives in Okareo, not in the editor. Scenarios, drivers, targets, runs, checks, and dashboards are all server-side. An engineer authors a scenario set in Claude Code, a conversational designer extends it from claude.ai that afternoon, and CI runs it against a pull request that night. Three surfaces, one project, one audit trail. The surface is just the driver's seat.

Four surfaces, one endpoint, one server-side project.

Now the part that actually earns the setup.

Workflow one: scenarios and drivers have to pair off

This is the concept teams get wrong most often, and getting it wrong makes every downstream number meaningless.

A driver is the simulated user: persona, tone, objective, escalation behavior. "Frustrated caller who wants a refund and will ask for a supervisor if not resolved in three turns."

A scenario is the substance: the specific facts that make a given conversation real. The part number. Whether it is backordered. The customer's order history. Which return window applies.

Run a driver without scenario data and you get a persona improvising. It invents a part number, your agent correctly reports that it cannot find that part, your check passes, and you have tested nothing except your own imagination. The scenario is what supplies the ground truth that makes both the conversation and the check meaningful.

Same driver, same agent, same checks. Only the grounding changed.

Here is where MCP changes the economics. Claude can read your actual system to build the scenario set. Point it at your parts catalog, your order schema, your policy articles, your existing test fixtures, your agent's own system prompt and tool definitions. Then ask for coverage:

"Read the parts catalog and the returns policy in docs/. Build a scenario set of 40 rows covering in-warranty returns, out-of-warranty returns, partial shipments, and backordered items. Include the actual SKUs and the real policy windows. Save it as 'Returns Coverage v1'."

"Now create three drivers against that set: a patient first-time customer, a frustrated repeat customer who has already called twice, and a caller who talks over the agent and changes their mind mid-request."

Forty scenarios crossed with three personas is a hundred and twenty distinct conversations, every one of them grounded in a real SKU with a real policy consequence. That used to be a sprint of harness code. It is now an afternoon of direction, and the person doing the directing can be the one who actually knows the returns policy.

Workflow two: sample the space, do not carpet-bomb it

The default instinct on a new agent is to run one enormous simulation. A thousand conversations, one number, ship it.

A thousand conversations gives you an aggregate score. It does not give you a diagnosis. And on a voice agent, it gives you a real invoice line and an hour of wall clock before you learn anything.

Restructure the same budget as a search instead of a survey:

Stage one is a smoke test. Twenty conversations spread thin across the whole behavioral surface. You are not measuring quality yet. You are looking for the dimension that is bleeding.

Stage two concentrates. Policy accuracy came back at 0.42 on partial-shipment refunds and clean everywhere else. Claude composes a focused run: sixty conversations on that one workflow, varying one dimension at a time. Patient versus frustrated. In-warranty versus out. Single item versus partial. Same target, same checks, narrowed aperture.

Stage three isolates. The failure only appears with frustrated callers on out-of-warranty partial shipments. Now you know whether you are looking at a retrieval problem or a de-escalation problem, and you know it from twenty conversations instead of a thousand.

The same budget, spent as a search instead of a survey.

This is adaptive sampling, and it is a genuinely awkward thing to express in a test script because the next run depends on the last run's results. It is a natural thing to express to a copilot that just read the results and holds the whole project context.

One more lever most teams miss: re-scoring is free. reevaluate_test_run scores transcripts you already paid for against a different set of checks. New compliance requirement lands on Tuesday? Score last month's runs against it before you place a single new call. Starting an investigation from a re-score rather than a fresh simulation is not a cost trick. It is evidence-first discipline, and it is faster.

Workflow three: production writes the next test set

The best scenario set you will ever have is the one your customers already generated.

scenario-from-traces takes production conversations, incident reports, and support tickets and turns them into a structured Okareo scenario set. The call where the agent confidently quoted the wrong deductible becomes a row. The escalation that should have happened four turns earlier becomes a row. The ambiguous phrasing that broke intent routing becomes five rows, because you want the variations too.

That set then runs on every change, forever. This is the loop that separates teams who ship agents from teams who keep re-litigating them: every production failure gets converted, once, into a permanent regression test, and the conversion is a conversation rather than a ticket.

Pair it with monitoring on live traffic and the pipeline closes. Drift shows up in production, becomes scenarios, gets simulated pre-release, and stops shipping.

Where it goes from there

Two more layers, briefly, because each deserves its own treatment.

Closed-loop improvement. /okareo:improve sets an objective and runs cycles: simulate, analyze failures, propose a fix, apply it, re-verify, track the delta. Human approval at each gate. We wrote that one up separately in Your Co-pilot Runs the Loop, including a real three-cycle narrative where the fixes landed in three different places, one of which was our own test harness.

Reporting. The reps family (reps-explore, reps-profile, reps-run) handles the part nobody enjoys: producing the standardized assessment your risk committee, your enterprise customer's security team, or your own quarterly review actually reads. reps-explore holds a few benign discovery conversations with an unknown agent and drafts a profile of its domain, intents, tools, and guardrails, with provenance on each inferred fact. reps-run executes a pillar and regenerates the report. The output is a document a non-engineer can act on, generated from real runs rather than assembled by hand the night before.

What this changes for the business

Three things, in the order your CFO will care about them.

Cycle time. The distance from "I wonder how it handles X" to a scored, repeatable, versioned test drops from days to minutes. Testing stops being a phase and becomes something that happens continuously, which is the only way it survives contact with a shipping schedule.

Who does the work. Conversational designers, QA leads, and compliance reviewers become first-class users of the test bench. They stop filing requests against an engineering backlog and start authoring coverage directly. On the accounts where this has taken hold, that reassignment is the change with the largest effect, and it is the one nobody predicts.

Auditability. Every scenario, run, transcript, and score is a versioned artifact in Okareo with a timestamp on it. When the regulator or the enterprise customer asks how you validated the agent, the answer is a report, not a Slack thread.

The sharp edges

Because they are real and you will hit them in the first hour.

The skills call a lot of MCP tools, so Claude Code will prompt for each one by default. Add a single allow rule for mcp__okareo__* and commit it to the project so your team inherits it. Skills still confirm in-chat before anything billed or destructive, so a server-wide allow rule does not mean silent charges.

Skills install per surface and do not sync. Plan for that if you are rolling out to a team across Claude Code and claude.ai.

And the copilot proposes; you approve. An agent that edits its own system prompt unsupervised is not an improvement loop, it is a liability. Every workflow here keeps a human at the gate on purpose.

Start with the MCP configuration guide, or install the plugin directly in Claude Code:

/plugin marketplace add okareo-ai/okareo-tools

/plugin install okareo@okareo-tools

Then ask it to list your scenarios. If they come back, you are connected, and the next question is just what you want to test.

This article was written with assistance from Claude.