LLM eval harnesses and AI agent evaluation
I am an AI engineer and consultant. I set up LLM eval harnesses and AI agent evaluations for teams that ship AI changes without proof. Each change gets a verdict before it merges.
I am based in Dallas-Fort Worth. I work with clients in Dallas-Fort Worth and Oklahoma City, remotely or on site.
I reply within two working days. NDA on request.
Ways to start
Each offer has a fixed scope and a clear result. You get a quote after the intro call.
-
LLM & Agent Eval Harness Setup
2-3 weeks- For
- Teams that ship LLM or AI agent changes without proof.
- You get
- An eval corpus, a scoring function, a CI gate with JSON verdicts, and a runbook.
-
AI Feature Audit
3 days- For
- Teams with an AI feature that feels unreliable.
- You get
- A written review of the feature, its data and its tests, and a fix plan in priority order.
What I have built
Client work uses codenames. I can share more detail under an NDA.
- One verify command. It runs the same checks on a laptop and in CI, and prints one JSON summary for agents.
- An identity eval: one frozen corpus, one scoring function, and a holdout split. Every matching strategy is scored on the same rows.
- A/B harnesses that score competing recognizers against frozen baselines.
- Model evals that grade the live product, per class, on full pages. Not only the training metric.
- Assistant evals for routing and for many phrasings of the same question.
- End-to-end journey probes: real client code, a real browser and known-answer fixtures.
See the proof: Harness Engineering: LLM eval harnesses and AI agent evaluation with pass / fail / error verdicts
Questions
- What is an LLM eval harness?
- An LLM eval harness runs your AI feature on a fixed set of examples and scores each result. It returns pass, fail or error, so a CI gate can stop a bad change. Error means the check could not run. It never counts as a pass.
- What does the setup include?
- You get an eval corpus from your own data, a scoring function, a CI gate that returns pass, fail or error as JSON, and a runbook. It takes two to three weeks.
- Can you evaluate AI agents, not only prompts?
- Yes. I write evals for agent routing and for many phrasings of the same question. Journey probes use the real product in a real browser, and each check has a known answer.
- Which data does the eval use?
- Your own data. I freeze the corpus, so every strategy is scored on the same rows. A holdout split stops overfitting. Model replies are cached, so runs repeat exactly.
- Where do you work?
- I am based in the Dallas-Fort Worth area. I work with clients in Dallas-Fort Worth and Oklahoma City. I also work remotely with teams anywhere, and I travel to work on site when it helps.
Tell me what you are building.
30 minutes. Tell me what you are building. I reply within two working days. Client details can stay private under a mutual NDA.