Josh SolisJS Book an intro callBook a call

LLM eval harnesses and AI agent evaluation

I am an AI engineer and consultant. I set up LLM eval harnesses and AI agent evaluations for teams that ship AI changes without proof. Each change gets a verdict before it merges.

I am based in Dallas-Fort Worth. I work with clients in Dallas-Fort Worth and Oklahoma City, remotely or on site.

I reply within two working days. NDA on request.

Dallas-Fort Worth · Oklahoma City · Remote and on site · 30-minute intro call

Ways to start

Each offer has a fixed scope and a clear result. You get a quote after the intro call.

  • LLM & Agent Eval Harness Setup

    2-3 weeks
    For
    Teams that ship LLM or AI agent changes without proof.
    You get
    An eval corpus, a scoring function, a CI gate with JSON verdicts, and a runbook.
  • AI Feature Audit

    3 days
    For
    Teams with an AI feature that feels unreliable.
    You get
    A written review of the feature, its data and its tests, and a fix plan in priority order.

What I have built

Client work uses codenames. I can share more detail under an NDA.

  • One verify command. It runs the same checks on a laptop and in CI, and prints one JSON summary for agents.
  • An identity eval: one frozen corpus, one scoring function, and a holdout split. Every matching strategy is scored on the same rows.
  • A/B harnesses that score competing recognizers against frozen baselines.
  • Model evals that grade the live product, per class, on full pages. Not only the training metric.
  • Assistant evals for routing and for many phrasings of the same question.
  • End-to-end journey probes: real client code, a real browser and known-answer fixtures.

Questions

What is an LLM eval harness?
An LLM eval harness runs your AI feature on a fixed set of examples and scores each result. It returns pass, fail or error, so a CI gate can stop a bad change. Error means the check could not run. It never counts as a pass.
What does the setup include?
You get an eval corpus from your own data, a scoring function, a CI gate that returns pass, fail or error as JSON, and a runbook. It takes two to three weeks.
Can you evaluate AI agents, not only prompts?
Yes. I write evals for agent routing and for many phrasings of the same question. Journey probes use the real product in a real browser, and each check has a known answer.
Which data does the eval use?
Your own data. I freeze the corpus, so every strategy is scored on the same rows. A holdout split stops overfitting. Model replies are cached, so runs repeat exactly.
Where do you work?
I am based in the Dallas-Fort Worth area. I work with clients in Dallas-Fort Worth and Oklahoma City. I also work remotely with teams anywhere, and I travel to work on site when it helps.

Tell me what you are building.

30 minutes. Tell me what you are building. I reply within two working days. Client details can stay private under a mutual NDA.