Josh SolisJS Book an intro callBook a call

2026-10-27

What ten months of building AI systems taught me

Seven lessons from building computer vision, catalog matching, AI agents and eval harnesses: data beats models, uncertainty is a feature, and a check that cannot run must say so.

I did not start from zero. I studied cognitive science at Rice and took computer science classes alongside it. Later I did a nine-month AI and ML bootcamp, then spent about five years in business: solutions engineer, account manager, and owner of my own company.

Over the last ten months I put that base to work on real problems. I built computer vision for construction takeoff, a system that matches collectibles to an exact catalog entry, platforms where AI agents do paid work, and the test harnesses that check all of it.

Each project taught me something I did not know going in. Here are the seven lessons that changed how I work.

1. The old questions are still the right questions

Cognitive science asks how a mind represents what it knows. Early AI asked the same thing. Should knowledge be symbols and rules, or patterns learned from examples? Each choice makes some problems easy and others impossible.

That question came back on every project. A label scheme for a detector is a representation. So is a catalog schema, or the JSON an agent reads to decide what to do next. When a system behaved badly, the cause was often how it represented the problem, not the code around it.

So I now start with one question: what does this system need to know, and in what form? The answer shapes everything after it.

2. The data is the product

On Blueprint Vision I trained detection models to find HVAC symbols on construction drawings. I tried a different architecture (YOLO, then RF-DETR) and tuned the training. Model choice mattered less than I expected.

The limit was the labels. They were not consistent enough, and a better model trained on the same labels learned the same mistakes.

So I changed the product, not the model. Every time an estimator fixes a box, that fix becomes clean training data. The app collects better data as people use it. My best detector reached mAP50 0.755, and the next gains will come from more clean labels, not a new architecture.

The lesson: before you tune the model, look hard at the data. Then design the product to make the data better.

3. Say when you are not sure

Collectible ID finds the exact printing of a collectible in a catalog of 151,260 items. Many printings look almost the same. The set, number, finish and language tell them apart.

A wrong match is worse than no match. If the app prices the wrong printing, a dealer loses money and stops trusting the app. So the system shows its confidence. When one candidate is left, it attaches a price. When two are left, it asks the dealer.

People think an AI system that says “I am not sure” is weaker. In practice, it is the one people can trust.

4. Speed is easy. Proof is hard.

AI coding agents write code fast. A fast change that nobody checks is a fast regression.

So I stopped asking “how fast can we build?” and started asking “how fast can we prove a change is safe?” Every change now passes a gate before it merges: build, unit tests, parity tests, browser smoke tests and API tests. One verify command runs the same checks on a laptop and in CI.

Review and tests set the real speed of a team, not typing. Once the gate was solid, I could let agents work in parallel and trust the result.

5. A check that cannot run must say so

An audit of my own setup found two silent failures. In both, the checks existed. Nothing forced them to run. One test that should have caught a problem was opt-in, and it was not turned on where the change was pushed.

A missing check looks just like a passing one. That is the dangerous part.

Now every check returns one of three answers: pass, fail or error. Error means the check could not run. It stops the line and alerts a person, and it never counts as a pass. I wrote more about this in pass, fail, error.

6. Give agents small, revocable authority

On Agent Systems I built platforms where AI agents take on tasks and get paid when the work checks out. The hard question was not how to make an agent capable. It was how much to trust it.

My answer is to give agents as little authority as the task needs, and make it easy to take back. Spend limits. Allow-lists. Session keys. Formal proofs on the policy logic. And code, not another model’s opinion, decides whether the work passed.

An agent with full keys and no limits is a risk, however good it is. An agent with small, checked authority is a tool you can rely on.

7. Scope like a consultant, build like an engineer

My years in sales and client work taught me something code did not: most projects fail at the start, not the end. The scope was vague, the goal moved, or nobody agreed on what “done” meant.

So I scope every engagement the same way. It starts with a short written scope: what I build, what you get, and how we will know it works. Then I build in small steps and show a working demo each week. At the end, the client gets the code, the tests and a runbook, so their team can run it without me.

The tests are part of the hand-over, not an extra. They are how the client knows the system still works after I leave.

What ties it together

Looking back, each lesson is the same idea in a different place. Do not trust what you cannot check. Check the data before the model. Check the match before the price. Check every change before it merges. Check that the check ran.

That is the work I do now: AI systems that ship, and the harnesses that prove they work.