AI evals on a real product
Five parts on evaluating SproutRoute, a family trip planner: what evals are, an AI audit of the evals, the process from rubric to rerun, human graders, and what changed in the product.
1
Parts
Part 1
What AI evals are, and why my trip planner needed them
A plain-language introduction to AI evals, using a family trip planner: what an eval is, its five parts, and why code, AI models and people each grade different things.
10 min read
Part 2
An eval score is only as good as where it came from. I had two AI models audit mine.
Two AI models reviewed my eval suite blind, from the raw traces. They confirmed most of my counts and found the bugs my checks were never looking for. The setup, the results and the recipe.
11 min read
Part 3
From rubric to rerun: how I run evals, followed through one car seat finding
The eval process step by step, from writing the rubric to rerunning the same trips, and one safety finding followed end to end: the London car seat card I closed too early.
Coming November 23, 2026
Part 4
Who grades the judgment calls? Three people, the same plans, no peeking
Code can check rules, and an AI judge is only as good as its agreement with people. How I set up three independent human graders for the judgment lines in my trip planner's evals, and what they found.
Coming November 30, 2026
Part 5
What the evals changed in the product, and what is still open
Every change the evals led to in a family trip planner, how each one was made and tested, the before-and-after numbers on the same seven trips, and the failures that did not move.
Coming December 3, 2026
2
Contact
Hiring for a product area?
I am looking for Principal / Staff IC ownership of a product area.