Some of my app’s trip plans reported a single day of driving several thousand miles. The cause was a stop with no map coordinates, placed in the Atlantic Ocean. That bug was live in production while I was reading plans and deciding they looked fine.

If you have asked an AI model to plan something, you have probably seen a smaller version of this. The answer comes back fast, well organized and confident. Somewhere in it is a mistake you only find if you check. A restaurant that closed years ago. A drive that takes four hours, not forty minutes. A step that contradicts the one before it.

When you use a chatbot yourself, you catch those mistakes because you read the answer. When you build a product on top of a model, you are no longer the reader.

Thousands of answers go to people who trust them. You need a way to know how often they are wrong, in what ways, and whether your latest change made things better or worse. That way of knowing is called an eval.

This series is about building evals for a real product, end to end, including the parts that went wrong. The one that surprised me most: my own evals were wrong too. This first part explains what an eval is, assuming no machine learning background, and shows the one I built for my own app.

What an AI eval is

Think about how a teacher grades an essay exam. There is no single correct essay, so the teacher can’t compare each answer to an answer key. Instead the teacher writes a rubric: does the essay answer the question, does it use evidence, is the argument in order.

Every student gets the same questions, and every essay is graded against the same rubric. When two teachers grade the same essay, they should mostly agree. If they don’t, the rubric is unclear.

An AI eval is the same idea applied to software. You write down what a good answer must have, one checkable line at a time. You keep a fixed set of requests to send the system. You run them, grade every answer against the rubric, and count the results. Then you change something, run the same requests again, and see whether the counts moved.

Evals are different from the tests most software teams already write. A normal test checks one exact answer: given this input, the function returns 42. That works because normal code gives the same output every time.

An AI model doesn’t. Ask it for a three-day plan twice and you get two different plans, and both might be fine. You can’t test for “the right plan”, because there isn’t one. You test for properties every good plan must have: the days are in the right city, no place appears twice, a toddler isn’t scheduled for six stops in a row.

That is why evals matter to product managers as much as to engineers. They are how you decide:

  • whether a feature is good enough to ship
  • whether a prompt change helped
  • whether switching to a cheaper model is safe

Without them, every one of those decisions comes down to somebody reading a few outputs and saying they look fine.

The product I’m evaluating

I build SproutRoute, a trip planner for families, with an AI coding agent doing most of the typing. A parent types something like “a week in Chicago with a 9 year old” and gets back a day-by-day itinerary, a packing list, travel tips and a car seat card for each child.

A plan isn’t one model call. It is built in steps, and some steps are a model and some are ordinary code.

A plan is built in steps, and each step trusts the one before.A vertical flow of five steps that build a SproutRoute plan: read the request (model), pick candidate places (code), schedule the days (model), add tips (model) plus packing and the car seat card (code), and the plan the parent sees. A bracket notes that each step reads the last step’s output as fact.SPROUTROUTE EVALS · PART 1 · FIG 1.1A plan is built in steps, and each steptrusts the one before.MODEL means a language model does the step. CODE means rules.1MODELRead the requestdates, places, who is travelling, ages2CODEPick candidate placesa shortlist from the attraction database3MODELSchedule the daysthree days per call; later calls see earlier days4Add tips, packing and car seat cardMODELon the tipsCODEon the packing and the card5The plan the parent seeseach step reads thelast step’s output asfactSource: SproutRoute plan pipeline. All trips synthetic.nitishprasad.com
Figure 1.1 A plan is built in steps, and each step treats the output of the step before it as fact. Open full size

The steps are what make this hard to check by eye. If the first step misreads the request, every later step builds on the mistake, and the final plan still reads well.

Small error rates also multiply. If each step is right 95% of the time, a plan that goes through ten steps is entirely right only about 60% of the time.

Ten steps that are each right 95% of the time are all right 60% of the time.A line chart of the chance that every step is right as the number of steps grows, when each step is right 95% of the time. It falls from 95% at one step to 77% at five and 60% at ten.SPROUTROUTE EVALS · PART 1 · FIG 1.2Ten steps that are each right 95% of thetime are all right 60% of the time.chance every step is right0%25%50%75%100%12345678910number of steps60%95%Source: arithmetic, 0.95 to the power n. All trips synthetic.nitishprasad.com
Figure 1.2 Ten steps that are each right 95% of the time are all right only about 60% of the time. Open full size

Until this fall, my way of checking SproutRoute was to plan a trip, read the result and decide it looked fine. It did look fine. That told me very little, because I was judging the last step and assuming the rest.

Why reading the output is not enough

A normal feature either works or it doesn’t. A button saves the form or it throws an error, and a test can tell you which.

An AI feature fails differently. The plan arrives on time, reads well, and is wrong in a way you only see if you check. A stop is in another state. A tip says to arrive in the morning for a stop scheduled in the afternoon. The packing list for a trip with a freezing forecast has no cold-weather gear on it. None of those throws an error. All of them were in my product.

Reading a handful of plans also can’t tell you how often something happens. If I read five plans and see one bad tip, I don’t know whether that is one plan in five or one in fifty. I also can’t tell whether next week’s change made it better.

Counting needs two things: the same requests run the same way every time, and a written definition of what counts as bad.

The five parts of an eval

None of these parts needs a machine learning background.

An eval has five parts, and it runs in a loop.A loop: inputs (20 written requests and 7 fixed trips) go through the real system, which produces traces of every model call, lookup and output. Scorers apply a 37-line yes/no rubric using code, a model or a person, and results are compared with pass rates set before looking. A dashed arrow returns from results to the run: fix, then rerun the same inputs.SPROUTROUTE EVALS · PART 1 · FIG 1.3An eval has five parts, and it runs in aloop.Inputs20 written requests, 7 fixed tripsRun the real systemTracesevery model call, lookup and outputScorers apply the rubric37 yes/no linesCODEMODELPERSONResults against thresholdspass rates set before lookingFix, then rerun thesame inputsSource: SproutRoute evals evidence pack. All trips synthetic.nitishprasad.com
Figure 1.3 An eval has five parts, and it runs in a loop: run the inputs, score the traces, compare with the thresholds, fix, and rerun. Open full size

A rubric. A written definition of good, one checkable line at a time. “Every stop is in or near the destination” is a line. “The plan is high quality” is a wish. Each line is a yes or no question about one output, so a failure points at one specific thing. My rubric has 37 lines in four groups:

  • Outcome (17 lines): Did the parent get what they asked for? Example: every stop is in or near the destination.
  • Trajectory (6 lines): Did the system get there soundly? Example: the plan is built from the shortlist of places the model was handed.
  • Governance (10 lines): What must it never do? Example: car seat guidance comes from the curated rules, not from model text.
  • Experience (4 lines): Does it read clearly? Example: a parent can tell what to do from the plan’s notes.

A set of inputs. The requests you run every time. Mine has 20 written prompts, chosen by crossing who is travelling, what they ask for, and how messy the request is. They run from a clean San Diego trip to a request that asks the app to book a hotel and give legal advice in the same sentence. Seven of the trips are fixed and used for every before-and-after comparison.

Traces. The full record of one run: the request, every model call and lookup inside it, and the final plan. When a plan is bad, the trace shows which step went wrong. Without it, you only know that something did.

Scorers. Whatever applies one rubric line to one trace and returns pass or fail. Choosing them is most of the work, so they get their own section below.

Thresholds. The pass rate you require before a release, written down before you see any results. If you set the bar after looking, you will set it wherever you already are.

Code, a model, or a person

Every rubric line needs a scorer, and there are three kinds to choose from.

Three ways to score a rubric line.A three-column comparison of code, model and person scorers. Code is best at rules with one right answer and is free and instant; a model handles judgment calls at volume but moves with its wording and pass mark; a person makes the judgment calls a parent would make but takes about 10 minutes a plan. In SproutRoute: 9 code checks, people on 4 judgment lines, and AI models audited the suite.SPROUTROUTE EVALS · PART 1 · FIG 1.4Three ways to score a rubric line.CODEMODELPERSONBEST ATrules with one right answerjudgment calls at volumejudgment calls a parent wouldmakeEXAMPLE LINE“No place is scheduled twice”“Each day is doable for a 3 yearold”“Each day is doable for a 3 yearold”COST AND SPEEDfree, instant, same answerevery timecheap and fast; moves with itswording and pass markslow and paid; about 10minutes a planHOW YOU KNOW IT’S RIGHTrun it on cases where you knowthe answercompare it with people beforetrusting ithave several people grade thesame plans and measureagreementIn SproutRoute: 9 code checks · people on 4 judgment lines · AI models audited the suite (Part 2)Source: SproutRoute evals evidence pack. All trips synthetic.nitishprasad.com
Figure 1.4 Three ways to score a rubric line: code for rules with one right answer, a model for judgment at volume, and people for the judgment calls a parent would make. Open full size

Code comes first wherever it fits. A program can tell whether the same restaurant appears twice, whether a day’s driving adds up to thousands of miles, or whether a tip names a different time from the slot the stop was given. It costs nothing to run, gives the same answer every time, and can run on every change.

Many lines don’t fit. Could a family with a 3 year old really get through this day? Does the plan keep everything the parent asked for, including the child’s age? Would a parent know what to do from the notes? Those are judgment calls, and no rule I can write in code answers them.

The usual shortcut for judgment calls is a second AI model acting as a judge, often called “LLM as a judge”. You give it the rubric line and the plan, and it returns pass or fail. It is fast and cheap.

The trouble is that you can’t tell whether the judge is right until you have compared it with people. If the judge and the planner share a blind spot, the scores look healthy and the plan is still wrong.

So the judgment lines in my suite are graded by people. I wrote a scoring guide for four of them:

  • The plan kept the request.
  • It is complete, with no repeats.
  • Each day is doable for this family.
  • It is clear for a parent.

Three graders are working through the same 20 plans on their own, after a practice round of five. They don’t see each other’s answers, and they don’t use AI tools. Each line gets Pass, Fail or Unsure, and every Fail or Unsure comes with a note that names the day and the stop.

Where the graders agree, I can trust the grade. Where they split, either the plan is a borderline case or my scoring guide is unclear. Both are worth knowing.

How often graders agree is called inter-rater agreement. It is the first thing to check before trusting any grader, human or model. It is also how you would test a model judge later: run it on the same plans and see whether it agrees with the people.

Part 4 of this series has the results of the human grading.

What the first run found

I ran the inputs through the live app, exported the traces and read them. A few of the things that had been in production while I thought it looked fine:

  • Some plans reported days of several thousand miles. A stop with no map coordinates was being placed at latitude zero, longitude zero, in the Atlantic Ocean, and the distance was calculated from there.
  • Timing tips contradicted the schedule in 19 of 22 plans. The tip and the schedule were written by different steps, and nothing compared them.
  • Every set of travel tips I collected contained a car seat law written by the model, next to a car seat card built from curated rules. Two sources for the same safety question, and only one of them was controlled.
  • A beach week in Myrtle Beach was planned across five states, because a short list of local attractions had been topped up from cities hundreds of miles away.

I fixed these and ran the same seven fixed trips again.

Three failures went to zero. One got worse.A dumbbell chart of four failures on 7 fixed test trips, before and after the fixes. A day with an impossible distance fell from 4 trips to 0, a timing tip that contradicts the schedule from 6 to 0, and model-written car seat law in the tips from 7 to 0. An outdoor stop that ends after sunset got worse, from 5 trips to 6.SPROUTROUTE EVALS · PART 1 · FIG 1.5Three failures went to zero. One gotworse.Before: Nov 13 buildAfter: Nov 14 build01234567trips with the failure, out of 7Day with an impossible distance40Timing tip contradicts theschedule60Model-written car seat law in thetips70Outdoor stop ends after sunset56Source: 7 fixed test trips, Nov 13 build vs Nov 14 build. All tripssynthetic.nitishprasad.com
Figure 1.5 On the seven fixed trips, three failures went to zero after the fixes. Outdoor stops ending after sunset went from five trips to six. Open full size

Three lines went to zero. One got worse: outdoor stops that end after dark, which no change in this round touched. Part 5 covers the full before and after, including what is still open.

The part I did not expect

My evals were wrong too. One check passed plans it should have failed. Another skipped an entire category of trip and reported nothing.

I found this out by asking two AI models, Fable 5.2 and Astra 6, to audit the suite independently from the raw traces. Neither saw my conclusions or each other’s work.

That is the main thing I would tell another product manager. Building the evals is the first half. Checking the evals is the second half, and it is where most of what I learned came from. If you are building evals for your own product, plan time for both halves.

AI reviewers can recheck the code and the counts. Only people can tell you whether the judgment calls are right, which is why the human grading is part of this series too.

Where the method comes from

I have been working with evals for a while. At Amazon, I used them while building Amplify, a seller workflow intelligence tool, and saw firsthand how much an AI product depends on them. When I started SproutRoute, I built my own version of evals from day one, and I thought they were working well.

Then I took the Reforge AI Evals course in the fall of 2026, taught by Sandhya Hegde and Justin Bauer of Calibre Labs. It gave me a formal structure for work I had been doing piecemeal, and that structure was really, really good. Tracing in particular changed how I looked at every plan the app produced.

The four-group rubric, the input grid for choosing test prompts, and the practice of reading traces and naming each type of failure all come from their course and templates. If you are a product manager starting from zero, start there.

On top of that, I added my own customizations, judgment calls and lessons from before and after the course to make the method work for a family trip planner.

What this series adds is one product taken all the way through: the rubric as written, the checks as code, what they caught, what they missed, how people graded the same plans, and what changed in the app as a result. The mistakes in applying the method are mine.

This is still a work in progress. Evals are an evolving field, and putting them into a real product takes many rounds of changes. Expect your rubric, inputs and checks to keep changing as your product does.

The series

The series in five parts.A list of the five parts of the series, with part 1, “What AI evals are”, highlighted as the current part. 1: What AI evals are. 2: The AI audit. 3: From rubric to rerun. 4: The human graders. 5: What changed.SPROUTROUTE EVALS · PART 1 OF 5 · SERIES MAPThe series in five parts.1THIS PARTWhat AI evals arerubric, inputs, traces, scorers, thresholds2The AI audittwo models checked my evals, blind3From rubric to rerunthe process, through one car seat finding4The human gradersthree people grade the same plans5What changedbefore and after on the same tripsSeries: AI evals on a real product. All trips synthetic.nitishprasad.com
Figure 1.6 The series in five parts: what evals are, the AI audit, from rubric to rerun, the human graders, and what changed. Open full size
  1. What AI evals are. This part.
  2. The AI audit. Two AI models checked my eval suite against the raw traces, blind to each other, and found what my checks missed. It includes how I set up the review, so you can run the same audit on your own suite.
  3. From rubric to rerun. The process step by step, followed through one finding end to end: a London trip given the wrong car seat guidance, and why my own check had dismissed it.
  4. The human graders. Three people graded the same plans on their own. How often they agreed with each other and with the automated checks.
  5. What changed in the product. Before and after numbers on the same test trips, and the list of what is still open.

Every trip in this series is synthetic. No real family’s data is used anywhere, and nothing here is safety advice about car seats.