The lesson first: an eval score is only worth something if you trust where it came from. Mine all came from one AI assistant checking its own work. So I had two other models audit the suite blind, starting from the raw traces. They found 57 problems, and the biggest ones were in places I’d never written a check for.

This is Part 2 of a five-part series on evaluating SproutRoute, a family trip planner I build with an AI coding agent. Part 1 covers what an eval is. An eval replaces “looks fine” with things you write down before you look: a rubric of yes or no lines, a fixed set of requests, the trace of every run, a scorer for each line, and a pass mark set in advance. This part is about checking the eval itself.

Here’s where the suite stood before anyone checked it:

ItemCount
Rubric lines37, in four groups
Rubric lines with a code check5
Written test requests20
Fixed test trips for before-and-after runs7
Plans traced from the live backend22

The scorecard showed misses on repeated stops, outdoor stops after sunset, and packing lists that ignored the forecast. Those felt like honest findings. I felt pretty good about the suite. Then I did the thing I’d been putting off and asked two AI models to audit it.

Why I needed an outside reader

An AI assistant wrote the eval suite with me over one long session. Same assistant wrote the checks, read the traces, counted the failures, drafted the fixes. I reviewed its work, but I was reading its summaries and its counts. If it had a blind spot, I got it too. If I had one, it had no reason to push back.

The checks had another problem. They were written while reading the same traces they’d later score, so they mostly found what we’d already seen. I wanted a reader who started from the raw data and didn’t care whether my conclusions held up.

How the review was set up

I wrote one review prompt and gave the same file to two different models, Fable 5.2 and Astra 6. Two on purpose. They each have their own habits, and I wanted to see where they’d agree without talking to each other. If both found the same thing alone, I trusted it more. I also went through the traces and the rubric myself, so I wasn’t just taking the models’ word for it.

Look first, read my conclusions second.Two reviewers, Fable 5.2 and Astra 6, work in separate lanes. In phase 1 each reads the rubric, inputs, raw traces and code and saves its own findings without seeing my results, failure codes, claims or the other reviewer’s file. In phase 2 each recounts 28 numbered claims from raw files. I compare the two reports, which gives 57 distinct findings.SPROUTROUTE EVALS · PART 2 · FIG 2.1Look first, read my conclusions second.Hidden in phase 1my results, my failure codes, my claims, the other reviewer’s fileREVIEWER A · FABLE 5.21LookReads the rubric, inputs, rawtraces, check code and productcode, then writes and saves itsown findings.2CheckOpens 28 numbered claims andrecounts each from raw files.CONFIRMEDPARTLYREFUTEDCOULD NOT VERIFYREVIEWER B · ASTRA 61LookReads the rubric, inputs, rawtraces, check code and productcode, then writes and saves itsown findings.2CheckOpens 28 numbered claims andrecounts each from raw files.CONFIRMEDPARTLYREFUTEDCOULD NOT VERIFYI compare the two reports57 distinct findingsRULESread only · no network · no secrets · say what you did not checkSource: SproutRoute evals evidence pack. All trips synthetic.nitishprasad.com
Figure 2.1 The blind review. Each reviewer formed its own view from the raw traces and saved it before it could see any of my conclusions, then rechecked 28 of my claims. Open full size

The prompt had two phases. In the first, the reviewer could read the rubric, the inputs, the raw traces, the check code and the product code. It couldn’t open anything with my conclusions in it: my results, my named failure codes, my list of claims, or the other reviewer’s file. It had to find the failures on its own, write its own version of two of my checks from the rubric text alone, and judge the method. Then save all of that to a file.

Only after saving could it open the second phase, a list of 28 numbered claims from my docs. For each one it had to rerun the command or recount from the raw files, then mark it Confirmed, Partly, Refuted or Could not verify.

The order matters more than I expected. If a reviewer reads my conclusions first, it mostly ends up checking my conclusions. Looking first is how it finds the stuff I never wrote down.

Five rules did most of the work:

  1. Read only. The reviewer could write one file, its own findings, and nothing else. That kept the review from turning into a rewrite, and it meant the code it described was the code I had.
  2. No network and no spending. Everything had to be recomputed from files on disk, so the reviewers had to say “could not verify” about anything they couldn’t recompute. Which was true.
  3. No secrets. Never open an environment file, and report any credential seen without copying it. Neither found one.
  4. Say what you didn’t check. “I found no problem” and “I did not look” are different results. Both reports came back with a section listing what was left unexamined, and I ended up reading that section about as closely as the findings.
  5. Don’t read the other reviewer’s output. I compared the two reports myself afterward.

What I asked them to try to break

The most useful instruction was to break one named function with adversarial inputs. The function decides whether a parent really stated a child’s age, which matters because the car seat card depends on it. I’d tested it on 18 prompts and it passed all of them. Each reviewer wrote a small script and beat it in minutes. “2 adults, kids, 5-day trip to Orlando” counted as a child aged five. So did “our 5 year anniversary” and “we last went 5 years ago.” “My 5 year old and her friend” passed as two children aged five, because nothing counted how many ages the parent had given.

Next most useful was asking each reviewer to build its own version of two checks from the rubric text and compare with mine. When the verdicts matched, I trusted both a bit more. When they didn’t, the difference usually pointed at a flaw. One of my checks flagged an indoor conservatory as an outdoor stop after dark and missed a wetlands park that really was one, because of how I’d defined “outdoor.”

The third was to look for checks that pass when they have nothing to look at. Both found one. My packing check passed any time the trace had no forecast to compare against.

What came back

Most of my counts held up, though not all of them.

Most of my counts held up.Two stacked bars of 26 checkable claims each. Fable 5.2 confirmed 12, partly confirmed 10, refuted 1 and could not verify 3. Astra 6 confirmed 8, partly confirmed 10, refuted 3 and could not verify 5. Both refuted the same claim: that the London car seat failure was a test-runner artifact.SPROUTROUTE EVALS · PART 2 · FIG 2.2Most of my counts held up.Fable 5.226 checkable claims12confirmed10partly1refuted3could not verifyAstra 626 checkable claims8confirmed10partly3refuted5could not verifyBoth refuted the same claim: that the London car seat failure was a test-runner artifact.Source: SproutRoute evals evidence pack. All trips synthetic.nitishprasad.com
Figure 2.2 Of 26 checkable claims, Fable 5.2 confirmed 12, partly confirmed 10, refuted 1 and could not verify 3. Astra 6 confirmed 8, partly confirmed 10, refuted 3 and could not verify 5. Open full size

The claim they both refuted was mine, about a car seat failure. I’d decided it was a quirk of the test runner. It wasn’t. Part 3 follows that one end to end.

So the counts were fine. The problems were in stuff I’d never counted at all. After merging the two reports and pairing up findings that described the same problem, I had 57 distinct findings.

57 distinct findings. 40 were found by both reviewers.A stacked bar of 57 distinct findings: 40 found by both reviewers, 14 found only by Fable 5.2 and 3 found only by Astra 6. Pairing findings across the two reports is a judgment call.SPROUTROUTE EVALS · PART 2 · FIG 2.357 distinct findings. 40 were found byboth reviewers.40found by both14Fable 5.2 only3Astra 6 onlyPairing findings across the two reports is a judgment call.Source: SproutRoute evals evidence pack. All trips synthetic.nitishprasad.com
Figure 2.3 The two reports merged into 57 distinct findings. 40 were found by both reviewers, 14 only by Fable 5.2 and 3 only by Astra 6. Open full size

Forty were found by both reviewers working blind. That convinced me more than either report would have alone. The 17 found by only one reviewer went both ways: each caught real defects the other missed.

Three bugs that no check was looking for

The biggest problems in the traces were ones I’d never written a check for.

The first was a week in Myrtle Beach with a three year old. The plan sent the family to Colonial Williamsburg in Virginia, then Charleston, Savannah, Richmond, St. Augustine in Florida, and Dollywood in Tennessee. Day 2 was Savannah at 9:00, back to Myrtle Beach at 12:30, and Richmond at 4:00. All in one day, with a toddler in a car seat.

The model followed the list it was given. The list was wrong.Myrtle Beach has few attractions in the database, so its shortlist was topped up from the 18 nearest cities with no distance limit, adding places such as Colonial Williamsburg, Charleston, Savannah, Richmond, St. Augustine and Dollywood. The model planned only from that list, so Day 2 runs Savannah at 9:00, Myrtle Beach at 12:30 and Richmond at 4:00. The fix: top-up cities must now be within 40 miles.SPROUTROUTE EVALS · PART 2 · FIG 2.4The model followed the list it was given.The list was wrong.1Myrtle Beach has few attractionsin the database,so the shortlist was topped up from the 18nearest cities, with no distance limit.ShortlistColonial Williamsburg, VACharleston, SCSavannah, GARichmond, VASt. Augustine, FLDollywood, TNno distance limit2The model plans only from theshortlist, as instructed.3Day 2 of the plan9:00Savannah, GA12:30Myrtle Beach, SC4:00Richmond, VAFIXTop-up cities must now be within 40 miles.Source: SproutRoute evals evidence pack. All trips synthetic.nitishprasad.com
Figure 2.4 The model followed the list it was given. Myrtle Beach's shortlist had been topped up from the 18 nearest cities with no distance limit, so the plan spanned five states. Open full size

I blamed the model, and I was wrong. The model had been handed a list of attractions and told to plan only from it. For a small city, that list got topped up from the 18 nearest cities in the database, however far away they were. The model did what it was told. My own check’s complaint about this trip was that a few stops weren’t on the shortlist. The shortlist was the problem.

The second was a day of sightseeing in Seattle that the app reported as 7,680 miles. That’s the distance from Seattle to the spot where the equator meets the prime meridian, out in the Atlantic, which is a long way to go for a day out. A stop with no coordinates went through Number(null), which is zero in JavaScript, so it landed at latitude 0, longitude 0. The app then warned the parent about heavy travel, added a 90 minute leg, and dropped stops that no longer fit. Before any fix, 15 of the 22 plans had a day like that. It was in the first trace of the set. Plain text. No check was looking at that field.

The third was timing. In 19 of 22 plans, a tip on a stop named a time that didn’t match when the stop was scheduled, like “best at sunset” on a stop booked for 9 in the morning. Tips and schedule came from different steps, and nothing compared them.

All three came from my side of the code: a database top-up with no distance limit, a type conversion, and two steps that never checked each other. A better model wouldn’t have fixed any of them.

The checks were wrong too

One of my five checks asked whether each stop came from the shortlist the model was handed. It worked by searching the prompt text for the stop’s name. Seemed fine to me. It got fooled. This is the part of evals I find hardest. Picking the scorer is most of the work, and this one was searching the wrong text.

How my check passed 16 stops the model made up.On the Asheville test trip, days 1 to 3 had no shortlist, so the model planned from memory. The prompt for day 4 onward listed those earlier stops, and the check only asked whether a stop’s name appeared anywhere in the prompt, so it passed 16 of 22 stops as from the shortlist and flagged 6. The check now reads only the shortlist rows, and all 22 stops count as from memory.SPROUTROUTE EVALS · PART 2 · FIG 2.5How my check passed 16 stops themodel made up.Asheville test trip, 22 stops.1Days 1 to 3No shortlist was given, so the model plans from memory.2The days 4+ prompt lists the earlier stops:“Activities already suggested: French BroadRiver Dog Park, …”3The check asks:“Does the stop’s name appear anywhere in the prompt?”4PASSResult16 of 22 counted as from the shortlist, 6 flagged.step 2 put thestops in thepromptFIXThe check reads only the shortlist rows. Now: no shortlist given, so all 22 stops are frommemory.Source: SproutRoute evals evidence pack. All trips synthetic.nitishprasad.com
Figure 2.5 How the shortlist check passed 16 stops the model made up. Later prompts quote earlier stops, and the check searched all of the prompt text. Open full size

Long trips are generated three days at a time, and each later prompt lists the stops already picked so the model won’t repeat them. So any stop the model made up in the first three days was “found in the prompt” by the time the check ran. One Asheville trip had no shortlist at all, which means all 22 of its stops came from the model’s memory. My check counted 16 of them as shortlist stops and flagged 6. An Austin plan had a walking tour of Times Square and Grand Central, copied from an example inside the prompt. My check passed it.

There were more. Two multi-stop trips were skipped by the checks without a word. A call refused by a production rate limit was getting filed as “skipped.” And I’d filed repeated stops as a model failure, when in two of the three trips with repeated attractions, my own server’s backfill had put the repeats there. It topped up thin days without checking earlier ones.

I rewrote the checks. The shortlist check reads the actual list rows. A failed call now counts as an error. Multi-stop plans are scored. And I added four checks for what the reviewers found, which took the suite from five code checks to nine. Here’s the same set of 22 plans scored by the old checks and the corrected ones:

Rubric lineOriginal checkCorrected check
Place repeated5 fail of 20 scored5 fail of 22
Outdoor stop after sunset9 fail of 209 fail of 22
Stop from the shortlist6 fail of 208 fail of 22
Packing fits the forecast2 fail of 102 fail of 10, skips when there is no forecast
Stop outside the destination areano check1 fail of 21
Route distance is plausibleno check15 fail of 22
Timing tip agrees with the scheduleno check19 fail of 22
No model-written car seat lawno check10 fail of 10 tips responses

Where the reviewers disagreed

The disagreements were about as useful as the agreements, because each one marked a spot where my rubric or my process left room for two readings.

QuestionFable 5.2Astra 6What happened
Was the London car seat failure a test-runner quirk?NoNoReal server defect. Fixed, test added
A week planned as 8 daysCount it as a failureNot a defect without a written decision on what “a week” meansI decided a week is 7 days. The parser was fixed
Outdoor stops after sunset, before fixes8 of 229 of 22The corrected check gives 9 of 22
An 18 month old rounded up to 2MissedFoundFixed, test added
The Austin plan “dropped” the parent’s extra asksConfirmedPartly: the asks were kept, only the acknowledgement was missingAstra 6’s reading was more exact

Astra 6 asking for a written decision on “a week” was fair. If two careful readers apply a rubric line differently, the line still needs work. So I wrote it down.

What happened to the 57 findings

Finding a problem took a lot less work than fixing it. Proving the fix took more again.

What happened to the 57 findings.A stacked bar of the 57 audit findings by outcome: 17 fixed and rerun, 9 fixed and test-covered, 9 accepted with no code change, and 22 deferred and still open. A fix with a rerun and a fix with a test are different amounts of evidence.SPROUTROUTE EVALS · PART 2 · FIG 2.6What happened to the 57 findings.17fixed and rerun9fixed andtest-covered9accepted with nocode change22deferred and still openNOTEA fix with a rerun and a fix with a test are different amounts of evidence.Source: SproutRoute evals evidence pack. All trips synthetic.nitishprasad.com
Figure 2.6 Of the 57 findings, 17 were fixed and rerun, 9 were fixed and covered by tests, 9 were accepted with no code change, and 22 are deferred and still open. Open full size

I keep those four groups apart on purpose. “Fixed and rerun” means the change exists and the same case was run again. “Fixed and test-covered” means the change exists and a test pins it, but no live run has shown the case yet. Calling all 26 “fixed” would blur that. Part 5 goes through the changes and the before-and-after numbers.

What I’d change next time

I’d hold back some traces. The reviewers and I all worked from the same set the checks were written from, so none of us had unseen data to test the checks against. A handful of plans kept aside would have given a fairer read.

I’d add people earlier. Two models agreeing counts for something. A parent who has taken a toddler on a trip, reading the plans and marking them, would catch other things. That’s what Part 4 is about.

And I’d run the review before changing anything. It came after my first round of fixes, and it probably would have changed them.

The recipe

If you already have a rubric and traces, you’ve done the hard part. Here’s how to audit them, the short version:

  1. Write one review prompt and give it to two models from different providers. Neither sees the other’s output.
  2. Phase 1 is look first. The reviewer reads the rubric, inputs, raw traces, check code and product code, finds its own failures, rebuilds two of your checks from the rubric text alone, and saves its findings before going further.
  3. Hide your conclusions in phase 1: results, failure codes, your claims, the other reviewer’s file.
  4. Phase 2 is check. Give it a numbered list of your claims and ask for a recount from raw files, each marked Confirmed, Partly, Refuted or Could not verify.
  5. Read only, offline, no secrets, and a required section on what it did not check.
  6. Ask it to break one named function with adversarial inputs, and to find any check that passes when it has nothing to look at.
  7. Merge the two reports yourself, pair the findings, and record a disposition for each: fixed and rerun, fixed and test-covered, accepted, deferred or rejected.

It took an afternoon. Reading the results was not fun. Still the most useful day this project has had.

Next, Part 3 walks through the eval process from rubric to rerun, and follows the London car seat finding end to end.