The lesson first: an eval score is only worth something if you trust where it came from. Mine all came from one AI assistant checking its own work. So I had two other models audit the suite blind, starting from the raw traces. They found 57 problems, and the biggest ones were in places I’d never written a check for.
This is Part 2 of a five-part series on evaluating SproutRoute, a family trip planner I build with an AI coding agent. Part 1 covers what an eval is. An eval replaces “looks fine” with things you write down before you look: a rubric of yes or no lines, a fixed set of requests, the trace of every run, a scorer for each line, and a pass mark set in advance. This part is about checking the eval itself.
Here’s where the suite stood before anyone checked it:
| Item | Count |
|---|---|
| Rubric lines | 37, in four groups |
| Rubric lines with a code check | 5 |
| Written test requests | 20 |
| Fixed test trips for before-and-after runs | 7 |
| Plans traced from the live backend | 22 |
The scorecard showed misses on repeated stops, outdoor stops after sunset, and packing lists that ignored the forecast. Those felt like honest findings. I felt pretty good about the suite. Then I did the thing I’d been putting off and asked two AI models to audit it.
Why I needed an outside reader
An AI assistant wrote the eval suite with me over one long session. Same assistant wrote the checks, read the traces, counted the failures, drafted the fixes. I reviewed its work, but I was reading its summaries and its counts. If it had a blind spot, I got it too. If I had one, it had no reason to push back.
The checks had another problem. They were written while reading the same traces they’d later score, so they mostly found what we’d already seen. I wanted a reader who started from the raw data and didn’t care whether my conclusions held up.
How the review was set up
I wrote one review prompt and gave the same file to two different models, Fable 5.2 and Astra 6. Two on purpose. They each have their own habits, and I wanted to see where they’d agree without talking to each other. If both found the same thing alone, I trusted it more. I also went through the traces and the rubric myself, so I wasn’t just taking the models’ word for it.
The prompt had two phases. In the first, the reviewer could read the rubric, the inputs, the raw traces, the check code and the product code. It couldn’t open anything with my conclusions in it: my results, my named failure codes, my list of claims, or the other reviewer’s file. It had to find the failures on its own, write its own version of two of my checks from the rubric text alone, and judge the method. Then save all of that to a file.
Only after saving could it open the second phase, a list of 28 numbered claims from my docs. For each one it had to rerun the command or recount from the raw files, then mark it Confirmed, Partly, Refuted or Could not verify.
The order matters more than I expected. If a reviewer reads my conclusions first, it mostly ends up checking my conclusions. Looking first is how it finds the stuff I never wrote down.
Five rules did most of the work:
- Read only. The reviewer could write one file, its own findings, and nothing else. That kept the review from turning into a rewrite, and it meant the code it described was the code I had.
- No network and no spending. Everything had to be recomputed from files on disk, so the reviewers had to say “could not verify” about anything they couldn’t recompute. Which was true.
- No secrets. Never open an environment file, and report any credential seen without copying it. Neither found one.
- Say what you didn’t check. “I found no problem” and “I did not look” are different results. Both reports came back with a section listing what was left unexamined, and I ended up reading that section about as closely as the findings.
- Don’t read the other reviewer’s output. I compared the two reports myself afterward.
What I asked them to try to break
The most useful instruction was to break one named function with adversarial inputs. The function decides whether a parent really stated a child’s age, which matters because the car seat card depends on it. I’d tested it on 18 prompts and it passed all of them. Each reviewer wrote a small script and beat it in minutes. “2 adults, kids, 5-day trip to Orlando” counted as a child aged five. So did “our 5 year anniversary” and “we last went 5 years ago.” “My 5 year old and her friend” passed as two children aged five, because nothing counted how many ages the parent had given.
Next most useful was asking each reviewer to build its own version of two checks from the rubric text and compare with mine. When the verdicts matched, I trusted both a bit more. When they didn’t, the difference usually pointed at a flaw. One of my checks flagged an indoor conservatory as an outdoor stop after dark and missed a wetlands park that really was one, because of how I’d defined “outdoor.”
The third was to look for checks that pass when they have nothing to look at. Both found one. My packing check passed any time the trace had no forecast to compare against.
What came back
Most of my counts held up, though not all of them.
The claim they both refuted was mine, about a car seat failure. I’d decided it was a quirk of the test runner. It wasn’t. Part 3 follows that one end to end.
So the counts were fine. The problems were in stuff I’d never counted at all. After merging the two reports and pairing up findings that described the same problem, I had 57 distinct findings.
Forty were found by both reviewers working blind. That convinced me more than either report would have alone. The 17 found by only one reviewer went both ways: each caught real defects the other missed.
Three bugs that no check was looking for
The biggest problems in the traces were ones I’d never written a check for.
The first was a week in Myrtle Beach with a three year old. The plan sent the family to Colonial Williamsburg in Virginia, then Charleston, Savannah, Richmond, St. Augustine in Florida, and Dollywood in Tennessee. Day 2 was Savannah at 9:00, back to Myrtle Beach at 12:30, and Richmond at 4:00. All in one day, with a toddler in a car seat.
I blamed the model, and I was wrong. The model had been handed a list of attractions and told to plan only from it. For a small city, that list got topped up from the 18 nearest cities in the database, however far away they were. The model did what it was told. My own check’s complaint about this trip was that a few stops weren’t on the shortlist. The shortlist was the problem.
The second was a day of sightseeing in Seattle that the app reported as 7,680 miles. That’s the distance from Seattle to the spot where the equator meets the prime meridian, out in the Atlantic, which is a long way to go for a day out. A stop with no coordinates went through Number(null), which is zero in JavaScript, so it landed at latitude 0, longitude 0. The app then warned the parent about heavy travel, added a 90 minute leg, and dropped stops that no longer fit. Before any fix, 15 of the 22 plans had a day like that. It was in the first trace of the set. Plain text. No check was looking at that field.
The third was timing. In 19 of 22 plans, a tip on a stop named a time that didn’t match when the stop was scheduled, like “best at sunset” on a stop booked for 9 in the morning. Tips and schedule came from different steps, and nothing compared them.
All three came from my side of the code: a database top-up with no distance limit, a type conversion, and two steps that never checked each other. A better model wouldn’t have fixed any of them.
The checks were wrong too
One of my five checks asked whether each stop came from the shortlist the model was handed. It worked by searching the prompt text for the stop’s name. Seemed fine to me. It got fooled. This is the part of evals I find hardest. Picking the scorer is most of the work, and this one was searching the wrong text.
Long trips are generated three days at a time, and each later prompt lists the stops already picked so the model won’t repeat them. So any stop the model made up in the first three days was “found in the prompt” by the time the check ran. One Asheville trip had no shortlist at all, which means all 22 of its stops came from the model’s memory. My check counted 16 of them as shortlist stops and flagged 6. An Austin plan had a walking tour of Times Square and Grand Central, copied from an example inside the prompt. My check passed it.
There were more. Two multi-stop trips were skipped by the checks without a word. A call refused by a production rate limit was getting filed as “skipped.” And I’d filed repeated stops as a model failure, when in two of the three trips with repeated attractions, my own server’s backfill had put the repeats there. It topped up thin days without checking earlier ones.
I rewrote the checks. The shortlist check reads the actual list rows. A failed call now counts as an error. Multi-stop plans are scored. And I added four checks for what the reviewers found, which took the suite from five code checks to nine. Here’s the same set of 22 plans scored by the old checks and the corrected ones:
| Rubric line | Original check | Corrected check |
|---|---|---|
| Place repeated | 5 fail of 20 scored | 5 fail of 22 |
| Outdoor stop after sunset | 9 fail of 20 | 9 fail of 22 |
| Stop from the shortlist | 6 fail of 20 | 8 fail of 22 |
| Packing fits the forecast | 2 fail of 10 | 2 fail of 10, skips when there is no forecast |
| Stop outside the destination area | no check | 1 fail of 21 |
| Route distance is plausible | no check | 15 fail of 22 |
| Timing tip agrees with the schedule | no check | 19 fail of 22 |
| No model-written car seat law | no check | 10 fail of 10 tips responses |
Where the reviewers disagreed
The disagreements were about as useful as the agreements, because each one marked a spot where my rubric or my process left room for two readings.
| Question | Fable 5.2 | Astra 6 | What happened |
|---|---|---|---|
| Was the London car seat failure a test-runner quirk? | No | No | Real server defect. Fixed, test added |
| A week planned as 8 days | Count it as a failure | Not a defect without a written decision on what “a week” means | I decided a week is 7 days. The parser was fixed |
| Outdoor stops after sunset, before fixes | 8 of 22 | 9 of 22 | The corrected check gives 9 of 22 |
| An 18 month old rounded up to 2 | Missed | Found | Fixed, test added |
| The Austin plan “dropped” the parent’s extra asks | Confirmed | Partly: the asks were kept, only the acknowledgement was missing | Astra 6’s reading was more exact |
Astra 6 asking for a written decision on “a week” was fair. If two careful readers apply a rubric line differently, the line still needs work. So I wrote it down.
What happened to the 57 findings
Finding a problem took a lot less work than fixing it. Proving the fix took more again.
I keep those four groups apart on purpose. “Fixed and rerun” means the change exists and the same case was run again. “Fixed and test-covered” means the change exists and a test pins it, but no live run has shown the case yet. Calling all 26 “fixed” would blur that. Part 5 goes through the changes and the before-and-after numbers.
What I’d change next time
I’d hold back some traces. The reviewers and I all worked from the same set the checks were written from, so none of us had unseen data to test the checks against. A handful of plans kept aside would have given a fairer read.
I’d add people earlier. Two models agreeing counts for something. A parent who has taken a toddler on a trip, reading the plans and marking them, would catch other things. That’s what Part 4 is about.
And I’d run the review before changing anything. It came after my first round of fixes, and it probably would have changed them.
The recipe
If you already have a rubric and traces, you’ve done the hard part. Here’s how to audit them, the short version:
- Write one review prompt and give it to two models from different providers. Neither sees the other’s output.
- Phase 1 is look first. The reviewer reads the rubric, inputs, raw traces, check code and product code, finds its own failures, rebuilds two of your checks from the rubric text alone, and saves its findings before going further.
- Hide your conclusions in phase 1: results, failure codes, your claims, the other reviewer’s file.
- Phase 2 is check. Give it a numbered list of your claims and ask for a recount from raw files, each marked Confirmed, Partly, Refuted or Could not verify.
- Read only, offline, no secrets, and a required section on what it did not check.
- Ask it to break one named function with adversarial inputs, and to find any check that passes when it has nothing to look at.
- Merge the two reports yourself, pair the findings, and record a disposition for each: fixed and rerun, fixed and test-covered, accepted, deferred or rejected.
It took an afternoon. Reading the results was not fun. Still the most useful day this project has had.
Next, Part 3 walks through the eval process from rubric to rerun, and follows the London car seat finding end to end.