Nitish Prasad

Writing

Long-form articles on building and evaluating AI products, with the evidence behind each claim.

1

Start here

New to the series? Start with part 1

2

Contents

AI evals on a real product

2 of 5 parts out · Part 3 on November 23, 2026

Five parts on evaluating SproutRoute, a family trip planner: what evals are, an AI audit of the evals, the process from rubric to rerun, human graders, and what changed in the product.

  1. Part 1

    What AI evals are, and why my trip planner needed them

    A plain-language introduction to AI evals, using a family trip planner: what an eval is, its five parts, and why code, AI models and people each grade different things.

    10 min read

  2. Part 2

    An eval score is only as good as where it came from. I had two AI models audit mine.

    Two AI models reviewed my eval suite blind, from the raw traces. They confirmed most of my counts and found the bugs my checks were never looking for. The setup, the results and the recipe.

    11 min read

  3. Part 3

    From rubric to rerun: how I run evals, followed through one car seat finding

    The eval process step by step, from writing the rubric to rerunning the same trips, and one safety finding followed end to end: the London car seat card I closed too early.

    Coming November 23, 2026

  4. Part 4

    Who grades the judgment calls? Three people, the same plans, no peeking

    Code can check rules, and an AI judge is only as good as its agreement with people. How I set up three independent human graders for the judgment lines in my trip planner's evals, and what they found.

    Coming November 30, 2026

  5. Part 5

    What the evals changed in the product, and what is still open

    Every change the evals led to in a family trip planner, how each one was made and tested, the before-and-after numbers on the same seven trips, and the failures that did not move.

    Coming December 3, 2026

3

Contact

Hiring for a product area?

I am looking for Principal / Staff IC ownership of a product area.

hello@nitishprasad.com