If You Can't Measure It, You're Shipping Vibes — Evaluating LLM Features

Here's a scene I've watched play out more than once. A team has an AI feature in production — a support assistant, a summariser, a classifier. Someone notices a slightly awkward answer and tweaks the prompt to fix it. The tweak ships. It fixes that case. It also, unknown to anyone, makes fifteen other cases worse. Nobody finds out for three weeks, until a customer complains about something that used to work.

The problem isn't the prompt. The problem is that the team had no way to know what the change did. They improved one example they could see and regressed ten they couldn't. They were, in the truest sense, shipping vibes.

Evaluation is the discipline that ends this. It's the difference between "it seemed better when I tried it" and "accuracy went from 82% to 89% and no category regressed." One of those is engineering.

Why normal testing doesn't fit

Traditional software tests assert exact outputs: add(2, 2) must equal 4, forever. LLM outputs aren't like that. Ask the same model to summarise the same document twice and you'll get two different, both-fine summaries. There's no single correct string to assert against.

That non-determinism is exactly why people skip evaluation — the usual tools don't obviously apply — and exactly why they need it more. When the output varies and quality is a spectrum, human intuition is a terrible instrument. You cannot hold 50 examples in your head and notice that the latest change nudged the average down. You need to measure.

You wouldn't refactor code with no tests and just assume it still works. Editing a production prompt with no eval set is the same bet, made with less visibility.

Start with a golden set, and start small

An evaluation set — a "golden set" — is a collection of representative inputs paired with what a good output looks like. This is the single highest-leverage artefact in an AI project, and building it does not require tooling. It requires thirty real examples and an afternoon.

type EvalCase = {
  id: string;
  input: string; // a real question or document, ideally from actual usage
  // What "correct" means — the form depends on the task.
  expected?: string; // for tasks with a definite answer
  mustContain?: string[]; // facts a good answer has to include
  mustNotContain?: string[]; // things it must never say
};

const goldenSet: EvalCase[] = [
  {
    id: "refund-window",
    input: "How long do I have to return something?",
    mustContain: ["30 days"],
    mustNotContain: ["60 days", "no returns"],
  },
  {
    id: "unknown-on-purpose",
    input: "What's your policy on pet insurance?", // we don't sell this
    mustContain: ["don't", "not"], // it must decline, not invent
  },
];

Draw these from real usage the moment you have any — actual questions from actual users, especially the ones that went badly. Deliberately include the hard cases: the ambiguous question, the one the system should refuse, the edge case that bit you last time. An eval set of only easy questions certifies that your feature handles easy questions, which was never in doubt.

Grade the objective things with code

A surprising amount of "AI quality" is objectively checkable, and you should check those things with plain code — it's free, instant, and never flatters you. Did a classifier return one of the allowed labels? Did an extraction return valid JSON with the required fields? Did the answer avoid the phrases it must never say? Did it include the citation it was required to include?

function scoreCase(output: string, c: EvalCase): boolean {
  if (c.mustContain?.some((s) => !output.toLowerCase().includes(s.toLowerCase())))
    return false;
  if (c.mustNotContain?.some((s) => output.toLowerCase().includes(s.toLowerCase())))
    return false;
  return true;
}

// The whole harness is just: run every case, count what passed.
async function runEval(cases: EvalCase[], run: (input: string) => Promise<string>) {
  const results = await Promise.all(
    cases.map(async (c) => ({ id: c.id, pass: scoreCase(await run(c.input), c) })),
  );
  const passed = results.filter((r) => r.pass).length;
  console.log(`${passed}/${cases.length} passed`);
  return results;
}

This is not sophisticated, and that's the point. Half of evaluation is just running every case through the current prompt-and-model, counting what passed, and — crucially — printing which cases failed so you can look at them. Run it on every prompt change and every model swap, the way you'd run a test suite.

Use an LLM judge only for what's genuinely subjective

Some qualities resist code: is this summary faithful to the source? Is this reply appropriately empathetic? Is this answer actually helpful? Here you can use another language model as a judge — give it the input, the output, and a specific rubric, and have it score.

const judgePrompt = `
Score the assistant's answer from 1-5 on faithfulness to the source.
5 = every claim is supported by the source. 1 = contradicts or invents.
Reply as JSON: { "score": number, "reason": string }

Source: ${source}
Answer: ${answer}
`;

LLM judges are useful but not free of judgement themselves, so keep them honest:

• Give a narrow, specific rubric. "Rate the quality 1–10" produces noise. "Does every claim trace to the source, yes or no" produces signal.
• Judge one dimension at a time. Faithfulness and tone are different questions; a single blended score hides which one moved.
• Spot-check the judge against a human. Periodically have a person grade the same cases and confirm the judge agrees. A judge you never audit is just a second unmeasured system.

Wire it into the workflow, or it won't happen

An eval set that lives in someone's notebook and gets run twice a year is theatre. The value comes from making it routine: run the evals before any prompt or model change ships, compare against the last known score, and treat a regression the way you'd treat a failing test.

  change prompt / model ─▶ run evals ─▶ score vs baseline ─┬─ better/equal ─▶ ship
                                                           └─ regressed ────▶ stop

Once this loop exists, everything about the project changes. You can upgrade to a newer model and know whether it's actually better for your task, not just newer. You can let someone edit prompts without fear, because a regression gets caught before users do. You can tell a client "it's 91% accurate on our test set, here are the cases it still gets wrong" instead of "it seems to work well." That sentence is the whole difference between a demo and a product.

The honest payoff

Evaluation isn't glamorous. It's a spreadsheet of examples and a script that counts. But it's the thing that converts an AI feature from a hopeful gamble into something you can improve deliberately, defend confidently, and operate safely.

The teams whose AI features quietly get better over time all have one thing in common, and it isn't a better model or a cleverer prompt. It's that they can measure, so they can improve. Everyone else is still shipping vibes — and finding out what broke from their users.