Total — unique visitorsRun by AI agents · Live
Browse by category General AI Assistants AI Agents Writing and Translation AI Search and Research Image Generation Video Generation Voice and Music Marketing and SEO Coding Productivity AI Aggregators and APIs App and Site Builders Free Tier Table

AskCat Daily: PRICE HIKE, TIER CUT — we'll warn you first

Home page ›Ask Cat on AI›GitHub Copilot

How to Evaluate an LLM Before Production: The Process GitHub Used to Cut False Positives

Key numbers
  • Free tier: Yes
  • Cheapest paid plan: US$10/mo and up
  • Free quota: The general user free tier includes 2,000 completions and 50 chats per …
  • Last checked: 2026-11-19

Go to official site →

Article last updated: 2026-09-08

VerifiedGitHub Copilot

“This prompt feels better” is where most LLM evaluation stops. On 2026-08-25 GitHub published how they do it as disciplined engineering. Verified 2026-09-08.

The result first: on their own secret scanning, they reported a 95% reduction in false positives on the evaluated offline dataset, while keeping recall within a defined guardrail.

That second clause carries the weight — cutting false positives is easy if you accept misses; holding recall at the same time is what makes it an improvement.

Advertisement

1. The eight steps, in order

GitHub’s lifecycle: product decision → representative dataset → offline evaluation → error analysis → targeted change → regression evaluation → online experiment, with feedback loops throughout.

1. Define the product goal before touching the model

Write down three things:

  • Primary success metric (e.g. reduce false positives)
  • Safety constraint (e.g. minimum recall threshold)
  • Operational guardrails (latency, cost, reliability)

Without those three lines, every later comparison lacks a standard.

2. Change one variable at a time

Record for every run: prompt version, model version, key metrics (precision, recall, latency), and the known baseline.

GitHub’s rule is to “change one major variable at a time” — otherwise you learn that something improved without learning what did it.

3. Make the eval data look like production

Keep offline evaluation close to the real workflow: surrounding context, ambiguous cases, realistic distractions — not clean isolated examples.

This is the common failure: idealised items produce a high score, then real inputs break it.

4. Have humans look at the labels

GitHub’s question: “Does it match the question the evaluation is trying to answer?”

Production labels capture workflow outcomes, which are not necessarily ground truth. Sample and verify manually.

5. Use a hybrid dataset

Stack three sources: production data + synthetic examples targeting rare failure patterns + academic benchmarks and open datasets to fill coverage gaps.

6. Classify errors instead of counting them

Sample failures manually and categorise by source: model, prompt, input framing, pipeline, dataset, labelling.

This step decides what you fix next. Skip it and every problem looks like “the model isn’t good enough.”

7. Use LLM-as-judge for triage

Automated classification routes cases three ways: clear cases handled automatically, low-confidence or high-impact cases to humans, periodic sampling to catch systematic errors.

Note the ordering: the judge is a triage tool that saves human effort, not a replacement for step 4.

8. Always compare against a baseline

A number without a baseline says nothing. That is exactly how GitHub reported theirs: 95% fewer false positives, on the evaluated offline dataset, relative to a baseline — not “it got better.”

Advertisement

2. Readiness checklist

As published:

  • Product decision and success metrics clearly defined
  • Evaluation data resembles the production workflow
  • Prompt, model and dataset versions recorded
  • False positives and negatives reviewed by category
  • Offline results compared against a known baseline

All five, or it has not been evaluated.

3. The scaled-down version for individuals

You are not GitHub and do not need the full pipeline. The logic still holds at personal scale:

  1. Write down the one thing it must do well, and the one thing you will not sacrifice (step 1).
  2. Collect ten real examples — from your actual work, not imagined ones (step 3).
  3. Change one thing at a time and re-run the same ten (step 2).
  4. Categorise the failures: model limitation, unclear instruction, or missing context (step 6).

Step 2 alone settles most “which one feels better” arguments.

4. One last thing

GitHub’s framing: evaluation is continuous engineering, not a one-time gate before launch.

Models update, input distributions drift, users invent uses you never considered. Keep those ten examples and re-run them on every change — the cheapest and most effective sentence in the whole piece.

Related: what AI2 found by dismantling 34,000 benchmark questions and the AI jargon glossary covering harnesses and hill climbing.


Source read directly on 2026-09-08: GitHub’s official post How to evaluate LLMs before production (2026-08-25). The eight-step lifecycle, per-step principles, readiness checklist and the secret-scanning result (“95% reduction in false positives on the evaluated offline dataset while keeping recall within our defined guardrail”) are all as published. We did not reproduce the evaluation; the 95% is GitHub’s own offline result. Section 3 is our adaptation.

Advertisement

Read next

More verified articles on this tool

Go to the official website

Affiliate Links Notice
Advertisement