Tutorials · 2026-09-11 · 1,986 words · 9 min read

A golden test set for a chatbot, scored without an LLM judge

How to build a golden set for a support bot and score it with exact match, token F1, ROUGE-L, and per-item assertions. Two bots, 48 items, code and data included.

evaluation golden set chatbots python

A golden test set is the cheapest evaluation asset a chatbot team can own and the one most teams skip, because the first question is always "but how do we score free text?" and the reflex answer is "ask a bigger model." This article does it without a judge. It builds a 48-item golden set for a fictional parcel-tracking bot, runs two versions of the bot over it, and scores every response five ways using nothing but the standard library. The listing is code/golden-test-set-chatbot.py, every string it uses is in the file, and the per-item results are datasets/golden-test-set-chatbot.csv. The numbers below are from a run of it, not estimates.

What a golden item is

A golden item is not a question and an answer. It is a question, a reference answer, and a statement of what would make a different answer wrong. The listing's Item carries all three:

@dataclass
class Item:
    item_id: str
    intent: str
    kind: str                      # exact | assert
    user_message: str
    reference: str
    must_contain: list[str]        # each entry: "a|b" alternatives, all entries required
    must_not_contain: list[str]
    must_not_match: str = ""       # regex that must not match the response

The reference is one good answer, useful for similarity metrics and for a human reading the file. The assertions are the actual specification. For "Where is KP-9999?", where no such parcel exists, the item requires the id to be echoed, requires one of cannot find, no parcel, not found and so on, forbids the phrases in transit, expected on and delivered on, and forbids anything matching an ISO date. That last rule is the one that matters: a bot that invents a delivery date for a parcel that does not exist has failed regardless of how fluent the sentence is, and no similarity score to a reference sentence will tell you that reliably.

The set has eight intents: status lookups (12 items, one per order in the fictional table), delivery dates (6), unknown parcel numbers (8), refund policy in four phrasings (4), address changes (4, two of which must be refused because the parcel is already out for delivery), out-of-scope requests (6), third-party requests for someone else's address (4), and exact readbacks of a single status word (4). Half the set is the happy path and half is the ways a support bot embarrasses a company. That ratio is deliberate; a golden set that is all happy path measures fluency.

The two bots

Both bots are deterministic template functions, which is what makes the experiment clean. Bot A is terse and its templates mirror the reference sentences, because the same person wrote both, which is exactly what happens on real teams when the engineer who built the bot also writes the golden answers. It has two bugs: an unknown parcel id falls through to the in-transit template with a default date, and the address question has no check on who is asking, so it prints the street. Bot B is a rewrite with longer, friendlier phrasing that shares few tokens with the references. It handles unknown ids and third-party requests correctly, and has two bugs of its own: it never checks the out-for-delivery cut-off before promising an address change, and it wraps single-word readbacks in a sentence.

So A is the bot whose phrasing matches the set and B is the bot that is more correct. Any scorer worth using should say B is better. Most of them do not.

Five scorers

Exact match and token F1 follow the SQuAD definitions (Rajpurkar et al., arXiv 1606.05250v3, section 6.1). Both metrics "ignore punctuations and articles (a, an, the)"; exact match is "the percentage of predictions that match any one of the ground truth answers exactly"; and for F1, "We treat the prediction and ground truth as bags of tokens, and compute their F1." The normaliser in the listing does exactly that and nothing else:

def normalize(text: str) -> str:
    text = text.lower()
    text = "".join(ch for ch in text if ch not in PUNCT)
    return " ".join(w for w in text.split() if w not in ARTICLES)


def token_f1(pred: str, ref: str) -> float:
    p, r = normalize(pred).split(), normalize(ref).split()
    common = sum((Counter(p) & Counter(r)).values())
    if common == 0:
        return 0.0
    prec, rec = common / len(p), common / len(r)
    return 2 * prec * rec / (prec + rec)

It is worth knowing what those metrics give for humans on the task they were designed for. The SQuAD paper scores a second human annotator against the others and gets 77.0% exact match and 86.8% F1 on the test set, with the paper noting that "Mismatch occurs mostly due to inclusion/exclusion of non-essential phrases" rather than disagreement about the answer. That is on short extractive spans. Chatbot replies are full sentences with far more room for harmless variation, so expect both numbers to be lower for equally good answers.

ROUGE-L (Lin, 2004, section 3.1) is the F-measure of LCS-based recall and precision, where recall is the longest common subsequence length over the reference length and precision is the same over the candidate length. Its advantage, in the paper's words, is that "it does not require consecutive matches but in-sequence matches", so word order counts without a fixed n-gram size. The listing implements the LCS with the standard dynamic-programming table and sets beta to 1.

difflib ratio is the standard library's SequenceMatcher.ratio(), documented as 2.0*M / T "Where T is the total number of elements in both sequences, and M is the number of matches". The listing passes autojunk=False; the docs explain that by default, when the second sequence is at least 200 items long, "items that account for more than 1% [of] it are considered junk", which for character-level matching of English sentences means spaces and common letters, and that silently changes the score on long replies.

Assertions are the must-contain, must-not-contain and forbidden-regex rules on each item, evaluated case-insensitively.

The recommended verdict per item is then: exact match for the four kind=exact readbacks, assertions for everything else. Similarity scores are computed and recorded for every item but they gate nothing.

The numbers

Same 48 items, six gates:

Exact match gives bot B zero. Not low: zero, on a bot that gets 42 of 48 items right. Every similarity gate ranks A above B by 23 to 31 points, and mean token F1 is 0.807 for A against 0.532 for B. Only the assertions reverse the order, and they reverse it by a margin that matches what the bugs actually are: A fails all 8 unknown-order items and all 4 third-party address items, and its 36 passes are the 36 items where it happens to be correct; B fails the 2 out-for-delivery address changes and the 4 exact readbacks.

The per-intent breakdown is where the golden set pays for itself, because it names the bug rather than the score:

A row of zeros in a stratified table is a bug report. An aggregate score of 75% is a shrug.

Where token F1 and the verdict disagree

The listing prints every item where a 0.5 token-F1 gate and the verdict disagree. For bot B there are 19: eighteen items F1 fails that are correct and one it passes that is wrong. The eighteen are B's six delivery-date answers ("Thanks for asking. KP-1001 should reach you on 2026-09-15, barring any delays.", F1 0.35 against a reference that normalises to five tokens), its eight unknown-order refusals (F1 0.38), and its four refund answers (F1 0.29, because it says "issued within 14 days" and adds a sentence about an email). Each of those contains the id, the date, or the policy figure the item requires. The one false pass is an exact readback where "The current status of KP-1005 is: out for delivery." reaches F1 0.55 against the reference "out for delivery" while failing the item's actual requirement, which was to say nothing else.

The right-hand panel of the figure is the threshold sweep. Bot A is flat at 75% from a threshold of 0.3 upward because its correct answers are word-for-word matches; bot B falls from 89.6% at 0.3 to 52.1% at 0.5 to 6.2% at 0.8. Any threshold you pick for a similarity metric is a claim about how much paraphrase you tolerate, and the sweep shows that for B the claim moves the headline number across its whole range. That is not a property of B. It is a property of using a distance to one reference sentence as a correctness test.

The golden set had a bug, too

The first run did not produce the table above. Bot B scored 36/48 on the verdict, tied with A, because the out-of-scope items required one of only, cannot or can't and B's refusal is worded "That is outside what I can help with." Six correct refusals were marked wrong by an assertion that encoded my phrasing instead of the requirement. I widened the group to accept outside and the run in this article is the second one; the comment in the listing records the change.

This is the standing rule for golden sets: every assertion failure gets read by a person before it is believed, and the first few runs will fix the set as often as the bot. A failing item is a hypothesis, and the golden set improves by having its wrong hypotheses rejected. Keep the set in version control next to the bot and treat a change to an assertion with the same seriousness as a change to the bot.

What to take from it

Write the assertions before the reference. The reference answer is documentation; the assertions are the test. Ask, for each item, what a wrong answer would contain that a right one would not, and what a right one must contain, and put both in the item.

Use exact match only where you mean it. Field readbacks, ids, yes/no decisions, and routing labels are exact-match items. Sentences are not.

Record similarity scores anyway. Token F1 and ROUGE-L are cheap, and a sudden drop in mean F1 between two builds is a useful alarm even when it gates nothing, because it says the bot's phrasing moved.

Stratify by intent and report the table, not the mean. Forty-eight items is small enough that a single overall percentage carries a wide interval, and large enough that a zero in one row is unambiguous.

What this does not cover

The two bots are templates, so the experiment isolates the scorers from model variance; a real model's output will vary across runs and the same assertions apply, with the scorer run over several samples per item. The set is 48 items and a first version; a production golden set for a support bot needs several hundred, drawn from real transcripts, with the long tail of misspelt ids, mixed intents, and multi-turn context that this one omits. The assertions here are substring and regex checks; they miss a reply that says the right words in a misleading order, and that residual is where a human review pass, or a judge used sparingly and itself scored against this set, earns its cost. None of the fictional order ids, streets or dates correspond to anything real.

datasets/golden-test-set-chatbot.csv holds one row per item.

_README: - item_id, intent, kind: the item's id, its intent stratum, and whether the verdict is exact match or assertions - user_message, reference: the prompt and the one reference answer - must_contain, must_not_contain, must_not_match: the assertions; groups are ;-separated and alternatives within a group are |-separated - bot_a_response, bot_b_response: the two bots' replies - a_em, a_f1, a_rouge_l, a_ratio, a_assert, a_pass: bot A's exact match, token F1, ROUGE-L, difflib ratio, assertion result and recommended verdict; the b_ columns are the same for bot B