Is your test set in the training data? Four overlap rules, measured

A contamination check answers one narrow question: does text from the test set appear in the training data? Four model reports have published a rule for answering it, and they do not agree on what "appear" means. This article puts the GPT-3, PaLM, Llama 2 and GPT-4 rules into one standard-library script, code/test-set-contamination-ngram-overlap.py, and measures what each catches on data where the answer is known: the 1,319 GSM8K test questions checked against GSM8K's own 7,473 training questions, a different set of problems, and against GSM-Plus, which holds eight edited copies of every test question. The GPT-3 rule flagged 3 test questions against the training split. It caught 69.1% to 95.6% of the questions whose copy had its numbers changed, a sentence added, a fact removed or the question reversed, and 0.8% of the questions whose copy was reworded. The per-item counts are in datasets/test-set-contamination-ngram-overlap.csv, and every number below comes from a run of the listing.

Four rules from four reports

GPT-3 (Brown et al., arXiv 2005.14165v4, Appendix C) defines a gram as "a lowercase, whitespace delimited word with no punctuation" and calls an example dirty if it has "any N-gram overlap with any training document". N is set per benchmark as "the 5th percentile example length in words", with "a minimum value of 8 on non-synthetic tasks" because of "spurious collisions at lower values of N", and a maximum of 13 for performance reasons. The same appendix describes the filter run before training: it removed each colliding 13-gram "as well as a 200 character window around it", skipped 13-grams that "matched more than 10 training documents", and, because of a bug, "failed on long documents such as books".

PaLM (Chowdhery et al., arXiv 2204.02311v5, Section 8) first sorted its 29 English benchmarks by how they were built, and did not count a dataset as contaminated when only its web-sourced context was in the training data. For the 10 sets it did count, it split the examples by "whether at least 70% of the 8-grams in question, prompt, or target were seen at least once" in its training data.

Llama 2 (Touvron et al., arXiv 2307.09288v2, Appendix A.6) works on tokens: "We consider a token to be contaminated if it appears in any token n-gram longer than 10 tokens in both the evaluation sample and the training set". A sample's contamination is the share of its tokens so marked. "Clean" is under 20% and "dirty" is 80% or more, a matched span may differ in up to four positions (the "skipgram budget"), and the search over the whole corpus took "around seven hours" on "an estimated 1,500 cores".

GPT-4 (OpenAI, arXiv 2303.08774v6, Appendix C) removes "all spaces and symbols, keeping only characters (including numbers)", takes "three substrings of 50 characters" at random from each evaluation example, and "A match is identified if any of the three sampled evaluation substrings is a substring of the processed training example."

Two of these rules ask whether any piece of the item leaked, and two ask whether most of it did. GPT-3 and GPT-4 flag on a single shared span. PaLM and Llama 2 were built to carve out a high-precision clean or dirty subset for a score comparison, so they need most of the item to be present. On the same item they can disagree, and the sections below measure how often.

One pass computes all four

For each word position i of a test item, the listing finds R[i], the longest run of words starting at i that also occurs in one training document. Candidate runs come from shared 5-grams and are extended word by word, so R[i] is exact for runs of five words or more and 0 below that:

    def runs(self, toks, exclude=None):
        """R[i] = the longest run starting at word i that also occurs in one document (0 if under K words)."""
        R = [0] * len(toks)
        for i in range(len(toks) - K + 1):
            best = 0
            for d, j in self.pos.get(" ".join(toks[i:i + K]), ()):
                if d == exclude:
                    continue
                doc, r = self.docs[d], K
                while i + r < len(toks) and j + r < len(doc) and toks[i + r] == doc[j + r]:
                    r += 1
                if r > best:
                    best = r
                    if i + best == len(toks):
                        break
            R[i] = best
        return R

Everything word-based follows from that array. The GPT-3 rule flags the item when the largest R[i] is at least N. The 8-gram starting at position i occurs in the corpus exactly when R[i] is at least 8, so PaLM's share is a count. Llama 2's covered tokens are the union of the spans from i to i + R[i] wherever R[i] is at least 10. The words are normalised the way GPT-3 describes: split on whitespace, Unicode punctuation and symbols removed, lowercased.

GPT-4's rule needs no sampling either. The listing counts all m windows of 50 characters in the processed item and the h of them that occur in a processed training example. The probability that three distinct random windows include at least one match is 1 - C(m-h, 3) / C(m, 3). The tables report that probability averaged over items, which is the share the rule would flag in expectation, and also the share with any matching window, which is what checking every window would flag. The report does not mention case, so the listing keeps it.

The answer key: GSM8K and GSM-Plus

GSM8K (Cobbe et al., arXiv 2110.14168v2) is "a dataset of 8.5K high quality linguistically diverse grade school math word problems", split into 7,473 training and 1,319 test questions. We read both files from OpenAI's repository at commit 3101c7d5 and compared them with the Hugging Face parquet copy at revision 740312ad: identical, row for row. The 5th-percentile test question is 24 words long, so GPT-3's N clips to 13. The shortest question is 53 characters after GPT-4's processing, so every item gets three 50-character draws.

GSM-Plus (Li et al., arXiv 2402.19255v2) created "eight variations for each question" of the GSM8K test set, 10,552 in all. GPT-4 drafted the variations and annotators revised them; the paper reports that the annotators revised 18.85% of the drafts. The eight types are numerical substitution (a number replaced by one with the same number of digits), digit expansion, integer-decimal-fraction conversion, adding operation, reversing operation (a given fact becomes the question), problem understanding (the question reworded), distractor insertion (the data file calls it "distraction insertion") and critical thinking (an essential statement removed). Every one of the 10,552 records names its seed question, and all 1,319 seeds match a GSM8K test question exactly.

We used the variations only as text to search. For each test question and each type, the corpus is that question's own variation, as if it alone had ended up in a training set; a final run puts all 10,552 into one corpus. Nothing is trained, which matters because the dataset card says its "utilization as a training set is prohibited". All checks use the question text, not the solutions, which is also what the GPT-4 report says it did.

What each rule caught

Against the training split, the GPT-3 rule flags 3 of 1,319 questions, GPT-4's rule 1.9 questions in expectation, and the PaLM and Llama 2 dirty rules none. On the edited copies the rules come apart. The 13-word rule catches 74.2% of number-swapped copies, PaLM's rule 10.2% and Llama 2's dirty rule 22.6%. The gap is arithmetic. Of the 8-word windows in the test questions, 51.4% contain a digit, and a copy loses every window that holds a swapped number, so a 70% bar is usually out of reach; GPT-3's rule needs only one 13-word stretch to survive intact. GPT-4's three random draws land in between at 45.8%, and checking every window instead of three would raise that to 77.6%.

One question shows the disagreement up close. Test item 0 is the GSM8K question that begins "Janet’s ducks lay 16 eggs per day", 52 words after normalisation. Its numerical-substitution copy changes 16 to 20, three to five, four to six and $2 to $3, and nothing else. The swaps cut the question into four shared runs of 5, 13, 10 and 17 words, the longest being "per fresh duck egg how much in dollars does she make every day at the farmers market". Against that copy:

• the GPT-3 rule flags it, because the longest shared run is 17 words;
• PaLM's rule does not: 19 of its 45 8-grams are found, 42%;
• Llama 2's rule calls it neither clean nor dirty: 40 of 52 words sit in runs of 10 or more, 77%;
• GPT-4's rule finds 35 of its 172 windows, so three draws catch it with probability 1 - C(137, 3) / C(172, 3), 49.7%.

Same question, same copy: flagged, not flagged, in between, and a coin flip. The reworded copy of the same question ("Every day, Janet's ducks produce 16 eggs") shares no run of five words, no 8-gram and no 50-character window with the original. Across all 1,319 questions the reworded copies are flagged 0.8% of the time by the 13-word rule and at most 1.8% of the time by any of the rules, the highest being Llama 2's 20% "not clean" line.

The last row is a different case: a training set that took all of GSM-Plus. A question then escapes only if all eight of its copies escape, and the 13-word rule flags 99.5%. A leaked augmentation set is easy to find. A lone rewording is not.

How long a match should be

An item shares an n-gram with the corpus exactly when its longest shared run is at least n words, so one column of the CSV gives every threshold:

On the training split, a 6-word rule flags 34.8% of the test questions and an 8-word rule 5.8%. Read the 77 items flagged at 8 words and most of the matches are the stock phrasing of the genre, such as "how many hours a week does he spend" and "what is the sum of the ages of": the "spurious collisions" GPT-3 gives as its reason for a floor of 8. At 11 words the training split still matches sentences such as "how much money will she have left over after she buys" and "calculate the total amount of money the company paid to the". From 13 words the split is almost silent (0.2%), while the rule still catches 74.2% of number-swapped copies; at 20 words that falls to 36.5%. No length separates reworded copies from unrelated problems: at 8 words the reworded copies are flagged 6.4% of the time and the unrelated training split 5.8%. For questions this long, 13 is a sound default; for short items, GPT-3's percentile rule lowers N, and with it the precision.

The questions the training split flagged

At 13 words the training split flags three test items. Reading them is the point of running the check:

• Item 632 shares 25 words with training question 20 (both counted from 0). Both are "bought stamps at the post office" problems with snowflake, truck and rose designs; the test version has Max buy 16 snowflake stamps, 3 more truck stamps and 9 fewer rose stamps, the training version has Bella buy 11, 9 more and 13 fewer. That is a numerical substitution with a new name, sitting across GSM8K's own split.
• Item 602 shares 19 words with training question 1314: a plane that travels 1200 miles in 3 hours against a train that travels 270, both asking "how many additional hours would it take to travel an additional" distance.
• Item 581 shares 15 words with training question 406, a sentence about the lengths of two movies; the two questions ask different things.

Inside the test set, the same check finds two pairs that are one problem asked both ways: items 418 and 558 (the lockers of Timothy, Zack and Peter) and items 488 and 761 (Jana's puppies), reversing-operation pairs in GSM-Plus's terms. The PaLM and Llama 2 dirty rules flag none of these seven items. The GPT-4 rule's chance on item 632 is 63.1% (52 of its 185 windows match).

For scale, the GPT-4 report put GSM-8K's contamination at "~1%" in its Table 11, estimated from 1,000 randomly chosen examples against its whole pre-training data. Its Appendix E says part of GSM-8K's training split was mixed into training on purpose, and recommends reading its GSM-8K score "as something in-between true few-shot transfer and full benchmark-specific tuning". Against the training split alone, the same rule gives 0.1% in expectation here, and 0.3% when every window is checked.

What these checks cannot see

Rewordings, first of all. Yang et al. (arXiv 2311.04850v2) make the point at scale in their abstract: string matching is "insufficient", "simple variations of test data (e.g., paraphrasing, translation) can easily bypass these decontamination measures", and a 13B model trained on such variations of benchmarks including MMLU and HumanEval reached scores "on par with GPT-4". Their LLM-based detector found that "8-18% of the HumanEval benchmark overlaps" with RedPajama-Data-1T and StarCoder-Data. We did not run that detector; the 0.8% above is what the n-gram rules see of a rewording.

The training data, second. Every rule here needs the corpus. For a hosted model with an unpublished training set, none of them can be run from outside; what remains is the vendor's own report and black-box tests that probe the model instead of the data, which this article does not cover.

Third, the details of the Llama 2 rule. It matched tokens from Llama 2's tokenizer with a four-token skip budget; the listing matches words exactly, so against a skip-gram version of the same word rule it can only under-count. Tokenizers also split the same text very differently (six tokenizers, measured), so a token count of 10 is not a word count of 10.

Fourth, the answers. The checks above read the questions only; a leaked worked solution is a separate check on the solution text.

Last, scale. The listing's index is plain Python. On GSM8K's training split it held 336,935 word positions in 203.0 MiB, 632 bytes per position, and took about 3 seconds to build. That is fine for a fine-tuning set of a few million words, but at 632 bytes per position, 100 million words would need about 63 GB. Past that size, store 8-byte hashes of the n-grams in a sorted array, 8 GB per billion positions for one n, or do what the reports did: GPT-3 used Apache Spark to "compute exact collisions", and Llama 2 used suffix arrays on a PySpark cluster.

What to do with a flag

Read the pairs. Of the three items flagged here, one is a renamed copy, one shares a template and one shares a sentence. The check produces candidates; reading them produces the verdict.

Report both scores. GPT-3 scored each benchmark on its clean subset and reported the relative change from the full score, PaLM reported its clean subsets beside the full sets, and Llama 2 compared four subsets and called a dataset affected only when all four had an absolute Z above 2. A difference between a clean and a full score needs an interval before it means anything, and bootstrap intervals on eval deltas shows how to compute one.

Check before training, not after. If you own the training data, run the check on it before fine-tuning and remove the colliding spans, as GPT-3's filter did. A golden test set needs the same care: once its items are pasted into prompts and few-shot examples, they can travel into training sets.

Expect rewordings to pass. An n-gram check bounds verbatim and lightly edited leakage, and nothing more.

Running it

On your own files, JSONL with a question field (or --field) or plain text with one document per line:

python test-set-contamination-ngram-overlap.py --test my_eval.jsonl --corpus my_training.jsonl --csv report.csv

The reproduction downloads three files, checks each against a pinned SHA-256, and prints every table in this article:

$ python test-set-contamination-ngram-overlap.py --reproduce --cache data --csv test-set-contamination-ngram-overlap.csv
GSM8K test: 1319 questions; GSM8K train: 7473; GSM-Plus: 10552 variations
5th-percentile test question: 24 words, so GPT-3's N = 13
8-word windows of the test questions that contain a digit: 26,595 of 51,768 (51.4%)
13-word windows of the test questions that contain a digit: 31,432 of 45,173 (69.6%)

Share of the 1,319 test questions each rule flags (gpt3 at N = 13)
training corpus                                    gpt3   palm  l2 dirty  l2 >=20%  gpt4 x3  gpt4 all
GSM8K train (7,473 other problems)                 0.2%   0.0%      0.0%      0.7%     0.1%      0.3%
GSM8K test, the other questions                    0.3%   0.0%      0.0%      0.6%     0.2%      0.6%
own variant: numerical substitution               74.2%  10.2%     22.6%     91.3%    45.8%     77.6%
own variant: digit expansion                      84.3%  30.6%     43.9%     94.1%    63.5%     86.7%
own variant: integer-decimal-fraction conversion  93.0%  44.3%     60.8%     98.6%    75.4%     92.6%
own variant: adding operation                     89.7%  58.3%     60.3%     95.0%    75.6%     90.1%
own variant: reversing operation                  69.1%  14.0%     13.0%     77.9%    52.3%     70.4%
own variant: problem understanding                 0.8%   0.1%      0.2%      1.8%     0.4%      1.0%
own variant: distraction insertion                80.2%  46.9%     53.2%     89.8%    63.8%     80.1%
own variant: critical thinking                    95.6%  59.1%     66.6%     98.9%    83.3%     95.1%
all 10,552 GSM-Plus variations                    99.5%  98.1%     98.2%     99.8%    97.1%     98.9%

Share flagged by a single shared run of at least n words (any-hit at n)
training corpus                                     n=5    n=6    n=8   n=10   n=13   n=20   n=30   n=50
GSM8K train (7,473 other problems)                67.9%  34.8%   5.8%   0.7%   0.2%   0.1%   0.0%   0.0%
GSM8K test, the other questions                   41.1%  16.0%   2.7%   0.8%   0.3%   0.0%   0.0%   0.0%
own variant: numerical substitution               99.8%  99.7%  97.3%  91.7%  74.2%  36.5%  12.6%   1.5%
own variant: digit expansion                      99.8%  99.8%  98.4%  94.6%  84.3%  55.6%  25.0%   3.9%
own variant: integer-decimal-fraction conversion 100.0% 100.0%  99.8%  98.6%  93.0%  67.9%  32.8%   6.2%
own variant: adding operation                     99.8%  99.5%  98.1%  95.2%  89.7%  71.6%  40.3%  11.1%
own variant: reversing operation                  95.9%  93.0%  86.4%  78.9%  69.1%  50.2%  27.7%   6.9%
own variant: problem understanding                37.8%  21.9%   6.4%   3.0%   0.8%   0.3%   0.1%   0.0%
own variant: distraction insertion                99.8%  99.1%  96.3%  90.2%  80.2%  59.5%  32.2%   7.5%
own variant: critical thinking                   100.0%  99.9%  99.7%  98.9%  95.6%  78.2%  46.0%   9.8%
all 10,552 GSM-Plus variations                   100.0% 100.0% 100.0%  99.8%  99.5%  91.0%  63.5%  17.7%

Test items sharing a run of 13+ words with a training question: 3 (item 632: 25 words; item 602: 19 words; item 581: 15 words)
Test items sharing a run of 13+ words with another test question: 4 (item 418: 18 words; item 558: 18 words; item 488: 14 words; item 761: 14 words)

datasets/test-set-contamination-ngram-overlap.csv has one row per test item and corpus, 1,319 items by 11 corpora, 14,509 rows. It holds our counts only, no question text: GSM8K's text is under the MIT licence and GSM-Plus's under CC BY-SA 4.0, and the listing fetches both.

_README:

• test_index: the item's line in GSM8K's test.jsonl, counted from 0
• corpus: gsm8k_train; gsm8k_test_others (the other 1,318 test questions); own_variant:<type> (only that item's GSM-Plus variation of that type); gsmplus_all (all 10,552 variations)
• words: the item's word count after normalisation
• longest_run: the longest run of words shared with one corpus document, 0 when under 5
• ngrams8, ngrams8_found: the item's 8-grams and how many occur in the corpus (PaLM's rule)
• tokens_covered_L10: words inside shared runs of 10 or more (Llama 2's rule, with words for tokens)
• sub50_windows, sub50_hits: 50-character windows of the processed item and how many occur in a processed corpus document (GPT-4's rule)