Tutorials · 2026-09-06 · 1,092 words · 5 min read

Structured output that never breaks: JSON Schema, validation, retries

A validate-and-retry loop for model JSON, with a small schema validator, a tolerant extractor, and measured retry rates from a 2,000-run simulation.

structured output JSON Schema validation python

Every extraction pipeline I have seen fail in production failed the same way: the model returned something that was almost JSON. A code fence around it. A sentence before it. A number as a string. A required key missing on one document in five hundred. The fix is not a better prompt, though prompts help; it is a loop that validates every response against a schema and, when validation fails, sends the error back to the model and asks again. This article builds that loop from scratch in the standard library, measures what it does, and says when to replace the home-made validator with the real one.

The listing is code/structured-output-json-schema.py. It runs offline: the model is a deterministic simulator that produces the failure modes above at fixed rates, so the numbers it prints are properties of the loop, not of any particular model.

The schema

The example task is invoice extraction. The schema uses the subset of JSON Schema (draft 2020-12, per the specification documents linked below) that covers most extraction jobs:

SCHEMA = {
    "type": "object",
    "properties": {
        "vendor": {"type": "string", "minLength": 1},
        "total": {"type": "number", "minimum": 0},
        "currency": {"type": "string", "enum": ["USD", "EUR", "GBP"]},
        "line_items": {
            "type": "array",
            "minItems": 1,
            "items": {
                "type": "object",
                "properties": {"sku": {"type": "string"}, "qty": {"type": "integer", "minimum": 1}},
                "required": ["sku", "qty"],
                "additionalProperties": False,
            },
        },
    },
    "required": ["vendor", "total", "currency", "line_items"],
    "additionalProperties": False,
}

Three of those keywords do most of the work in practice. required catches the missing-key failure. additionalProperties: false catches the model inventing a notes field that your database has no column for. enum and minimum catch semantically wrong values that are syntactically fine, which is the class of error a JSON parser will never see.

The validator

The file implements type, properties, required, additionalProperties, enum, minimum, maximum, minLength, items and minItems, recursively, in about 45 lines. The important design choice is that every failure raises an error whose message names the JSON path and the rule that failed:

def validate(value, schema, path="$") -> None:
    t = schema.get("type")
    if t == "object":
        if not isinstance(value, dict):
            raise SchemaError(f"{path}: expected object, got {type(value).__name__}")
        for key in schema.get("required", []):
            if key not in value:
                raise SchemaError(f"{path}: missing required key {key!r}")
        props = schema.get("properties", {})
        if schema.get("additionalProperties") is False:
            extra = set(value) - set(props)
            if extra:
                raise SchemaError(f"{path}: unexpected keys {sorted(extra)}")
        for key, sub in props.items():
            if key in value:
                validate(value[key], sub, f"{path}.{key}")

$.line_items[0].qty: 0 below minimum 1 is a message a model can act on. ValidationError is not. The retry loop depends on that difference.

One detail worth copying: bool is a subclass of int in Python, so isinstance(True, int) is true. The integer and number checks exclude booleans explicitly, or a model that writes "qty": true will pass.

The extractor

Before validation, the raw response has to become a Python object, and models decorate their JSON. The extractor tries a fenced block first, then takes the first balanced {...} span by counting braces:

FENCE = re.compile(r"```(?:json)?\s*(.*?)```", re.S)

def extract_json(text: str):
    m = FENCE.search(text)
    if m:
        text = m.group(1)
    start = text.find("{")
    if start < 0:
        raise SchemaError("no JSON object in response")
    depth = 0
    for i in range(start, len(text)):
        if text[i] == "{":
            depth += 1
        elif text[i] == "}":
            depth -= 1
            if depth == 0:
                return json.loads(text[start:i + 1])
    raise SchemaError("unbalanced braces in response")

Brace counting is not a JSON parser and will be fooled by a } inside a string value. In practice the fenced-block path handles most decorated responses, and the brace scan is a fallback for prose-wrapped ones; if a string field legitimately contains braces, parse with json.JSONDecoder.raw_decode from the first { instead.

The loop

def extract_with_retries(prompt, schema, rng, max_attempts=3):
    messages = [{"role": "user", "content": prompt}]
    for attempt in range(1, max_attempts + 1):
        raw = call_model(messages, rng)
        try:
            obj = extract_json(raw)
            validate(obj, schema)
            return obj, attempt, None
        except (SchemaError, json.JSONDecodeError) as e:
            err = f"{type(e).__name__}: {e}"
            messages.append({"role": "assistant", "content": raw})
            messages.append({"role": "user", "content": f"That response failed validation: {err}. "
                             "Return only a JSON object matching the schema, no prose, no code fence."})
    return None, max_attempts, err

The failed response goes back into the conversation as the assistant turn, followed by the error as a user turn. The model sees what it wrote and exactly which rule it broke. The alternative, resending the original prompt with a higher temperature, throws that information away.

What the simulation measured

The simulator fails 40% of first attempts, choosing uniformly among seven failure modes (fence, prose, wrong type, missing key, out-of-range value, extra key, truncated JSON), and halves the failure rate on each retry to stand in for the effect of error feedback. Those rates are inputs, not findings; I set them to be pessimistic. With seed 7 and 2,000 items:

runs: 2000
valid on attempt 1: 1418 (70.9%)
valid on attempt 2: 494 (24.7%)
valid on attempt 3: 80 (4.0%)
gave up after 3:    8 (0.40%)
mean model calls per item: 1.335

Note that "valid on attempt 1" is 70.9%, not 60%: two of the seven failure modes (fence and prose) are repaired by the extractor without a retry, so they never count as failures. That is the extractor earning its keep. With a 40% raw failure rate, the loop reaches 99.6% valid output at a cost of 1.34 model calls per item. If your model's real first-attempt failure rate is 10%, the same arithmetic gives roughly 1.05 calls per item. Measure yours; the script's call_model is the only function to replace.

Writing schemas the model can satisfy

The loop tolerates a bad schema, but a bad schema raises the retry rate and the bill, so a few habits pay for themselves. Keep required fields to the ones you will actually reject a record without; every extra required key is another way for a document that lacks that information to fail three times and land in the review queue. Use enum for anything with a closed vocabulary, and put the vocabulary in the prompt as well as the schema, because the model cannot see the schema unless you show it. Prefer null in the schema ("type": ["string", "null"]) over omission for fields that are legitimately absent, so that "not present" and "the model forgot" are distinguishable in the output; the small validator here checks the first listed type only, and this is one of the places the full library earns its install.

Numbers deserve a rule of their own. The simulator's most common real-world failure, a total returned as the string "41.50", is what happens when the model copies text from the document verbatim, and it is correct behaviour from the model's point of view. Decide once whether monetary fields are numbers or strings, say so in the prompt, and enforce it in the schema; do not silently coerce in code, or the same document will produce a different record depending on which path it took.

Finally, log the failures. The err string that goes back to the model is also the best diagnostic you have: a week of them, grouped by JSON path and rule, tells you which fields the model finds hard, which is where the prompt needs an example, and which are hard because the documents are ambiguous, which no prompt will fix.

When to use the real validator

The validator in the file is deliberately partial. It does not implement $ref, oneOf, pattern, format, uniqueItems, or conditional schemas, and it does not report multiple errors at once. For production, install jsonschema (4.26.0 on PyPI at the time of writing) and call jsonschema.validate(obj, SCHEMA); its ValidationError carries json_path and message, which is all the retry prompt needs. The reason to read the small one first is that once you have written the recursion yourself, the full library's error output stops being mysterious.

Two more production notes. First, if your provider supports constrained decoding against a schema, use it; it removes the syntactic failures entirely and leaves only semantic ones, which validation still has to catch. Second, keep max_attempts small. Three is enough: the 0.4% that fail three times in the simulation are, in real systems, usually documents the model cannot extract at all, and they belong in a review queue, not a fourth retry.