All articles
9 articles, newest first. Browse by topic or subscribe to the RSS feed.
2026-09
-
A golden test set for a chatbot, scored without an LLM judge
How to build a golden set for a support bot and score it with exact match, token F1, ROUGE-L, and per-item assertions. Two bots, 48 items, code and data included.
-
An agent tool-use loop from scratch: states and stop rules
A tool-use loop built as an explicit state machine with six stop conditions and a token budget, measured over 2,000 simulated episodes. Code included.
-
Structured output that never breaks: JSON Schema, validation, retries
A validate-and-retry loop for model JSON, with a small schema validator, a tolerant extractor, and measured retry rates from a 2,000-run simulation.
-
A minimal retrieval pipeline in Python: chunking, TF-IDF, evaluation
A complete retrieval pipeline in the standard library, run on 216 arXiv abstracts, with Recall@k and MRR measured for whole-document versus chunked indexing.
-
Systems ·
KV cache arithmetic: memory per token, batch size, and context cost
Bytes of KV cache per token from a config.json, worked for Qwen3-8B and Qwen3-32B at four context lengths on an 80 GB device, with GQA and int8 KV as the two levers.
-
Gemini 3.8 Flash released: what changed, from the primary sources
Google's 2 September 2026 release of Gemini 3.8 Flash, dated from the announcement, with the model page limits, the pricing expiry, and the eval table's footnotes.
-
Bootstrap CIs for eval deltas: is a 4-point gap on 500 items real?
A paired bootstrap for the accuracy gap between two systems on 500 items, with McNemar's test, coverage checked over 1,000 replays, and the code and per-item data.
-
EarlyEval explained: stopping agent benchmark runs early
A reading of arXiv:2609.02783, which trains classifiers to halt agent runs once the outcome is predictable, and what its Table III actually shows.
-
Covert Indirect Prompt Injection (ICoA) explained
A reading of arXiv:2608.30362, which splits attack success into covert and overt outcomes on AgentDojo and shows the covert share can be engineered.