Systems
Inference, quantisation, GPUs, cost, latency: the engineering under the model.
Systems covers the engineering underneath the model: memory, number formats, throughput and what they cost. Both articles so far take their figures from primary material, a model's own config.json and a project's own source header, rather than from rules of thumb.
KV cache arithmetic works out the bytes of cache per token from a model's config.json, for Qwen3-8B and Qwen3-32B at four context lengths on an 80 GB device, and shows the two levers that change the answer: grouped-query attention and an int8 KV cache. What a quantisation format actually costs derives the real bits per weight of every GGUF block type from llama.cpp's ggml-common.h (Q4_K stores 4.5 bits per weight, not 4), joins them to llama.cpp's published size and perplexity tables, and names the formats that are strictly worse than another.
Read the KV cache piece first. Its one-line formula explains why long context is expensive, and the quantisation piece then shows what a smaller number format buys back. The TORQUE reading in Paper Readings continues the quantisation thread with a measured re-implementation. Planned for this section: vLLM against llama.cpp serving measurements, speculative decoding, batching and throughput, and GPU cost per million tokens with dated prices from the providers' own pages.
-
What a quantisation format actually costs, derived from the source
Q4_K is not four bits per weight. Exact costs derived from ggml-common.h, joined to llama.cpp's measured size and perplexity tables, and the dominated formats named.
-
KV cache arithmetic: memory per token, batch size, and context cost
Bytes of KV cache per token from a config.json, worked for Qwen3-8B and Qwen3-32B at four context lengths on an 80 GB device, with GQA and int8 KV as the two levers.