Research notebook
Each note isolates one question we kept being asked, answers it with a small model or simulation we actually ran, and states its limits. Charts are generated from the scripts' output, not drawn by hand. Subscribe via Atom.
Training methods
What the knobs in DPO and GRPO actually do.
GRPO group size under a fixed rollout budget
A reproducible simulation of GRPO with binary verifiable rewards: how often groups carry no gradient, why smaller groups can win under a fixed budget, and what std normalisation really does.
Read →SimulationWhat DPO's β actually controls — and why chosen likelihoods fall
A derivation of the DPO loss and a reproducible simulation sweeping β from 0.01 to 1: reward vs KL, held-out preference accuracy, and why the log-probability of chosen responses goes down.
Read →Evaluation
How big an eval needs to be, and how far to trust a judge.
How many eval examples do you need to trust a 3-point gain?
Power analysis for LLM evals with a reproducible simulation: minimum detectable effect by eval size, why paired comparisons need fewer items, and a sizing table you can use before you build the golden set.
Read →StatisticsLLM-as-a-judge agreement: what Cohen's κ hides
Why the same LLM judge can score κ = 0.80 on one eval and 0.43 on another, how judge error biases pass rates and shrinks model gaps, and how to correct for it. Closed-form results plus simulation.
Read →Cost and infrastructure
Deterministic models with every assumption on the page.
When does a fine-tuned model beat the API on cost?
A transparent cost model comparing a self-hosted fine-tuned model with a hosted frontier API: the break-even token volume, the full cost curve, and which assumptions actually move the answer.
Read →ModelHow much GPU memory does fine-tuning need?
The memory formula for full fine-tuning, LoRA and QLoRA — weights, gradients, Adam state and activations — with a computed table for 8B to 70B models and the minimum H100 count for each.
Read →Methodology
All experiments are seeded Python (numpy, scipy) and run on a laptop CPU in under three minutes. Each page states its model, assumptions and limitations; these are controlled toy settings chosen to isolate one mechanism, not benchmarks of production training runs. Results tables and SVG charts are generated at build time from the scripts' JSON output.
Bring a model and a metric.
Tell us what the model should get better at. A post-training engineer replies within one business day with a plan, a compute estimate and an eval design.