GPU choice for fine-tuning is mostly a memory question, sometimes a bandwidth question, and only occasionally a FLOPs question. The hourly price is the least useful number to compare, because a GPU that's 40% more expensive per hour can be cheaper per run. Here are the published specifications, what they mean for each kind of run, and how to compare properly.
The specifications that matter#
| H100 SXM | H200 SXM | B200 | |
|---|---|---|---|
| Memory | 80 GB HBM3 | 141 GB HBM3e | 192 GB HBM3e (180 GB usable in HGX/DGX B200 systems) |
| Memory bandwidth | 3.35 TB/s | 4.8 TB/s | 8 TB/s |
| Dense BF16 tensor | ~989 TFLOPS | ~989 TFLOPS | ~2,250 TFLOPS |
From NVIDIA product specifications. Dense figures; NVIDIA often quotes sparse throughput, which is 2× these values. H200 uses the same Hopper compute as H100 with larger, faster memory.
The H200's advantage over the H100 is entirely memory: 76% more capacity and 43% more bandwidth at the same compute. B200 roughly doubles dense compute and more than doubles bandwidth versus H100.
What fits where#
Using the accounting in how much GPU memory fine-tuning needs (4k context, activation checkpointing):
| Run | Memory | H100 80GB | H200 141GB | B200 180GB |
|---|---|---|---|---|
| LoRA, 32B | ~72 GB | 1 (barely) | 1, with room for batch | 1 |
| LoRA, 70B | ~153 GB | 4 (sharded) | 2 (sharded) | 1 |
| QLoRA, 70B | ~51 GB | 1 | 1 | 1 |
| Full FT, 8B | ~130 GB | 2 | 1 | 1 |
| Full FT, 70B | ~1.1 TB | 16–32 | 8–16 | 8 |
Fewer GPUs per job means less inter-GPU communication, which is often the real speed-up. A 70B LoRA run on a single B200 avoids sharding entirely.
Compare cost per run, not per hour#
Training time for a compute-bound run is roughly , where is per-GPU dense throughput. Cost per run is then
Notice the GPU count cancels: for compute-bound work, cost depends on price per unit of effective throughput. With B200 delivering about 2.3× the dense BF16 throughput of an H100, B200 is cheaper per run whenever its hourly price is less than about 2.3× the H100's at equal utilisation. H200, with the same compute as H100, is cheaper per run only when its extra memory lets you use fewer GPUs, larger batches or skip sharding — common for LoRA on 70B, rare for small models.
RL runs are different: most time goes to generating rollouts, which is memory-bandwidth-bound. There H200 and B200's bandwidth advantages (1.4× and 2.4× over H100) translate more directly into speed.
Recommendations#
- LoRA SFT/DPO up to ~32B: H100. Widely available and the cheapest per hour; memory is sufficient.
- LoRA on 70B or long-context runs: H200 if priced within ~30% of H100, because it halves GPU count.
- Full fine-tuning or large GRPO runs: B200 when available, judged on cost per run.
- Prototyping on a budget: QLoRA on whatever single GPU is cheapest.
POFE's broker quotes all three across providers and picks per job; see the cost calculator, fine-tuning cost guide and break-even model.
Frequently asked questions#
Is H200 better than H100 for fine-tuning?
H200 has the same compute as H100 but 141 GB of memory instead of 80 GB and 43% more bandwidth. It is better when memory limits you — LoRA on 70B models, long contexts, larger batches — and offers little for small models that already fit on an H100.
How much faster is B200 than H100 for training?
B200's dense BF16 throughput is about 2.25 PFLOPS versus about 0.99 for H100, roughly 2.3×, with 2.4× the memory bandwidth. Real speed-ups depend on whether the run is compute-, memory- or communication-bound.
What GPU do I need to fine-tune Llama 70B?
With QLoRA, one 80 GB GPU. With LoRA in bf16, about 150 GB: four H100s, two H200s or one B200. Full fine-tuning needs on the order of 1.1 TB across the cluster.
References#
- NVIDIA. H100 Tensor Core GPU — product specifications. www.nvidia.com/en-us/data-center/h100/
- NVIDIA. H200 Tensor Core GPU — product specifications. www.nvidia.com/en-us/data-center/h200/
- NVIDIA. HGX platform (B200) — product specifications. www.nvidia.com/en-us/data-center/hgx/