gpu-cloud-hpc • 8 min read (1415 words)

NVIDIA H100 vs. H200 vs. Blackwell B200: Published Specs, Dense vs. Sparse TFLOPS & Roofline Token Economics

Architectural comparison of Hopper and Blackwell datacenter GPUs based on official NVIDIA datasheets, Dense vs. 2:4 Sparse FP8 Tensor ratings, theoretical HBM3e roofline math, and public cloud rate cards.

Architectural diagram comparing NVIDIA H100, H200, and B200 GPU memory subsystems

Methodology: Analytical model based on official NVIDIA Hopper/Blackwell architecture datasheets and public cloud rate cards (AWS, GCP, Lambda, RunPod). All throughput figures represent theoretical memory-bandwidth roofline models, not proprietary lab measurements.

1. Understanding Dense vs. 2:4 Structured Sparsity in Official GPU Datasheets

When evaluating datacenter accelerators for Large Language Model (LLM) training and inference, hardware datasheets frequently publish peak Tensor Core throughput using 2:4 structured sparsity—a hardware feature introduced in NVIDIA Ampere and carried forward into Hopper (H100/H200) and Blackwell (B200) that doubles theoretical FLOPS when exactly two out of every four contiguous weights are zero.

However, almost all production open-weight foundation models (Llama-3.1-70B, Llama-3.3-70B, Qwen-2.5-72B, Mistral-Large, and DeepSeek-V3) are distributed and served as dense weight matrices. Unless an engineering team explicitly prunes a model to a 2:4 sparse pattern and fine-tunes it to recover accuracy loss, the GPU executes dense matrix multiplications (GEMM), which run at exactly 50% of the advertised sparse TFLOPS rating.

Furthermore, transformer inference is divided into two distinct computational regimes:
- Prefill (Prompt Processing — Compute-Bound): Evaluates all input prompt tokens in parallel. Prefill latency (Time to First Token, or TTFT) scales primarily with Dense FP8 / BF16 Tensor Core TFLOPS.
- Autoregressive Decode (Token Generation — Memory-Bandwidth-Bound): Generates one output token per step per sequence. At low-to-moderate batch sizes, every decode step must stream the entire set of active model weights and KV-cache tensors from High Bandwidth Memory (HBM3 or HBM3e) into the Streaming Multiprocessors (SMs).

Consequently, during the decode phase, HBM memory bandwidth (TB/s) and VRAM capacity (GB) dictate maximum token generation speed and determine whether a model fits on a single GPU (TP=1) or must be sharded across multiple GPUs using Tensor Parallelism (TP=2, TP=4, or TP=8).

2. Official Hardware Specifications: Dense vs. Sparse TFLOPS & Public Cloud Rate Cards

The table below compiles official hardware specifications from NVIDIA's Hopper and Blackwell datasheets alongside published on-demand hourly rate ranges across specialist GPU providers (Lambda, RunPod) and hyperscalers (AWS EC2 P5, Google Cloud GPU Pricing).

Key Architectural Observations from the Specifications

1. Identical Compute Dies on H100 vs. H200: Notice that the H100 SXM5 and H200 SXM share the exact same GH100 compute die (989.5 TFLOPS Dense BF16 and 1,978.9 TFLOPS Dense FP8). The upgrade in the H200 is entirely in the memory subsystem: transitioning from 5 stacks of HBM3 (80 GB @ 3.35 TB/s) to 6 stacks of HBM3e (141 GB @ 4.80 TB/s)—a 76.2% increase in VRAM capacity and a 43.3% increase in memory bandwidth. 2. Blackwell B200 Dual-Reticle Architecture: The B200 SXM pairs two reticle-limited Blackwell dies connected via a 10 TB/s chip-to-chip link, delivering 2.27x the Dense FP8 compute (4,500 TFLOPS vs. 1,978.9 TFLOPS) and 2.38x the memory bandwidth (8.00 TB/s vs. 3.35 TB/s) of the H100 SXM5, while also introducing native FP4 Tensor Core support (9,000 TFLOPS Dense / 18,000 TFLOPS Sparse).

3. Theoretical Roofline Math: Why VRAM Capacity Changes Tensor Parallelism Economics

We can model the impact of HBM capacity and bandwidth on a 70-billion-parameter dense model (Llama-3.1-70B-Instruct) using first-principles roofline arithmetic.

Step 1: Memory Footprint Calculation (FP16 vs. FP8)

- In FP16 / BF16 (2 bytes per parameter): $$ ext{Model Weights} = 70 imes 10^9 imes 2 ext{ bytes} = 140 ext{ GB}$$ Because 140 GB exceeds the 80 GB VRAM of a single H100, running unquantized BF16 inference requires at least 2x H100 GPUs (TP=2, 160 GB total VRAM), leaving only ~14 GB across both GPUs for the KV-cache after CUDA context and activation buffers. - In FP8 E4M3 Quantization (1 byte per parameter): $$ ext{Model Weights} = 70 imes 10^9 imes 1 ext{ byte} pprox 70 ext{ GB}$$ On a single 80 GB H100, loading 70 GB of weights plus ~4 GB of runtime overhead leaves only ~6 GB for the KV-cache. With Llama-3.1-70B (80 layers, 8 KV heads with GQA, 128 head dimension), each token of context consumes: $$ ext{KV Cache per Token (FP16)} = 2 imes 80 imes 8 imes 128 imes 2 ext{ bytes} = 327,680 ext{ bytes} pprox 0.3125 ext{ MB}$$ Thus, 6 GB of free VRAM holds only ~19,200 tokens of total active context across all concurrent users—forcing production multi-user servers on H100 to either restrict concurrency or scale out to 2x H100 (TP=2). - On a Single 141 GB H200 (TP=1): Loading the same 70 GB FP8 model leaves ~65 GB of free HBM3e exclusively for the KV-cache. Especially when combined with FP8 KV-cache quantization (0.156 MB/token), a single H200 can hold over 400,000 tokens of concurrent KV-cache without splitting the model across two GPUs or paying the NVLink all-reduce latency penalty after every transformer layer.

Step 2: Single-Stream Memory-Bandwidth Roofline Ceiling

For a single active sequence (batch_size = 1), the theoretical upper bound on decode speed is governed strictly by how fast the GPU can read 70 GB of FP8 weights from HBM: $$ ext{Theoretical Max Decode Speed (Batch=1)} = rac{ ext{Memory Bandwidth (GB/s)}}{ ext{Model Weight Size (GB)}}$$

- 1x H100 SXM5 (70 GB FP8 @ 3,350 GB/s): $rac{3350}{70} pprox 47.8 ext{ tokens/sec}$ theoretical ceiling (typically ~34–40 tok/s in practice at ~75–82% effective bandwidth utilization).
- 1x H200 SXM (70 GB FP8 @ 4,800 GB/s): $rac{4800}{70} pprox 68.5 ext{ tokens/sec}$ theoretical ceiling (~50–56 tok/s at 75–82% utilization).
- 1x B200 SXM (70 GB FP8 @ 8,000 GB/s): $rac{8000}{70} pprox 114.2 ext{ tokens/sec}$ theoretical ceiling (~85–94 tok/s at 75–82% utilization).

4. How to Run Your Own Reproducible vLLM Benchmarks

To measure actual throughput and latency on your own cloud instances, use the official vllm serving engine and its built-in benchmark CLI (vllm bench serve, or benchmarks/benchmark_serving.py from the official vLLM repository):

# 1. Start the vLLM OpenAI-Compatible Server (Example: FP8 model + FP8 KV-Cache on 1x H200) vllm serve neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 \ --tensor-parallel-size 1 \ --kv-cache-dtype fp8 \ --max-model-len 16384 \ --gpu-memory-utilization 0.92 \ --enable-chunked-prefill \ --port 8000

# 2. Run the Official vLLM Serving Benchmark CLI Against Your Endpoint vllm bench serve \ --backend vllm \ --model neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 \ --dataset-name random \ --random-input-len 2048 \ --random-output-len 512 \ --max-concurrency 64 \ --num-prompts 300 \ --save-result

Calculating Your True Cost per 1 Million Output Tokens

Once vllm bench serve outputs your Output token throughput (tok/s) for your target concurrency and P95 latency SLA, plug that number into the following formula alongside your provider's hourly rate:

$$ ext{Cost per 1M Output Tokens (\$)} = \left(rac{ ext{Hourly GPU Node Cost (\$)}}{3,600 ext{ sec/hr}}
ight) imes \left(rac{1,000,000 ext{ tokens}}{ ext{Sustained Output Throughput (tok/s)}}
ight)$$

You can also compare cluster burn rates against managed API pricing interactively using our GPU Cluster & Token Cost Calculators.