Speed test matrix
Olmo-3.1-7B-RL-Zero-Math on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF164 contexts (8K–59K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 59K, 64K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- olmo
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-allenai-vllm/bin/vllm" serve allenai/Olmo-3.1-7B-RL-Zero-Math --served-model-name olmo --trust-remote-code --tensor-parallel-size 1 --max-model-len 65536 --gpu-memory-utilization 0.869211 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context31.5K171 ms TTFT
16K context29.8K356 ms TTFT
32K context29.4K714 ms TTFT
59K context28.7K1,347 ms TTFT
64K context28.1K1,489 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 166.747 ms TTFT | 304.962 ms TTFT | 515.979 ms TTFT | 778.3117 ms TTFT | 1,036.6199 ms TTFT | 1,258.0336 ms TTFT | 1,419.5400 ms TTFT | Over capacity |
| 16K | 162.174 ms TTFT | 294.376 ms TTFT | 486.1115 ms TTFT | 714.2187 ms TTFT | 916.6313 ms TTFT | 1,075.5352 ms TTFT | Over capacity | Over capacity |
| 32K | 154.280 ms TTFT | 274.4119 ms TTFT | 442.7192 ms TTFT | 611.8296 ms TTFT | 753.8365 ms TTFT | Over capacity | Over capacity | Over capacity |
| 59K | 142.2135 ms TTFT | 245.5156 ms TTFT | 380.2292 ms TTFT | 493.4331 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 31.5K | — | 171 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 29.8K | — | 356 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 29.4K | — | 714 ms | — | — | 1 | — | — |
| Prefill | 59K | 1 | Valid | 28.7K | — | 1,347 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 28.1K | — | 1,489 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 166.7 | 167.5 | 47 ms | 6.0 ms | 1.0 | 1 | 5,001 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 304.9 | 154.0 | 62 ms | 6.5 ms | 1.9 | 2 | 9,146 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 515.9 | 133.3 | 79 ms | 7.5 ms | 4.0 | 4 | 15,477 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 778.3 | 98.5 | 117 ms | 10.2 ms | 8.0 | 0 | 23,344 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,036.6 | 65.6 | 199 ms | 15.3 ms | 16.0 | 0 | 31,104 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,258.0 | 39.7 | 336 ms | 25.2 ms | 32.0 | 0 | 37,728 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,419.5 | 22.3 | 400 ms | 44.7 ms | 64.0 | 0 | 42,560 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 162.1 | 163.1 | 74 ms | 6.1 ms | 1.0 | 1 | 4,864 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 294.3 | 148.7 | 76 ms | 6.7 ms | 2.0 | 2 | 8,830 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 486.1 | 124.7 | 115 ms | 8.0 ms | 4.0 | 4 | 14,582 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 714.2 | 90.1 | 187 ms | 11.1 ms | 8.0 | 0 | 21,424 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 916.6 | 57.8 | 313 ms | 17.3 ms | 16.0 | 0 | 27,504 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,075.5 | 33.8 | 352 ms | 29.5 ms | 32.0 | 0 | 32,256 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 154.2 | 155.2 | 80 ms | 6.4 ms | 1.0 | 1 | 4,625 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 274.4 | 139.3 | 119 ms | 7.2 ms | 2.0 | 2 | 8,236 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 442.7 | 111.2 | 192 ms | 9.0 ms | 4.0 | 0 | 13,280 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 611.8 | 77.1 | 296 ms | 13.0 ms | 8.0 | 0 | 18,352 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 753.8 | 47.3 | 365 ms | 21.1 ms | 16.0 | 0 | 22,624 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 1 | Valid | 142.2 | 143.8 | 135 ms | 7.0 ms | 1.0 | 1 | 4,266 | 30.0 s |
| Sustained decode | 59K | 2 | Valid | 245.5 | 143.1 | 156 ms | 7.1 ms | 2.0 | 2 | 7,364 | 30.0 s |
| Sustained decode | 59K | 4 | Valid | 380.2 | 95.5 | 292 ms | 10.5 ms | 4.0 | 0 | 11,404 | 30.0 s |
| Sustained decode | 59K | 8 | Valid | 493.4 | 62.0 | 331 ms | 16.1 ms | 8.0 | 0 | 14,800 | 30.0 s |
| Sustained decode | 59K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.