Speed test matrix
Olmo-3-7B-Think on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF164 contexts (8K–59K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 59K, 64K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- olmo
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-allenai-vllm/bin/vllm" serve allenai/Olmo-3-7B-Think --served-model-name olmo --trust-remote-code --tensor-parallel-size 1 --max-model-len 65536 --gpu-memory-utilization 0.869211 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context32K170 ms TTFT
16K context30.1K353 ms TTFT
32K context29.9K703 ms TTFT
59K context29.4K1,315 ms TTFT
64K context28K1,497 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 165.746 ms TTFT | 303.461 ms TTFT | 511.888 ms TTFT | 774.6133 ms TTFT | 1,033.8215 ms TTFT | 1,253.6308 ms TTFT | 1,418.8433 ms TTFT | Over capacity |
| 16K | 161.457 ms TTFT | 292.589 ms TTFT | 483.4121 ms TTFT | 710.8182 ms TTFT | 913.6240 ms TTFT | 1,071.1343 ms TTFT | Over capacity | Over capacity |
| 32K | 153.295 ms TTFT | 272.2126 ms TTFT | 439.6176 ms TTFT | 609.5280 ms TTFT | 751.1314 ms TTFT | Over capacity | Over capacity | Over capacity |
| 59K | 141.4132 ms TTFT | 245.3161 ms TTFT | 378.4234 ms TTFT | 491.0354 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 32K | — | 170 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 30.1K | — | 353 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 29.9K | — | 703 ms | — | — | 1 | — | — |
| Prefill | 59K | 1 | Valid | 29.4K | — | 1,315 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 28K | — | 1,497 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 165.7 | 166.5 | 46 ms | 6.0 ms | 1.0 | 1 | 4,972 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 303.4 | 153.1 | 61 ms | 6.5 ms | 2.0 | 2 | 9,100 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 511.8 | 132.4 | 88 ms | 7.6 ms | 3.9 | 4 | 15,353 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 774.6 | 98.0 | 133 ms | 10.2 ms | 8.0 | 0 | 23,240 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,033.8 | 65.4 | 215 ms | 15.3 ms | 16.0 | 0 | 31,008 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,253.6 | 39.5 | 308 ms | 25.3 ms | 32.0 | 0 | 37,600 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,418.8 | 22.3 | 433 ms | 44.8 ms | 64.0 | 0 | 42,560 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 161.4 | 162.2 | 57 ms | 6.2 ms | 1.0 | 1 | 4,843 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 292.5 | 147.9 | 89 ms | 6.8 ms | 2.0 | 2 | 8,776 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 483.4 | 124.8 | 121 ms | 8.0 ms | 4.0 | 4 | 14,500 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 710.8 | 89.6 | 182 ms | 11.2 ms | 8.0 | 0 | 21,320 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 913.6 | 57.5 | 240 ms | 17.4 ms | 16.0 | 0 | 27,408 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,071.1 | 33.7 | 343 ms | 29.7 ms | 32.0 | 0 | 32,128 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 153.2 | 154.5 | 95 ms | 6.5 ms | 1.0 | 1 | 4,597 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 272.2 | 138.3 | 126 ms | 7.2 ms | 2.0 | 2 | 8,166 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 439.6 | 110.4 | 176 ms | 9.1 ms | 4.0 | 0 | 13,188 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 609.5 | 76.8 | 280 ms | 13.0 ms | 8.0 | 0 | 18,280 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 751.1 | 47.2 | 314 ms | 21.2 ms | 16.0 | 0 | 22,528 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 1 | Valid | 141.4 | 143.1 | 132 ms | 7.0 ms | 1.0 | 1 | 4,242 | 30.0 s |
| Sustained decode | 59K | 2 | Valid | 245.3 | 123.0 | 161 ms | 8.1 ms | 2.0 | 2 | 7,335 | 29.9 s |
| Sustained decode | 59K | 4 | Valid | 378.4 | 95.0 | 234 ms | 10.5 ms | 4.0 | 0 | 11,352 | 30.0 s |
| Sustained decode | 59K | 8 | Valid | 491.0 | 61.7 | 354 ms | 16.2 ms | 8.0 | 0 | 14,736 | 30.0 s |
| Sustained decode | 59K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.