Speed test matrix
Olmo-3-7B-Instruct on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF164 contexts (8K–59K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 59K, 64K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- olmo
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-allenai-vllm/bin/vllm" serve allenai/Olmo-3-7B-Instruct --served-model-name olmo --trust-remote-code --tensor-parallel-size 1 --max-model-len 65536 --gpu-memory-utilization 0.869211 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context31K174 ms TTFT
16K context30.5K347 ms TTFT
32K context30.1K699 ms TTFT
59K context28.6K1,352 ms TTFT
64K context28.2K1,485 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 165.743 ms TTFT | 303.550 ms TTFT | 512.754 ms TTFT | 775.4134 ms TTFT | 1,034.7225 ms TTFT | 1,255.4281 ms TTFT | 1,420.0443 ms TTFT | Over capacity |
| 16K | 161.366 ms TTFT | 292.675 ms TTFT | 483.287 ms TTFT | 711.0176 ms TTFT | 914.1260 ms TTFT | 1,072.8352 ms TTFT | Over capacity | Over capacity |
| 32K | 153.477 ms TTFT | 272.9109 ms TTFT | 440.5175 ms TTFT | 609.6287 ms TTFT | 751.2359 ms TTFT | Over capacity | Over capacity | Over capacity |
| 59K | 141.5131 ms TTFT | 245.3168 ms TTFT | 379.0256 ms TTFT | 491.8343 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 31K | — | 174 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 30.5K | — | 347 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30.1K | — | 699 ms | — | — | 1 | — | — |
| Prefill | 59K | 1 | Valid | 28.6K | — | 1,352 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 28.2K | — | 1,485 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 165.7 | 166.5 | 43 ms | 6.0 ms | 1.0 | 1 | 4,971 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 303.5 | 153.1 | 50 ms | 6.5 ms | 2.0 | 2 | 9,106 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 512.7 | 132.4 | 54 ms | 7.6 ms | 4.0 | 4 | 15,379 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 775.4 | 98.1 | 134 ms | 10.2 ms | 8.0 | 0 | 23,256 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,034.7 | 65.5 | 225 ms | 15.3 ms | 16.0 | 0 | 31,056 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,255.4 | 39.6 | 281 ms | 25.3 ms | 32.0 | 0 | 37,664 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,420.0 | 22.3 | 443 ms | 44.7 ms | 64.0 | 0 | 42,560 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 161.3 | 162.2 | 66 ms | 6.2 ms | 1.0 | 1 | 4,840 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 292.6 | 147.9 | 75 ms | 6.8 ms | 2.0 | 2 | 8,778 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 483.2 | 124.4 | 87 ms | 8.0 ms | 4.0 | 4 | 14,494 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 711.0 | 89.6 | 176 ms | 11.2 ms | 8.0 | 0 | 21,328 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 914.1 | 57.6 | 260 ms | 17.4 ms | 16.0 | 0 | 27,408 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,072.8 | 33.7 | 352 ms | 29.6 ms | 32.0 | 0 | 32,160 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 153.4 | 154.4 | 77 ms | 6.5 ms | 1.0 | 1 | 4,601 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 272.9 | 138.4 | 109 ms | 7.2 ms | 2.0 | 2 | 8,186 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 440.5 | 110.7 | 175 ms | 9.0 ms | 4.0 | 0 | 13,216 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 609.6 | 76.8 | 287 ms | 13.0 ms | 8.0 | 0 | 18,288 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 751.2 | 47.2 | 359 ms | 21.2 ms | 16.0 | 0 | 22,544 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 1 | Valid | 141.5 | 143.0 | 131 ms | 7.0 ms | 1.0 | 1 | 4,244 | 30.0 s |
| Sustained decode | 59K | 2 | Valid | 245.3 | 123.0 | 168 ms | 8.1 ms | 2.0 | 2 | 7,337 | 29.9 s |
| Sustained decode | 59K | 4 | Valid | 379.0 | 95.2 | 256 ms | 10.5 ms | 4.0 | 0 | 11,368 | 30.0 s |
| Sustained decode | 59K | 8 | Valid | 491.8 | 61.8 | 343 ms | 16.2 ms | 8.0 | 0 | 14,752 | 30.0 s |
| Sustained decode | 59K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.