Speed test matrix
Olmo-3.1-32B-Instruct-SFT on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF164 contexts (8K–59K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 59K, 64K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- olmo
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-allenai-vllm/bin/vllm" serve allenai/Olmo-3.1-32B-Instruct-SFT --served-model-name olmo --trust-remote-code --tensor-parallel-size 1 --max-model-len 65536 --gpu-memory-utilization 0.871691 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context8,414639 ms TTFT
16K context8,0591,314 ms TTFT
32K context7,9262,651 ms TTFT
59K context7,6165,068 ms TTFT
64K context7,5725,529 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 50.455 ms TTFT | 99.375 ms TTFT | 193.385 ms TTFT | 365.7123 ms TTFT | 666.1212 ms TTFT | 1,119.7351 ms TTFT | 1,641.7535 ms TTFT | Over capacity |
| 16K | 50.178 ms TTFT | 98.894 ms TTFT | 191.378 ms TTFT | 358.3188 ms TTFT | 640.0298 ms TTFT | 1,046.6333 ms TTFT | Over capacity | Over capacity |
| 32K | 49.788 ms TTFT | 97.7111 ms TTFT | 187.3158 ms TTFT | 345.1230 ms TTFT | 601.8444 ms TTFT | Over capacity | Over capacity | Over capacity |
| 59K | 49.0136 ms TTFT | 95.6176 ms TTFT | 181.2257 ms TTFT | 325.5441 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 8,414.0 | — | 639 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 8,059.0 | — | 1,314 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 7,926.0 | — | 2,651 ms | — | — | 1 | — | — |
| Prefill | 59K | 1 | Valid | 7,616.0 | — | 5,068 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,572.0 | — | 5,529 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 50.4 | 50.4 | 55 ms | 19.8 ms | 1.0 | 0 | 1,511 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 99.3 | 49.7 | 75 ms | 20.1 ms | 2.0 | 0 | 2,978 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 193.3 | 48.4 | 85 ms | 20.7 ms | 4.0 | 0 | 5,796 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 365.7 | 45.8 | 123 ms | 21.8 ms | 8.0 | 0 | 10,968 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 666.1 | 41.7 | 212 ms | 24.0 ms | 16.0 | 0 | 19,984 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,119.7 | 35.2 | 351 ms | 28.4 ms | 32.0 | 0 | 33,568 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,641.7 | 25.8 | 535 ms | 38.8 ms | 64.0 | 0 | 49,216 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 50.1 | 50.1 | 78 ms | 19.9 ms | 1.0 | 0 | 1,503 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 98.8 | 49.4 | 94 ms | 20.2 ms | 2.0 | 0 | 2,962 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 191.3 | 47.8 | 78 ms | 20.9 ms | 4.0 | 0 | 5,736 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 358.3 | 44.8 | 188 ms | 22.3 ms | 8.0 | 0 | 10,744 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 640.0 | 40.1 | 298 ms | 24.9 ms | 16.0 | 0 | 19,200 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,046.6 | 32.8 | 333 ms | 30.5 ms | 32.0 | 0 | 31,392 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 49.7 | 49.7 | 88 ms | 20.1 ms | 1.0 | 0 | 1,491 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 97.7 | 48.9 | 111 ms | 20.5 ms | 2.0 | 0 | 2,930 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 187.3 | 46.9 | 158 ms | 21.3 ms | 4.0 | 0 | 5,616 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 345.1 | 43.2 | 230 ms | 23.1 ms | 8.0 | 0 | 10,353 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 601.8 | 37.7 | 444 ms | 26.5 ms | 16.0 | 0 | 18,048 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 1 | Valid | 49.0 | 49.0 | 136 ms | 20.4 ms | 1.0 | 0 | 1,470 | 30.0 s |
| Sustained decode | 59K | 2 | Valid | 95.6 | 47.8 | 176 ms | 20.9 ms | 2.0 | 0 | 2,866 | 30.0 s |
| Sustained decode | 59K | 4 | Valid | 181.2 | 45.3 | 257 ms | 22.1 ms | 4.0 | 0 | 5,436 | 30.0 s |
| Sustained decode | 59K | 8 | Valid | 325.5 | 40.7 | 441 ms | 24.6 ms | 8.0 | 0 | 9,763 | 30.0 s |
| Sustained decode | 59K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.