Speed test matrix
Olmo-3.1-32B-Instruct on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev151+g73a78e6f1 · BF164 contexts (8K–59K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 59K, 64K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev151+g73a78e6f1
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a78e6f1f38
Run
- Served model
- olmo
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-allenai-vllm-73a78e6f1f38/bin/vllm" serve allenai/Olmo-3.1-32B-Instruct --served-model-name olmo --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser olmo3 --max-model-len 65536 --gpu-memory-utilization 0.84 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN
Prefill
prompt tok/s by context length8K context5,618963 ms TTFT
16K context5,4051,965 ms TTFT
32K context5,2404,016 ms TTFT
59K context4,9767,764 ms TTFT
64K context4,9428,478 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 19.869 ms TTFT | 40.894 ms TTFT | 80.7150 ms TTFT | 146.6195 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 16K | 19.778 ms TTFT | 40.5103 ms TTFT | 79.9162 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 19.596 ms TTFT | 40.0153 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 59K | 19.2131 ms TTFT | 39.0191 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 5,618.0 | — | 963 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 5,405.0 | — | 1,965 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,240.0 | — | 4,016 ms | — | — | 1 | — | — |
| Prefill | 59K | 1 | Valid | 4,976.0 | — | 7,764 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 4,942.0 | — | 8,478 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 19.8 | 19.8 | 69 ms | 50.6 ms | 1.0 | 0 | 594 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 40.8 | 20.3 | 94 ms | 49.1 ms | 2.0 | 0 | 1,222 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 80.7 | 20.1 | 150 ms | 49.6 ms | 4.0 | 0 | 2,420 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 146.6 | 18.3 | 195 ms | 54.6 ms | 8.0 | 0 | 4,392 | 30.0 s |
| Sustained decode | 8K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 19.7 | 19.7 | 78 ms | 50.8 ms | 1.0 | 0 | 591 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 40.5 | 20.2 | 103 ms | 49.4 ms | 2.0 | 0 | 1,212 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 79.9 | 20.0 | 162 ms | 50.0 ms | 4.0 | 0 | 2,396 | 30.0 s |
| Sustained decode | 16K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 19.5 | 19.5 | 96 ms | 51.3 ms | 1.0 | 0 | 584 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 40.0 | 20.0 | 153 ms | 50.1 ms | 2.0 | 0 | 1,198 | 30.0 s |
| Sustained decode | 32K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 1 | Valid | 19.2 | 19.2 | 131 ms | 52.0 ms | 1.0 | 0 | 576 | 30.0 s |
| Sustained decode | 59K | 2 | Valid | 39.0 | 19.5 | 191 ms | 51.3 ms | 2.0 | 0 | 1,168 | 30.0 s |
| Sustained decode | 59K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 59K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.