Speed test matrix
granite-4.2-8b on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- granite-4.2-8b
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-ibm-granite-vllm-73a5831127a9/bin/vllm" serve ibm-granite/granite-4.2-8b --served-model-name granite-4.2-8b --tensor-parallel-size 1 --reasoning-parser granite_thinking_parser --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 131072 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --attention-backend FLASH_ATTN
Prefill
prompt tok/s by context length8K context17.8K461 ms TTFT
16K context15.4K1,050 ms TTFT
32K context12.2K2,643 ms TTFT
64K context8,5157,558 ms TTFT
123K context5,46022,621 ms TTFT
128K context5,32824,126 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 72.540 ms TTFT | 138.353 ms TTFT | 271.572 ms TTFT | 494.4116 ms TTFT | 825.1192 ms TTFT | 1,259.8280 ms TTFT | Over capacity | Over capacity |
| 16K | 68.556 ms TTFT | 128.383 ms TTFT | 246.5105 ms TTFT | 424.4157 ms TTFT | 645.3274 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 61.393 ms TTFT | 111.5122 ms TTFT | 207.4175 ms TTFT | 328.1280 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 50.4168 ms TTFT | 77.7218 ms TTFT | 148.9313 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 123K | 37.6221 ms TTFT | 53.3319 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 17.8K | — | 461 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 15.4K | — | 1,050 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 12.2K | — | 2,643 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,515.0 | — | 7,558 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 5,460.0 | — | 22,621 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,328.0 | — | 24,126 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 72.5 | 72.6 | 40 ms | 13.8 ms | 1.0 | 0 | 2,174 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 138.3 | 69.3 | 53 ms | 14.4 ms | 2.0 | 0 | 4,148 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 271.5 | 68.1 | 72 ms | 14.7 ms | 4.0 | 0 | 8,144 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 494.4 | 62.1 | 116 ms | 16.1 ms | 8.0 | 0 | 14,833 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 825.1 | 51.7 | 192 ms | 19.3 ms | 16.0 | 0 | 24,752 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,259.8 | 39.5 | 280 ms | 25.3 ms | 32.0 | 0 | 37,792 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 68.5 | 68.6 | 56 ms | 14.6 ms | 1.0 | 0 | 2,055 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 128.3 | 64.2 | 83 ms | 15.6 ms | 2.0 | 0 | 3,848 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 246.5 | 61.8 | 105 ms | 16.2 ms | 4.0 | 0 | 7,392 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 424.4 | 53.1 | 157 ms | 18.8 ms | 8.0 | 0 | 12,728 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 645.3 | 40.4 | 274 ms | 24.8 ms | 16.0 | 0 | 19,345 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 61.3 | 61.3 | 93 ms | 16.3 ms | 1.0 | 0 | 1,838 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 111.5 | 55.8 | 122 ms | 17.9 ms | 2.0 | 0 | 3,344 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 207.4 | 51.9 | 175 ms | 19.3 ms | 4.0 | 0 | 6,220 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 328.1 | 41.0 | 280 ms | 24.4 ms | 8.0 | 0 | 9,840 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 50.4 | 50.5 | 168 ms | 19.8 ms | 1.0 | 0 | 1,513 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 77.7 | 38.9 | 218 ms | 25.7 ms | 2.0 | 0 | 2,332 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 148.9 | 37.2 | 313 ms | 26.9 ms | 4.0 | 0 | 4,464 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 37.6 | 37.6 | 221 ms | 26.6 ms | 1.0 | 0 | 1,128 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 53.3 | 26.7 | 319 ms | 37.4 ms | 2.0 | 0 | 1,600 | 30.0 s |
| Sustained decode | 123K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.