Speed test matrix
granite-4.2-3b on H200 NVL 141GB
vllm · 0.25.0 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.25.0
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- granite-4.2-3b
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-ibm-vllm/bin/vllm" serve ibm-granite/granite-4.2-3b --served-model-name granite-4.2-3b --tensor-parallel-size 1 --reasoning-parser granite_thinking_parser --reasoning-parser-plugin /workspace/compute/recipes/nvidia-h200/ibm-granite/plugin/granite_thinking_parser.py --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 131072 --gpu-memory-utilization 0.866765 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16
Prefill
prompt tok/s by context length8K context49.7K165 ms TTFT
16K context45.2K359 ms TTFT
32K context31.7K1,018 ms TTFT
64K context21.8K2,948 ms TTFT
123K context13.5K9,143 ms TTFT
128K context13.3K9,644 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 271.355 ms TTFT | 510.064 ms TTFT | 929.286 ms TTFT | 1,552.9133 ms TTFT | 2,414.6155 ms TTFT | 3,266.1284 ms TTFT | 4,193.1478 ms TTFT | Over capacity |
| 16K | 256.692 ms TTFT | 473.387 ms TTFT | 815.4135 ms TTFT | 1,250.9168 ms TTFT | 1,786.2258 ms TTFT | 2,294.8462 ms TTFT | 2,689.8645 ms TTFT | Over capacity |
| 32K | 235.8103 ms TTFT | 414.5160 ms TTFT | 647.1191 ms TTFT | 910.9314 ms TTFT | 1,203.4511 ms TTFT | 1,408.1758 ms TTFT | Over capacity | Over capacity |
| 64K | 204.2151 ms TTFT | 326.6232 ms TTFT | 468.0340 ms TTFT | 601.7526 ms TTFT | 720.2921 ms TTFT | Over capacity | Over capacity | Over capacity |
| 123K | 161.1296 ms TTFT | 239.0326 ms TTFT | 313.7535 ms TTFT | 369.6890 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 49.7K | — | 165 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 45.2K | — | 359 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 31.7K | — | 1,018 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 21.8K | — | 2,948 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 13.5K | — | 9,143 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 13.3K | — | 9,644 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 271.3 | 272.4 | 55 ms | 3.7 ms | 1.0 | 2 | 8,139 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 510.0 | 256.5 | 64 ms | 3.9 ms | 2.0 | 4 | 15,298 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 929.2 | 233.9 | 86 ms | 4.3 ms | 4.0 | 4 | 27,873 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,552.9 | 196.8 | 133 ms | 5.1 ms | 8.0 | 8 | 46,584 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,414.6 | 156.3 | 155 ms | 6.4 ms | 16.0 | 16 | 72,440 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,266.1 | 103.6 | 284 ms | 9.7 ms | 32.0 | 0 | 97,985 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,193.1 | 66.4 | 478 ms | 15.1 ms | 64.0 | 0 | 125,824 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 256.6 | 258.5 | 92 ms | 3.9 ms | 1.0 | 2 | 7,702 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 473.3 | 238.6 | 87 ms | 4.2 ms | 2.0 | 4 | 14,198 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 815.4 | 205.5 | 135 ms | 4.9 ms | 4.0 | 4 | 24,463 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,250.9 | 159.2 | 168 ms | 6.3 ms | 8.0 | 8 | 37,525 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,786.2 | 112.5 | 258 ms | 8.9 ms | 16.0 | 0 | 53,584 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,294.8 | 72.0 | 462 ms | 13.9 ms | 32.0 | 0 | 68,832 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,689.8 | 42.2 | 645 ms | 23.7 ms | 64.0 | 0 | 80,704 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 235.8 | 237.1 | 103 ms | 4.2 ms | 1.0 | 1 | 7,075 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 414.5 | 209.0 | 160 ms | 4.8 ms | 1.9 | 2 | 12,436 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 647.1 | 164.1 | 191 ms | 6.1 ms | 4.0 | 4 | 19,414 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 910.9 | 114.3 | 314 ms | 8.8 ms | 8.0 | 0 | 27,320 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,203.4 | 75.4 | 511 ms | 13.3 ms | 16.0 | 0 | 36,095 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,408.1 | 44.0 | 758 ms | 22.7 ms | 32.0 | 0 | 42,240 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 204.2 | 205.6 | 151 ms | 4.9 ms | 1.0 | 1 | 6,124 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 326.6 | 165.5 | 232 ms | 6.0 ms | 2.0 | 2 | 9,799 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 468.0 | 117.4 | 340 ms | 8.5 ms | 4.0 | 0 | 14,048 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 601.7 | 75.4 | 526 ms | 13.3 ms | 8.0 | 0 | 18,056 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 720.2 | 45.0 | 921 ms | 22.2 ms | 16.0 | 0 | 21,600 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 161.1 | 163.6 | 296 ms | 6.1 ms | 1.0 | 1 | 4,832 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 239.0 | 121.0 | 326 ms | 8.3 ms | 2.0 | 2 | 7,168 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 313.7 | 78.6 | 535 ms | 12.7 ms | 4.0 | 0 | 9,408 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 369.6 | 46.2 | 890 ms | 21.6 ms | 8.0 | 0 | 11,088 | 30.0 s |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.