Speed test matrix
diffusiongemma-26B-A4B-it on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-google-vllm-73a5831127a9/bin/vllm" serve google/diffusiongemma-26B-A4B-it --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 262144 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-seqs 4 --attention-backend TRITON_ATTN --moe-backend triton
Prefill
prompt tok/s by context length8K context8,0801,015 ms TTFT
16K context10.7K1,522 ms TTFT
32K context9,0463,578 ms TTFT
64K context7,4948,615 ms TTFT
128K context4,99125,843 ms TTFT
251K context2,81689,764 ms TTFT
256K context2,77892,815 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 646.5580 ms TTFT | 641.5619 ms TTFT | 761.21,012 ms TTFT | Looped | 869.1Requests queued39,347 ms TTFT | Looped | Looped | Over capacity |
| 16K | 854.7504 ms TTFT | Looped | 1,469.41,316 ms TTFT | 1,132.9Requests queued14,319 ms TTFT | Looped | Looped | Over capacity | Over capacity |
| 32K | 537.6660 ms TTFT | Looped | Looped | 878.3Requests queued19,845 ms TTFT | Looped | Over capacity | Over capacity | Over capacity |
| 64K | Looped | 541.01,868 ms TTFT | 421.92,538 ms TTFT | Looped | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 169.92,159 ms TTFT | 245.73,413 ms TTFT | 241.25,820 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 114.211,834 ms TTFT | 108.517,643 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 8,080.0 | — | 1,015 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.7K | — | 1,522 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,046.0 | — | 3,578 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,494.0 | — | 8,615 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,991.0 | — | 25,843 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 2,816.0 | — | 89,764 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 2,778.0 | — | 92,815 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 646.5 | 410.4 | 580 ms | 2.4 ms | 1.0 | 4 | 16,384 | 25.3 s |
| Sustained decode | 8K | 2 | Valid | 641.5 | 246.4 | 619 ms | 4.4 ms | 2.0 | 5 | 18,944 | 29.5 s |
| Sustained decode | 8K | 4 | Valid | 761.2 | 223.9 | 1,012 ms | 4.5 ms | 4.0 | 4 | 22,784 | 29.9 s |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Requests queued | 869.1 | 190.1 | 39,347 ms | 5.3 ms | 4.0 | 18 | 25,856 | 29.7 s |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 854.7 | 777.9 | 504 ms | 1.7 ms | 1.0 | 6 | 22,784 | 26.7 s |
| Sustained decode | 16K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 4 | Valid | 1,469.4 | 332.0 | 1,316 ms | 3.0 ms | 4.0 | 10 | 43,776 | 29.8 s |
| Sustained decode | 16K | 8 | Requests queued | 1,132.9 | 221.4 | 14,319 ms | 4.6 ms | 4.0 | 24 | 33,536 | 29.6 s |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 537.6 | 425.9 | 660 ms | 2.4 ms | 1.0 | 4 | 16,128 | 30.0 s |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Requests queued | 878.3 | 201.0 | 19,845 ms | 5.0 ms | 4.0 | 16 | 25,856 | 29.4 s |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 2 | Valid | 541.0 | 237.0 | 1,868 ms | 4.2 ms | 1.9 | 3 | 15,872 | 29.3 s |
| Sustained decode | 64K | 4 | Valid | 421.9 | 127.6 | 2,538 ms | 7.8 ms | 4.0 | 2 | 11,776 | 27.9 s |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 169.9 | 141.0 | 2,159 ms | 8.4 ms | 1.0 | 1 | 4,352 | 25.6 s |
| Sustained decode | 128K | 2 | Valid | 245.7 | 71.9 | 3,413 ms | 13.9 ms | 2.0 | 1 | 5,632 | 22.9 s |
| Sustained decode | 128K | 4 | Valid | 241.2 | 103.4 | 5,820 ms | 9.7 ms | 4.0 | 0 | 5,376 | 22.3 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 114.2 | 107.9 | 11,834 ms | 9.3 ms | 1.0 | 0 | 2,816 | 24.7 s |
| Sustained decode | 251K | 2 | Valid | 108.5 | 57.8 | 17,643 ms | 26.1 ms | 2.0 | 0 | 3,072 | 28.3 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.