Speed test matrix
Qwen3.8-27B-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-redhatai-vllm-73a5831127a9d2b8/bin/vllm" serve RedHatAI/Qwen3.8-27B-NVFP4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8
Prefill
prompt tok/s by context length8K context12.9K640 ms TTFT
16K context12.1K1,347 ms TTFT
32K context10.6K3,054 ms TTFT
64K context8,2027,860 ms TTFT
128K context5,70022,586 ms TTFT
251K context3,59070,271 ms TTFT
256K context3,53572,769 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 56.868 ms TTFT | 103.997 ms TTFT | 201.5177 ms TTFT | 403.5279 ms TTFT | 715.0533 ms TTFT | 1,185.81,053 ms TTFT | 1,732.31,747 ms TTFT | 2,348.33,115 ms TTFT |
| 16K | 56.2111 ms TTFT | 103.1187 ms TTFT | 199.2289 ms TTFT | 393.7470 ms TTFT | 679.4869 ms TTFT | 1,089.31,541 ms TTFT | 1,533.12,538 ms TTFT | Over capacity |
| 32K | 55.2179 ms TTFT | 101.2281 ms TTFT | 195.0478 ms TTFT | 375.91,023 ms TTFT | 617.41,861 ms TTFT | 937.43,128 ms TTFT | Over capacity | Over capacity |
| 64K | 53.1155 ms TTFT | 97.8203 ms TTFT | 187.4290 ms TTFT | 346.6441 ms TTFT | 526.2758 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 49.4287 ms TTFT | 91.7382 ms TTFT | 172.9644 ms TTFT | 297.8869 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 43.31,193 ms TTFT | 81.61,722 ms TTFT | 150.62,682 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 12.9K | — | 640 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 12.1K | — | 1,347 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10.6K | — | 3,054 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,202.0 | — | 7,860 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,700.0 | — | 22,586 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,590.0 | — | 70,271 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,535.0 | — | 72,769 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 56.8 | 56.7 | 68 ms | 17.6 ms | 1.0 | 0 | 1,702 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 103.9 | 51.9 | 97 ms | 19.3 ms | 2.0 | 0 | 3,118 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 201.5 | 50.4 | 177 ms | 19.8 ms | 4.0 | 0 | 6,044 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 403.5 | 50.4 | 279 ms | 19.8 ms | 8.0 | 0 | 12,104 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 715.0 | 44.7 | 533 ms | 22.4 ms | 16.0 | 0 | 21,440 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,185.8 | 37.0 | 1,053 ms | 27.0 ms | 32.0 | 0 | 35,552 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,732.3 | 26.8 | 1,747 ms | 37.3 ms | 64.0 | 0 | 51,968 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,348.3 | 17.8 | 3,115 ms | 56.3 ms | 128.0 | 0 | 70,450 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 56.2 | 56.2 | 111 ms | 17.8 ms | 1.0 | 0 | 1,685 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 103.1 | 51.5 | 187 ms | 19.4 ms | 2.0 | 0 | 3,092 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 199.2 | 49.8 | 289 ms | 20.1 ms | 4.0 | 0 | 5,976 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 393.7 | 49.2 | 470 ms | 20.3 ms | 8.0 | 0 | 11,808 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 679.4 | 42.4 | 869 ms | 23.6 ms | 16.0 | 0 | 20,368 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,089.3 | 33.8 | 1,541 ms | 29.6 ms | 32.0 | 0 | 32,672 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,533.1 | 23.2 | 2,538 ms | 43.0 ms | 64.0 | 0 | 45,952 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 55.2 | 55.2 | 179 ms | 18.1 ms | 1.0 | 0 | 1,656 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 101.2 | 50.6 | 281 ms | 19.8 ms | 2.0 | 0 | 3,036 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 195.0 | 48.7 | 478 ms | 20.5 ms | 4.0 | 0 | 5,848 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 375.9 | 47.1 | 1,023 ms | 21.3 ms | 8.0 | 0 | 11,278 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 617.4 | 38.5 | 1,861 ms | 25.9 ms | 16.0 | 0 | 18,512 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 937.4 | 28.7 | 3,128 ms | 34.8 ms | 32.0 | 0 | 28,096 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 53.1 | 53.1 | 155 ms | 18.8 ms | 1.0 | 0 | 1,592 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 97.8 | 48.9 | 203 ms | 20.4 ms | 2.0 | 0 | 2,932 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 187.4 | 46.8 | 290 ms | 21.4 ms | 4.0 | 0 | 5,620 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 346.6 | 43.2 | 441 ms | 23.1 ms | 8.0 | 0 | 10,392 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 526.2 | 32.7 | 758 ms | 30.6 ms | 16.0 | 0 | 15,776 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 49.4 | 49.4 | 287 ms | 20.2 ms | 1.0 | 0 | 1,480 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 91.7 | 45.8 | 382 ms | 21.8 ms | 2.0 | 0 | 2,748 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 172.9 | 43.1 | 644 ms | 23.2 ms | 4.0 | 0 | 5,184 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 297.8 | 37.1 | 869 ms | 27.0 ms | 8.0 | 0 | 8,928 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 43.3 | 43.4 | 1,193 ms | 23.0 ms | 1.0 | 0 | 1,300 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 81.6 | 40.6 | 1,722 ms | 24.6 ms | 2.0 | 0 | 2,448 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 150.6 | 37.0 | 2,682 ms | 27.1 ms | 4.0 | 0 | 4,518 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.