Speed test matrix
Qwen3.5-397B-A17B-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.1.dev1+g9fedd24fa · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev1+g9fedd24fa
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
9fedd24fa864
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.9
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-nvidia-vllm-pr-9fedd24fa864/bin/vllm" serve nvidia/Qwen3.5-397B-A17B-NVFP4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.91 --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3 --attention-config '{"disable_flashinfer_q_quantization":true}' --mamba-cache-dtype auto --mamba-ssm-cache-dtype float32 --enable-prefix-caching --mamba-cache-mode align --attention-backend FLASHINFER --moe-backend flashinfer_cutlass --quantization modelopt_fp4Prefill
prompt tok/s by context length8K context11.1K736 ms TTFT
16K context10.9K1,491 ms TTFT
32K context10.8K2,999 ms TTFT
64K context9,7526,606 ms TTFT
128K context8,18015,733 ms TTFT
251K context6,22440,523 ms TTFT
256K context6,14841,837 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 92.2214 ms TTFT | 164.7322 ms TTFT | 285.4529 ms TTFT | 481.1946 ms TTFT | 746.81,795 ms TTFT | 1,031.63,528 ms TTFT | 1,533.57,167 ms TTFT | 2,214.514,965 ms TTFT |
| 16K | 92.0221 ms TTFT | 164.3348 ms TTFT | 284.6522 ms TTFT | 476.0928 ms TTFT | 740.81,749 ms TTFT | 1,021.43,427 ms TTFT | 1,496.26,941 ms TTFT | 2,138.914,393 ms TTFT |
| 32K | 91.5194 ms TTFT | 163.2295 ms TTFT | 282.3412 ms TTFT | 472.7710 ms TTFT | 728.01,320 ms TTFT | 992.02,560 ms TTFT | 1,435.55,214 ms TTFT | Over capacity |
| 64K | 90.1372 ms TTFT | 161.2543 ms TTFT | 277.5805 ms TTFT | 459.11,377 ms TTFT | 700.12,519 ms TTFT | 943.24,846 ms TTFT | Over capacity | Over capacity |
| 128K | 87.5416 ms TTFT | 156.3572 ms TTFT | 269.4818 ms TTFT | 437.81,293 ms TTFT | 649.12,246 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 82.6727 ms TTFT | 148.61,006 ms TTFT | 254.51,354 ms TTFT | 402.42,186 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 11.1K | — | 736 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.9K | — | 1,491 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10.8K | — | 2,999 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,752.0 | — | 6,606 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 8,180.0 | — | 15,733 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,224.0 | — | 40,523 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,148.0 | — | 41,837 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 92.2 | 92.2 | 214 ms | 10.8 ms | 1.0 | 0 | 2,767 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 164.7 | 82.1 | 322 ms | 12.2 ms | 2.0 | 0 | 4,938 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 285.4 | 70.9 | 529 ms | 14.1 ms | 4.0 | 0 | 8,560 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 481.1 | 59.2 | 946 ms | 16.9 ms | 8.0 | 0 | 14,432 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 746.8 | 45.1 | 1,795 ms | 22.2 ms | 16.0 | 0 | 22,400 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,031.6 | 30.1 | 3,528 ms | 33.2 ms | 32.0 | 0 | 30,944 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,533.5 | 21.1 | 7,167 ms | 47.4 ms | 64.0 | 0 | 45,952 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,214.5 | 14.0 | 14,965 ms | 71.3 ms | 128.0 | 0 | 66,432 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 92.0 | 92.0 | 221 ms | 10.9 ms | 1.0 | 0 | 2,761 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 164.3 | 81.9 | 348 ms | 12.2 ms | 2.0 | 0 | 4,928 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 284.6 | 70.7 | 522 ms | 14.2 ms | 4.0 | 0 | 8,536 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 476.0 | 58.7 | 928 ms | 17.0 ms | 8.0 | 0 | 14,272 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 740.8 | 44.8 | 1,749 ms | 22.3 ms | 16.0 | 0 | 22,224 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,021.4 | 29.9 | 3,427 ms | 33.4 ms | 32.0 | 0 | 30,624 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,496.2 | 20.7 | 6,941 ms | 48.3 ms | 64.0 | 0 | 44,864 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,138.9 | 13.8 | 14,393 ms | 72.7 ms | 128.0 | 0 | 64,128 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 91.5 | 91.5 | 194 ms | 10.9 ms | 1.0 | 0 | 2,745 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 163.2 | 81.5 | 295 ms | 12.3 ms | 2.0 | 0 | 4,894 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 282.3 | 70.3 | 412 ms | 14.2 ms | 4.0 | 0 | 8,468 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 472.7 | 58.5 | 710 ms | 17.1 ms | 8.0 | 0 | 14,176 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 728.0 | 44.6 | 1,320 ms | 22.4 ms | 16.0 | 0 | 21,840 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 992.0 | 29.8 | 2,560 ms | 33.6 ms | 32.0 | 0 | 29,760 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,435.5 | 20.8 | 5,214 ms | 48.2 ms | 64.0 | 0 | 43,008 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 90.1 | 90.2 | 372 ms | 11.1 ms | 1.0 | 0 | 2,704 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 161.2 | 80.3 | 543 ms | 12.4 ms | 2.0 | 0 | 4,836 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 277.5 | 68.7 | 805 ms | 14.6 ms | 4.0 | 0 | 8,324 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 459.1 | 56.1 | 1,377 ms | 17.8 ms | 8.0 | 0 | 13,768 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 700.1 | 41.7 | 2,519 ms | 24.0 ms | 16.0 | 0 | 20,992 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 943.2 | 26.8 | 4,846 ms | 37.3 ms | 32.0 | 0 | 28,288 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 87.5 | 87.6 | 416 ms | 11.4 ms | 1.0 | 0 | 2,625 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 156.3 | 78.0 | 572 ms | 12.8 ms | 2.0 | 0 | 4,688 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 269.4 | 66.9 | 818 ms | 14.9 ms | 4.0 | 0 | 8,080 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 437.8 | 53.9 | 1,293 ms | 18.6 ms | 8.0 | 0 | 13,128 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 649.1 | 39.2 | 2,246 ms | 25.5 ms | 16.0 | 0 | 19,472 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 82.6 | 82.7 | 727 ms | 12.1 ms | 1.0 | 0 | 2,477 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 148.6 | 74.2 | 1,006 ms | 13.5 ms | 2.0 | 0 | 4,458 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 254.5 | 63.1 | 1,354 ms | 15.8 ms | 4.0 | 0 | 7,632 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 402.4 | 49.3 | 2,186 ms | 20.3 ms | 8.0 | 0 | 12,064 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.