Speed test matrix
Qwen3.5-397B-A17B-NVFP4-V2 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.1.dev1+g9fedd24fa · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev1+g9fedd24fa
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
9fedd24fa864
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.9
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-nvidia-vllm-pr-9fedd24fa864/bin/vllm" serve nvidia/Qwen3.5-397B-A17B-NVFP4-V2 --served-model-name qwen --trust-remote-code --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.91 --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3 --attention-config '{"disable_flashinfer_q_quantization":true}' --mamba-cache-dtype auto --mamba-ssm-cache-dtype float32 --enable-prefix-caching --mamba-cache-mode align --attention-backend FLASHINFER --linear-backend cutlass --moe-backend flashinfer_cutlass --quantization modelopt_mixedPrefill
prompt tok/s by context length8K context11.8K694 ms TTFT
16K context11.5K1,407 ms TTFT
32K context11.4K2,835 ms TTFT
64K context10.3K6,256 ms TTFT
128K context8,55715,039 ms TTFT
251K context6,42539,255 ms TTFT
256K context6,34940,512 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 102.5204 ms TTFT | 183.5304 ms TTFT | 327.9498 ms TTFT | 502.6892 ms TTFT | 787.41,696 ms TTFT | 1,093.93,337 ms TTFT | 1,621.06,776 ms TTFT | 2,271.514,142 ms TTFT |
| 16K | 101.7212 ms TTFT | 183.4302 ms TTFT | 325.8494 ms TTFT | 501.5883 ms TTFT | 782.51,659 ms TTFT | 1,084.63,254 ms TTFT | 1,587.36,581 ms TTFT | 2,185.613,698 ms TTFT |
| 32K | 101.1188 ms TTFT | 181.8257 ms TTFT | 324.3403 ms TTFT | 494.8683 ms TTFT | 770.21,269 ms TTFT | 1,052.22,476 ms TTFT | 1,503.55,034 ms TTFT | Over capacity |
| 64K | 99.5365 ms TTFT | 179.4503 ms TTFT | 317.4781 ms TTFT | 483.81,335 ms TTFT | 738.02,450 ms TTFT | 991.64,692 ms TTFT | Over capacity | Over capacity |
| 128K | 97.2420 ms TTFT | 173.8548 ms TTFT | 307.1746 ms TTFT | 457.41,186 ms TTFT | 679.82,066 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 90.6731 ms TTFT | 163.2942 ms TTFT | 286.61,338 ms TTFT | 418.72,177 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 11.8K | — | 694 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 11.5K | — | 1,407 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 11.4K | — | 2,835 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 10.3K | — | 6,256 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 8,557.0 | — | 15,039 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,425.0 | — | 39,255 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,349.0 | — | 40,512 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 102.5 | 102.5 | 204 ms | 9.8 ms | 1.0 | 0 | 3,075 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 183.5 | 91.5 | 304 ms | 10.9 ms | 2.0 | 0 | 5,504 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 327.9 | 81.4 | 498 ms | 12.3 ms | 4.0 | 0 | 9,835 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 502.6 | 61.9 | 892 ms | 16.1 ms | 8.0 | 0 | 15,072 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 787.4 | 47.7 | 1,696 ms | 21.0 ms | 16.0 | 0 | 23,616 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,093.9 | 32.1 | 3,337 ms | 31.1 ms | 32.0 | 0 | 32,800 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,621.0 | 22.4 | 6,776 ms | 44.6 ms | 64.0 | 0 | 48,576 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,271.5 | 14.6 | 14,142 ms | 68.4 ms | 128.0 | 0 | 68,096 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 101.7 | 101.7 | 212 ms | 9.8 ms | 1.0 | 0 | 3,051 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 183.4 | 91.5 | 302 ms | 10.9 ms | 2.0 | 0 | 5,502 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 325.8 | 80.9 | 494 ms | 12.4 ms | 4.0 | 0 | 9,772 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 501.5 | 61.8 | 883 ms | 16.2 ms | 8.0 | 0 | 15,040 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 782.5 | 47.5 | 1,659 ms | 21.1 ms | 16.0 | 0 | 23,472 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,084.6 | 31.9 | 3,254 ms | 31.4 ms | 32.0 | 0 | 32,512 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,587.3 | 22.1 | 6,581 ms | 45.3 ms | 64.0 | 0 | 47,616 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,185.6 | 14.2 | 13,698 ms | 70.3 ms | 128.0 | 0 | 65,536 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 101.1 | 101.1 | 188 ms | 9.9 ms | 1.0 | 0 | 3,032 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 181.8 | 90.9 | 257 ms | 11.0 ms | 2.0 | 0 | 5,454 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 324.3 | 80.7 | 403 ms | 12.4 ms | 4.0 | 0 | 9,724 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 494.8 | 61.4 | 683 ms | 16.3 ms | 8.0 | 0 | 14,840 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 770.2 | 47.1 | 1,269 ms | 21.2 ms | 16.0 | 0 | 23,104 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,052.2 | 31.6 | 2,476 ms | 31.6 ms | 32.0 | 0 | 31,552 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,503.5 | 21.8 | 5,034 ms | 46.0 ms | 64.0 | 0 | 45,056 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 99.5 | 99.6 | 365 ms | 10.0 ms | 1.0 | 0 | 2,986 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 179.4 | 89.4 | 503 ms | 11.2 ms | 2.0 | 0 | 5,382 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 317.4 | 78.7 | 781 ms | 12.7 ms | 4.0 | 0 | 9,520 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 483.8 | 59.2 | 1,335 ms | 16.9 ms | 8.0 | 0 | 14,512 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 738.0 | 44.0 | 2,450 ms | 22.7 ms | 16.0 | 0 | 22,128 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 991.6 | 28.3 | 4,692 ms | 35.4 ms | 32.0 | 0 | 29,728 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 97.2 | 97.3 | 420 ms | 10.3 ms | 1.0 | 0 | 2,915 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 173.8 | 86.8 | 548 ms | 11.5 ms | 2.0 | 0 | 5,214 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 307.1 | 76.3 | 746 ms | 13.1 ms | 4.0 | 0 | 9,212 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 457.4 | 56.3 | 1,186 ms | 17.8 ms | 8.0 | 0 | 13,720 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 679.8 | 41.2 | 2,066 ms | 24.3 ms | 16.0 | 0 | 20,384 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 90.6 | 90.7 | 731 ms | 11.0 ms | 1.0 | 0 | 2,718 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 163.2 | 81.5 | 942 ms | 12.3 ms | 2.0 | 0 | 4,896 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 286.6 | 71.0 | 1,338 ms | 14.1 ms | 4.0 | 0 | 8,596 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 418.7 | 51.3 | 2,177 ms | 19.5 ms | 8.0 | 0 | 12,560 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.