Speed test matrix
Qwen3.8-Flash-Next-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.28.1rc1.dev562+g9c2d21046 · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.28.1rc1.dev562+g9c2d21046
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
9c2d21046bb3
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-nvidia-qwen38-vllm-9c2d21046/bin/vllm" serve nvidia/Qwen3.8-Flash-Next-NVFP4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.77 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --enable-expert-parallel --kv-cache-dtype bfloat16 --mamba-cache-dtype auto --mamba-ssm-cache-dtype float32 --enable-prefix-caching --mamba-cache-mode align --attention-config '{"indexer_kv_dtype":"bf16"}' --no-enable-flashinfer-autotune --moe-backend flashinfer_cutlass --speculative-config '{"method":"mtp","num_speculative_tokens":1,"moe_backend":"triton"}' --quantization modeloptPrefill
prompt tok/s by context length8K context16.3K504 ms TTFT
16K context16.7K966 ms TTFT
32K context16.9K1,902 ms TTFT
64K context16.9K3,803 ms TTFT
128K context16.5K7,745 ms TTFT
251K context15.8K15,863 ms TTFT
256K context15.6K16,361 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 142.8101 ms TTFT | 266.2120 ms TTFT | 494.6216 ms TTFT | 841.51,123 ms TTFT | 1,414.2582 ms TTFT | 2,044.51,342 ms TTFT | 3,209.02,280 ms TTFT | 4,428.54,181 ms TTFT |
| 16K | 141.2114 ms TTFT | 269.0159 ms TTFT | 492.4181 ms TTFT | 851.4438 ms TTFT | 1,442.21,016 ms TTFT | 2,049.61,482 ms TTFT | 3,192.52,407 ms TTFT | Over capacity |
| 32K | 143.4143 ms TTFT | 271.2154 ms TTFT | 492.1191 ms TTFT | 842.2535 ms TTFT | 1,418.4782 ms TTFT | 2,034.71,376 ms TTFT | 3,166.52,244 ms TTFT | Over capacity |
| 64K | 140.1202 ms TTFT | 273.3218 ms TTFT | 497.4217 ms TTFT | 842.9605 ms TTFT | 1,403.3955 ms TTFT | 2,015.81,665 ms TTFT | Over capacity | Over capacity |
| 128K | 140.6340 ms TTFT | 265.1371 ms TTFT | 491.0462 ms TTFT | 837.01,030 ms TTFT | 1,395.21,822 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 140.3708 ms TTFT | 261.1918 ms TTFT | 482.1941 ms TTFT | Looped | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 16.3K | — | 504 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 16.7K | — | 966 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 16.9K | — | 1,902 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 16.9K | — | 3,803 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 16.5K | — | 7,745 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 15.8K | — | 15,863 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 15.6K | — | 16,361 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 142.8 | 145.2 | 101 ms | 6.9 ms | 1.0 | 1 | 4,284 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 266.2 | 133.7 | 120 ms | 7.5 ms | 2.0 | 2 | 7,984 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 494.6 | 119.7 | 216 ms | 8.4 ms | 4.0 | 3 | 14,837 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 841.5 | 95.5 | 1,123 ms | 10.5 ms | 8.0 | 0 | 25,247 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,414.2 | 83.9 | 582 ms | 11.9 ms | 16.0 | 0 | 42,422 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,044.5 | 61.0 | 1,342 ms | 16.4 ms | 32.0 | 0 | 61,331 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,209.0 | 45.7 | 2,280 ms | 21.9 ms | 64.0 | 0 | 96,211 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,428.5 | 32.1 | 4,181 ms | 31.1 ms | 128.0 | 0 | 132,784 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 141.2 | 141.4 | 114 ms | 7.1 ms | 1.0 | 1 | 4,236 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 269.0 | 135.0 | 159 ms | 7.4 ms | 2.0 | 2 | 8,068 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 492.4 | 123.8 | 181 ms | 8.1 ms | 4.0 | 4 | 14,772 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 851.4 | 106.5 | 438 ms | 9.4 ms | 8.0 | 0 | 25,534 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,442.2 | 89.2 | 1,016 ms | 11.2 ms | 16.0 | 0 | 43,251 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,049.6 | 63.1 | 1,482 ms | 15.8 ms | 32.0 | 0 | 61,462 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,192.5 | 48.1 | 2,407 ms | 20.8 ms | 64.0 | 0 | 95,760 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 143.4 | 140.9 | 143 ms | 7.1 ms | 1.0 | 1 | 4,302 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 271.2 | 136.0 | 154 ms | 7.4 ms | 2.0 | 2 | 8,135 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 492.1 | 123.8 | 191 ms | 8.1 ms | 4.0 | 4 | 14,759 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 842.2 | 104.7 | 535 ms | 9.5 ms | 8.0 | 0 | 25,267 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,418.4 | 87.8 | 782 ms | 11.4 ms | 16.0 | 0 | 42,535 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,034.7 | 62.5 | 1,376 ms | 16.0 ms | 32.0 | 0 | 61,035 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,166.5 | 47.8 | 2,244 ms | 20.9 ms | 64.0 | 0 | 94,968 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 140.1 | 141.4 | 202 ms | 7.1 ms | 1.0 | 1 | 4,203 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 273.3 | 136.6 | 218 ms | 7.3 ms | 2.0 | 2 | 8,196 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 497.4 | 123.4 | 217 ms | 8.1 ms | 4.0 | 4 | 14,919 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 842.9 | 104.2 | 605 ms | 9.6 ms | 8.0 | 0 | 25,277 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,403.3 | 86.8 | 955 ms | 11.5 ms | 16.0 | 0 | 42,095 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,015.8 | 61.5 | 1,665 ms | 16.3 ms | 32.0 | 0 | 60,460 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 140.6 | 141.4 | 340 ms | 7.1 ms | 1.0 | 1 | 4,217 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 265.1 | 134.4 | 371 ms | 7.4 ms | 2.0 | 2 | 7,950 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 491.0 | 122.5 | 462 ms | 8.2 ms | 4.0 | 4 | 14,714 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 837.0 | 103.9 | 1,030 ms | 9.6 ms | 8.0 | 0 | 25,110 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,395.2 | 85.5 | 1,822 ms | 11.7 ms | 16.0 | 0 | 41,844 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 140.3 | 143.8 | 708 ms | 7.0 ms | 1.0 | 1 | 4,208 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 261.1 | 132.9 | 918 ms | 7.5 ms | 1.9 | 2 | 7,830 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 482.1 | 122.0 | 941 ms | 8.2 ms | 4.0 | 3 | 14,458 | 30.0 s |
| Sustained decode | 251K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.