Speed test matrix
Qwen3.6-35B-A3B-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.6.post3.dev11207+g13e5dcf04 · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev11207+g13e5dcf04
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
13e5dcf04f08
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-nvidia-sglang-pr-13e5dcf04f082f8040f46d49/bin/sglang" serve --model-path nvidia/Qwen3.6-35B-A3B-NVFP4 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.79 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3 --mamba-ssm-dtype float32 --attention-backend flashinfer --fp4-gemm-backend marlin --moe-runner-backend marlin
Prefill
prompt tok/s by context length8K context33.9K241 ms TTFT
16K context31.8K507 ms TTFT
32K context27.2K1,181 ms TTFT
64K context20.7K3,097 ms TTFT
128K context13.9K9,203 ms TTFT
251K context8,41629,809 ms TTFT
256K context8,26630,953 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 293.434 ms TTFT | 427.951 ms TTFT | 762.466 ms TTFT | 1,209.596 ms TTFT | 1,845.4156 ms TTFT | 2,606.6303 ms TTFT | 3,521.6624 ms TTFT | 3,544.8Requests queued1,228 ms TTFT |
| 16K | 284.941 ms TTFT | 422.365 ms TTFT | 757.890 ms TTFT | 1,171.7156 ms TTFT | 1,725.9295 ms TTFT | 2,494.7566 ms TTFT | 3,146.81,171 ms TTFT | 3,258.4Requests queued2,194 ms TTFT |
| 32K | 273.359 ms TTFT | 410.176 ms TTFT | 719.5111 ms TTFT | 1,113.6185 ms TTFT | 1,622.3567 ms TTFT | 2,198.81,143 ms TTFT | 2,589.52,288 ms TTFT | Over capacity |
| 64K | 254.3102 ms TTFT | 389.1159 ms TTFT | 670.8248 ms TTFT | 995.1394 ms TTFT | 1,399.71,234 ms TTFT | 1,758.92,569 ms TTFT | Over capacity | Over capacity |
| 128K | 225.7194 ms TTFT | 351.0380 ms TTFT | 592.4566 ms TTFT | 839.21,495 ms TTFT | 1,071.93,058 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 182.7461 ms TTFT | 292.9703 ms TTFT | 469.01,732 ms TTFT | 614.43,527 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 33.9K | — | 241 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 31.8K | — | 507 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 27.2K | — | 1,181 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 20.7K | — | 3,097 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 13.9K | — | 9,203 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 8,416.0 | — | 29,809 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,266.0 | — | 30,953 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 293.4 | 294.2 | 34 ms | 3.4 ms | 1.0 | 2 | 8,801 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 427.9 | 214.5 | 51 ms | 4.7 ms | 2.0 | 2 | 12,836 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 762.4 | 191.9 | 66 ms | 5.2 ms | 4.0 | 4 | 22,870 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,209.5 | 153.5 | 96 ms | 6.5 ms | 8.0 | 8 | 36,284 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,845.4 | 116.3 | 156 ms | 8.6 ms | 16.0 | 0 | 55,360 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,606.6 | 82.6 | 303 ms | 12.1 ms | 32.0 | 0 | 78,176 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,521.6 | 55.9 | 624 ms | 17.9 ms | 64.0 | 0 | 105,664 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 3,544.8 | 46.3 | 1,228 ms | 21.6 ms | 82.0 | 0 | 106,355 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 284.9 | 285.8 | 41 ms | 3.5 ms | 1.0 | 2 | 8,547 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 422.3 | 211.8 | 65 ms | 4.7 ms | 2.0 | 2 | 12,668 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 757.8 | 190.0 | 90 ms | 5.3 ms | 4.0 | 4 | 22,733 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,171.7 | 149.1 | 156 ms | 6.7 ms | 8.0 | 8 | 35,150 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,725.9 | 109.1 | 295 ms | 9.2 ms | 16.0 | 0 | 51,775 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,494.7 | 78.7 | 566 ms | 12.7 ms | 32.0 | 0 | 74,816 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,146.8 | 49.8 | 1,171 ms | 20.1 ms | 64.0 | 0 | 94,400 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 3,258.4 | 42.5 | 2,194 ms | 23.5 ms | 82.0 | 0 | 97,744 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 273.3 | 274.3 | 59 ms | 3.6 ms | 1.0 | 2 | 8,198 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 410.1 | 205.7 | 76 ms | 4.9 ms | 2.0 | 2 | 12,303 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 719.5 | 181.3 | 111 ms | 5.5 ms | 4.0 | 4 | 21,583 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,113.6 | 140.1 | 185 ms | 7.1 ms | 8.0 | 8 | 33,408 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,622.3 | 102.2 | 567 ms | 9.8 ms | 16.0 | 0 | 48,656 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,198.8 | 69.6 | 1,143 ms | 14.4 ms | 32.0 | 0 | 65,952 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,589.5 | 41.6 | 2,288 ms | 24.1 ms | 64.0 | 0 | 77,632 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 254.3 | 256.1 | 102 ms | 3.9 ms | 1.0 | 2 | 7,628 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 389.1 | 195.7 | 159 ms | 5.1 ms | 2.0 | 2 | 11,672 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 670.8 | 169.5 | 248 ms | 5.9 ms | 3.9 | 4 | 20,122 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 995.1 | 126.1 | 394 ms | 7.9 ms | 7.9 | 8 | 29,860 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,399.7 | 89.0 | 1,234 ms | 11.2 ms | 16.0 | 0 | 41,984 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,758.9 | 56.9 | 2,569 ms | 17.6 ms | 32.0 | 0 | 52,768 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 225.7 | 227.2 | 194 ms | 4.4 ms | 1.0 | 1 | 6,772 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 351.0 | 177.6 | 380 ms | 5.6 ms | 2.0 | 2 | 10,529 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 592.4 | 150.6 | 566 ms | 6.6 ms | 4.0 | 4 | 17,769 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 839.2 | 107.1 | 1,495 ms | 9.3 ms | 8.0 | 0 | 25,175 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,071.9 | 70.2 | 3,058 ms | 14.2 ms | 16.0 | 0 | 32,144 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 182.7 | 185.7 | 461 ms | 5.4 ms | 1.0 | 1 | 5,480 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 292.9 | 149.8 | 703 ms | 6.7 ms | 2.0 | 2 | 8,785 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 469.0 | 119.7 | 1,732 ms | 8.4 ms | 4.0 | 1 | 13,971 | 29.8 s |
| Sustained decode | 251K | 8 | Valid | 614.4 | 80.5 | 3,527 ms | 12.4 ms | 8.0 | 0 | 18,432 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.