Speed test matrix
Qwen3.6-27B-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.6.post3.dev11207+g13e5dcf04 · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev11207+g13e5dcf04
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
13e5dcf04f08
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-nvidia-sglang-pr-13e5dcf04f082f8040f46d49/bin/sglang" serve --model-path nvidia/Qwen3.6-27B-NVFP4 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.78 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3 --mamba-ssm-dtype float32 --attention-backend flashinfer --fp4-gemm-backend marlin
Prefill
prompt tok/s by context length8K context6,9401,174 ms TTFT
16K context6,6042,442 ms TTFT
32K context6,0985,266 ms TTFT
64K context5,17512,382 ms TTFT
128K context3,98032,168 ms TTFT
251K context2,72991,917 ms TTFT
256K context2,70094,765 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 68.441 ms TTFT | 127.770 ms TTFT | 245.472 ms TTFT | 470.7121 ms TTFT | 790.9175 ms TTFT | 1,187.1313 ms TTFT | 1,164.7Requests queued609 ms TTFT | Over capacity |
| 16K | 67.41,275 ms TTFT | 126.282 ms TTFT | 240.3121 ms TTFT | 453.2168 ms TTFT | 741.6285 ms TTFT | 1,053.3574 ms TTFT | Over capacity | Over capacity |
| 32K | 65.867 ms TTFT | 123.199 ms TTFT | 231.5164 ms TTFT | 423.5295 ms TTFT | 657.7578 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 62.8106 ms TTFT | 117.4178 ms TTFT | 216.1319 ms TTFT | 372.7642 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 57.4215 ms TTFT | 106.8387 ms TTFT | 186.9777 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 49.3487 ms TTFT | 89.9870 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,940.0 | — | 1,174 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,604.0 | — | 2,442 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,098.0 | — | 5,266 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,175.0 | — | 12,382 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 3,980.0 | — | 32,168 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 2,729.0 | — | 91,917 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 2,700.0 | — | 94,765 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 68.4 | 68.4 | 41 ms | 14.6 ms | 1.0 | 0 | 2,052 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 127.7 | 63.9 | 70 ms | 15.6 ms | 2.0 | 0 | 3,831 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 245.4 | 61.4 | 72 ms | 16.3 ms | 4.0 | 0 | 7,359 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 470.7 | 58.9 | 121 ms | 17.0 ms | 8.0 | 0 | 14,120 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 790.9 | 49.4 | 175 ms | 20.2 ms | 16.0 | 0 | 23,712 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,187.1 | 37.1 | 313 ms | 27.0 ms | 32.0 | 0 | 35,584 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 1,164.7 | 35.3 | 609 ms | 28.3 ms | 34.0 | 0 | 34,918 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 67.4 | 67.5 | 1,275 ms | 14.8 ms | 1.0 | 0 | 2,023 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 126.2 | 63.1 | 82 ms | 15.8 ms | 2.0 | 0 | 3,786 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 240.3 | 60.1 | 121 ms | 16.6 ms | 4.0 | 0 | 7,208 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 453.2 | 56.6 | 168 ms | 17.7 ms | 8.0 | 0 | 13,592 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 741.6 | 46.3 | 285 ms | 21.6 ms | 16.0 | 0 | 22,240 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,053.3 | 32.9 | 574 ms | 30.4 ms | 32.0 | 0 | 31,584 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 65.8 | 65.8 | 67 ms | 15.2 ms | 1.0 | 0 | 1,973 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 123.1 | 61.5 | 99 ms | 16.3 ms | 2.0 | 0 | 3,692 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 231.5 | 57.9 | 164 ms | 17.3 ms | 4.0 | 0 | 6,944 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 423.5 | 53.0 | 295 ms | 18.9 ms | 8.0 | 0 | 12,704 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 657.7 | 41.1 | 578 ms | 24.3 ms | 16.0 | 0 | 19,728 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 62.8 | 62.8 | 106 ms | 15.9 ms | 1.0 | 0 | 1,884 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 117.4 | 58.7 | 178 ms | 17.0 ms | 2.0 | 0 | 3,521 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 216.1 | 54.1 | 319 ms | 18.5 ms | 4.0 | 0 | 6,484 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 372.7 | 46.8 | 642 ms | 21.4 ms | 8.0 | 0 | 11,176 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 57.4 | 57.4 | 215 ms | 17.4 ms | 1.0 | 0 | 1,720 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 106.8 | 53.5 | 387 ms | 18.7 ms | 2.0 | 0 | 3,204 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 186.9 | 47.0 | 777 ms | 21.3 ms | 4.0 | 0 | 5,607 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 49.3 | 49.3 | 487 ms | 20.3 ms | 1.0 | 0 | 1,479 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 89.9 | 45.2 | 870 ms | 22.1 ms | 2.0 | 0 | 2,697 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.