Speed test matrix
Qwen3.8-27B-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.6.post3.dev11565+gfa7784d14 · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev11565+gfa7784d14
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
fa7784d140ee
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-radixark-sglang-fa7784d140ee/bin/sglang" serve --model-path RadixArk/Qwen3.8-27B-NVFP4 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.78 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --linear-attn-backend triton --attention-backend flashinfer --fp4-gemm-backend flashinfer_cutlass --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
Prefill
prompt tok/s by context length8K context12.8K639 ms TTFT
16K context11.8K1,367 ms TTFT
32K context10.2K3,143 ms TTFT
64K context7,8348,185 ms TTFT
128K context5,34623,954 ms TTFT
251K context3,26576,851 ms TTFT
256K context3,21879,521 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 101.841 ms TTFT | 190.8294 ms TTFT | 338.0316 ms TTFT | 563.8119 ms TTFT | 859.0423 ms TTFT | 885.9Requests queued34,779 ms TTFT | Not run | Over capacity |
| 16K | 96.339 ms TTFT | 193.662 ms TTFT | 325.566 ms TTFT | 556.195 ms TTFT | 787.7189 ms TTFT | 802.8Requests queued39,934 ms TTFT | Over capacity | Over capacity |
| 32K | 104.441 ms TTFT | 199.667 ms TTFT | 343.177 ms TTFT | 471.890 ms TTFT | 664.9134 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 88.545 ms TTFT | 169.3124 ms TTFT | 287.3102 ms TTFT | 405.9153 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 78.356 ms TTFT | 170.793 ms TTFT | 234.4137 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 79.365 ms TTFT | 136.896 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 12.8K | — | 639 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 11.8K | — | 1,367 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10.2K | — | 3,143 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,834.0 | — | 8,185 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,346.0 | — | 23,954 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,265.0 | — | 76,851 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,218.0 | — | 79,521 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 101.8 | 102.2 | 41 ms | 9.8 ms | 1.0 | 0 | 3,055 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 190.8 | 97.0 | 294 ms | 10.3 ms | 2.0 | 0 | 5,724 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 338.0 | 85.2 | 316 ms | 11.7 ms | 4.0 | 0 | 10,137 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 563.8 | 71.4 | 119 ms | 14.0 ms | 8.0 | 0 | 16,904 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 859.0 | 55.1 | 423 ms | 18.2 ms | 16.0 | 0 | 25,756 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 885.9 | 55.5 | 34,779 ms | 18.0 ms | 15.7 | 16 | 26,551 | 30.0 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 96.3 | 98.2 | 39 ms | 10.2 ms | 1.0 | 0 | 2,888 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 193.6 | 98.8 | 62 ms | 10.1 ms | 2.0 | 0 | 5,803 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 325.5 | 83.0 | 66 ms | 12.1 ms | 4.0 | 0 | 9,759 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 556.1 | 68.0 | 95 ms | 14.7 ms | 8.0 | 0 | 16,673 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 787.7 | 50.2 | 189 ms | 19.9 ms | 16.0 | 0 | 23,628 | 30.0 s |
| Sustained decode | 16K | 32 | Requests queued | 802.8 | 51.1 | 39,934 ms | 19.6 ms | 15.8 | 16 | 24,069 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 104.4 | 105.7 | 41 ms | 9.5 ms | 1.0 | 0 | 3,132 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 199.6 | 101.1 | 67 ms | 10.0 ms | 2.0 | 0 | 5,987 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 343.1 | 89.3 | 77 ms | 11.2 ms | 4.0 | 0 | 10,292 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 471.8 | 60.7 | 90 ms | 16.5 ms | 8.0 | 0 | 14,143 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 664.9 | 41.9 | 134 ms | 23.9 ms | 16.0 | 0 | 19,938 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 88.5 | 90.3 | 45 ms | 11.1 ms | 1.0 | 0 | 2,654 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 169.3 | 85.7 | 124 ms | 11.7 ms | 2.0 | 0 | 5,079 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 287.3 | 71.2 | 102 ms | 14.1 ms | 4.0 | 0 | 8,623 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 405.9 | 51.0 | 153 ms | 19.6 ms | 8.0 | 0 | 12,176 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 78.3 | 79.3 | 56 ms | 12.6 ms | 1.0 | 0 | 2,349 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 170.7 | 87.2 | 93 ms | 11.5 ms | 2.0 | 0 | 5,120 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 234.4 | 58.4 | 137 ms | 17.1 ms | 4.0 | 0 | 7,027 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 79.3 | 80.4 | 65 ms | 12.4 ms | 1.0 | 0 | 2,377 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 136.8 | 70.6 | 96 ms | 14.2 ms | 2.0 | 0 | 4,103 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.