Speed test matrix
Qwen3.5-0.8B on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-0.8B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.69 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-flashinfer-allreduce-fusion --mamba-radix-cache-strategy extra_buffer --max-running-requests 128 --chunked-prefill-size 16384 --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
Prefill
prompt tok/s by context length8K context78.9K104 ms TTFT
16K context82.7K196 ms TTFT
32K context73.8K438 ms TTFT
64K context62.3K1,034 ms TTFT
128K context46K2,799 ms TTFT
251K context29.5K8,546 ms TTFT
256K context29.4K8,749 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 411.549 ms TTFT | 784.149 ms TTFT | 1,548.352 ms TTFT | 2,674.552 ms TTFT | 4,035.359 ms TTFT | Looped | 6,333.71,816 ms TTFT | 6,829.23,575 ms TTFT |
| 16K | 387.075 ms TTFT | 783.882 ms TTFT | 1,398.583 ms TTFT | 2,439.078 ms TTFT | 3,830.2113 ms TTFT | Looped | Looped | Looped |
| 32K | 353.2141 ms TTFT | 722.9146 ms TTFT | 1,319.9147 ms TTFT | 2,253.6147 ms TTFT | Looped | Looped | Looped | Over capacity |
| 64K | Looped | 630.8386 ms TTFT | Looped | Looped | 2,617.82,314 ms TTFT | 3,044.98,205 ms TTFT | Over capacity | Over capacity |
| 128K | 270.6497 ms TTFT | 510.7634 ms TTFT | 933.31,437 ms TTFT | Looped | 1,796.411,157 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 209.4941 ms TTFT | 384.21,409 ms TTFT | 694.72,346 ms TTFT | 936.27,504 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 78.9K | — | 104 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 82.7K | — | 196 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 73.8K | — | 438 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 62.3K | — | 1,034 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 46K | — | 2,799 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 29.5K | — | 8,546 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 29.4K | — | 8,749 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 411.5 | 406.2 | 49 ms | 2.5 ms | 1.0 | 3 | 12,344 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 784.1 | 400.5 | 49 ms | 2.5 ms | 2.0 | 6 | 23,522 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,548.3 | 386.5 | 52 ms | 2.6 ms | 4.0 | 12 | 46,450 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,674.5 | 324.6 | 52 ms | 3.1 ms | 7.9 | 18 | 80,237 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 4,035.3 | 250.3 | 59 ms | 4.0 ms | 15.9 | 32 | 121,060 | 30.0 s |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Valid | 6,333.7 | 99.8 | 1,816 ms | 10.0 ms | 64.0 | 2 | 189,928 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,829.2 | 55.4 | 3,575 ms | 18.1 ms | 128.0 | 0 | 204,754 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 387.0 | 385.4 | 75 ms | 2.6 ms | 1.0 | 3 | 11,609 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 783.8 | 383.7 | 82 ms | 2.6 ms | 2.0 | 6 | 23,514 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,398.5 | 348.2 | 83 ms | 2.9 ms | 4.0 | 10 | 41,886 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,439.0 | 300.3 | 78 ms | 3.3 ms | 8.0 | 16 | 73,164 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 3,830.2 | 238.2 | 113 ms | 4.2 ms | 15.9 | 23 | 114,907 | 30.0 s |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Valid | 353.2 | 352.4 | 141 ms | 2.8 ms | 1.0 | 2 | 10,597 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 722.9 | 364.8 | 146 ms | 2.7 ms | 2.0 | 6 | 21,685 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,319.9 | 330.0 | 147 ms | 3.0 ms | 4.0 | 8 | 39,593 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,253.6 | 279.4 | 147 ms | 3.6 ms | 7.8 | 16 | 67,623 | 30.0 s |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 2 | Valid | 630.8 | 324.1 | 386 ms | 3.1 ms | 2.0 | 4 | 18,923 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Valid | 2,617.8 | 168.2 | 2,314 ms | 5.9 ms | 15.8 | 16 | 78,537 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 3,044.9 | 104.8 | 8,205 ms | 9.5 ms | 31.8 | 10 | 91,342 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 270.6 | 271.0 | 497 ms | 3.7 ms | 1.0 | 2 | 8,117 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 510.7 | 265.3 | 634 ms | 3.8 ms | 1.9 | 4 | 15,320 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 933.3 | 235.5 | 1,437 ms | 4.2 ms | 3.8 | 5 | 27,998 | 30.0 s |
| Sustained decode | 128K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 16 | Valid | 1,796.4 | 122.5 | 11,157 ms | 8.2 ms | 15.6 | 9 | 53,886 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 209.4 | 212.3 | 941 ms | 4.7 ms | 1.0 | 1 | 6,280 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 384.2 | 196.9 | 1,409 ms | 5.1 ms | 1.9 | 2 | 11,525 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 694.7 | 189.0 | 2,346 ms | 5.3 ms | 3.9 | 4 | 20,838 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 936.2 | 130.7 | 7,504 ms | 7.7 ms | 7.8 | 6 | 27,604 | 29.5 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.