Speed test matrix
Qwen3.5-0.8B-Base on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_rtxpro6k-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-0.8B-Base --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.79 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context87.2K94 ms TTFT
16K context75.4K215 ms TTFT
32K context68.9K469 ms TTFT
64K context66.6K967 ms TTFT
128K context49.7K2,591 ms TTFT
251K context32.5K7,763 ms TTFT
256K context32.1K8,021 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 512.446 ms TTFT | 1,004.963 ms TTFT | 1,889.972 ms TTFT | Looped | Looped | 8,639.8172 ms TTFT | 10.8K1,114 ms TTFT | Looped |
| 16K | 488.297 ms TTFT | 964.1108 ms TTFT | Looped | Looped | Looped | Looped | Looped | 9,439.77,751 ms TTFT |
| 32K | 452.4162 ms TTFT | 897.5246 ms TTFT | 1,694.4252 ms TTFT | 2,968.5145 ms TTFT | 4,605.5147 ms TTFT | Looped | 6,657.27,438 ms TTFT | Over capacity |
| 64K | 395.3261 ms TTFT | 782.5250 ms TTFT | 1,481.1253 ms TTFT | 2,497.7257 ms TTFT | Looped | 4,347.74,026 ms TTFT | Over capacity | Over capacity |
| 128K | 316.3495 ms TTFT | 627.7482 ms TTFT | 1,191.7484 ms TTFT | 1,849.8500 ms TTFT | 2,387.54,125 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 231.5922 ms TTFT | 459.71,536 ms TTFT | 860.42,298 ms TTFT | 1,253.24,260 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 87.2K | — | 94 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 75.4K | — | 215 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 68.9K | — | 469 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 66.6K | — | 967 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 49.7K | — | 2,591 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 32.5K | — | 7,763 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 32.1K | — | 8,021 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 512.4 | 514.5 | 46 ms | 1.9 ms | 1.0 | 4 | 15,371 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 1,004.9 | 506.6 | 63 ms | 2.0 ms | 2.0 | 8 | 30,146 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,889.9 | 476.9 | 72 ms | 2.1 ms | 4.0 | 12 | 56,695 | 30.0 s |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Valid | 8,639.8 | 277.4 | 172 ms | 3.6 ms | 31.7 | 64 | 259,200 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 10.8K | 178.5 | 1,114 ms | 5.6 ms | 63.7 | 64 | 323,790 | 30.0 s |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 488.2 | 492.5 | 97 ms | 2.0 ms | 1.0 | 3 | 14,657 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 964.1 | 489.5 | 108 ms | 2.0 ms | 1.9 | 8 | 28,922 | 30.0 s |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Valid | 9,439.7 | 85.6 | 7,751 ms | 11.7 ms | 128.0 | 0 | 283,134 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 452.4 | 459.3 | 162 ms | 2.2 ms | 1.0 | 3 | 13,572 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 897.5 | 458.9 | 246 ms | 2.2 ms | 2.0 | 6 | 26,923 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,694.4 | 434.9 | 252 ms | 2.3 ms | 4.0 | 12 | 50,831 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,968.5 | 379.6 | 145 ms | 2.6 ms | 7.9 | 24 | 89,061 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 4,605.5 | 294.2 | 147 ms | 3.4 ms | 15.9 | 32 | 138,161 | 30.0 s |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Valid | 6,657.2 | 115.0 | 7,438 ms | 8.7 ms | 63.8 | 35 | 199,498 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 395.3 | 405.9 | 261 ms | 2.5 ms | 0.9 | 3 | 11,858 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 782.5 | 401.6 | 250 ms | 2.5 ms | 1.9 | 6 | 23,474 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,481.1 | 382.6 | 253 ms | 2.6 ms | 3.9 | 12 | 44,432 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 2,497.7 | 319.4 | 257 ms | 3.1 ms | 7.9 | 16 | 74,931 | 30.0 s |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Valid | 4,347.7 | 142.6 | 4,026 ms | 7.0 ms | 31.7 | 32 | 130,434 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 316.3 | 327.2 | 495 ms | 3.1 ms | 1.0 | 2 | 9,490 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 627.7 | 325.1 | 482 ms | 3.1 ms | 1.9 | 4 | 18,830 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 1,191.7 | 310.8 | 484 ms | 3.2 ms | 3.9 | 8 | 35,751 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,849.8 | 244.8 | 500 ms | 4.1 ms | 7.8 | 13 | 55,458 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 2,387.5 | 153.5 | 4,125 ms | 6.5 ms | 15.8 | 16 | 71,620 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 231.5 | 239.0 | 922 ms | 4.2 ms | 1.0 | 1 | 6,946 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 459.7 | 237.6 | 1,536 ms | 4.2 ms | 1.9 | 2 | 13,789 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 860.4 | 223.3 | 2,298 ms | 4.5 ms | 3.9 | 4 | 25,810 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 1,253.2 | 164.7 | 4,260 ms | 6.1 ms | 7.7 | 8 | 37,597 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.