Speed test matrix
Qwen3.5-0.8B-Base on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9160+g1cf2b8c54
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1cf2b8c54d81
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_rtxpro6k-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-0.8B-Base --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.70 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-radix-cache-strategy extra_buffer --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
Prefill
prompt tok/s by context length8K context78.4K104 ms TTFT
16K context81K200 ms TTFT
32K context74.2K435 ms TTFT
64K context61.7K1,044 ms TTFT
128K context44.9K2,864 ms TTFT
251K context28.9K8,739 ms TTFT
256K context28.4K9,049 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 277.950 ms TTFT | 554.053 ms TTFT | 1,024.752 ms TTFT | Looped | 2,829.7274 ms TTFT | Looped | Not run | Not run |
| 16K | 274.577 ms TTFT | 530.479 ms TTFT | 1,018.477 ms TTFT | 1,675.0285 ms TTFT | 2,642.3513 ms TTFT | 3,371.71,788 ms TTFT | Not run | Not run |
| 32K | 237.4137 ms TTFT | 490.0140 ms TTFT | 926.2138 ms TTFT | 1,517.6587 ms TTFT | 2,214.91,045 ms TTFT | Looped | Not run | Over capacity |
| 64K | 239.6263 ms TTFT | 440.7373 ms TTFT | 788.4619 ms TTFT | 1,250.71,339 ms TTFT | 1,709.63,886 ms TTFT | Looped | Over capacity | Over capacity |
| 128K | 189.0502 ms TTFT | 413.4719 ms TTFT | Looped | 922.23,733 ms TTFT | 1,214.17,814 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 123.9936 ms TTFT | 255.11,412 ms TTFT | 455.03,769 ms TTFT | 655.97,533 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 78.4K | — | 104 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 81K | — | 200 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 74.2K | — | 435 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 61.7K | — | 1,044 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 44.9K | — | 2,864 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 28.9K | — | 8,739 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 28.4K | — | 9,049 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 277.9 | 288.2 | 50 ms | 3.5 ms | 1.0 | 2 | 8,335 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 554.0 | 282.8 | 53 ms | 3.5 ms | 2.0 | 4 | 16,621 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,024.7 | 271.6 | 52 ms | 3.7 ms | 4.0 | 8 | 30,737 | 30.0 s |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Valid | 2,829.7 | 183.0 | 274 ms | 5.5 ms | 15.9 | 16 | 84,880 | 30.0 s |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 274.5 | 279.9 | 77 ms | 3.6 ms | 1.0 | 2 | 8,236 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 530.4 | 276.0 | 79 ms | 3.6 ms | 1.9 | 4 | 15,913 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,018.4 | 259.2 | 77 ms | 3.9 ms | 4.0 | 7 | 30,550 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,675.0 | 213.3 | 285 ms | 4.7 ms | 8.0 | 8 | 50,252 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,642.3 | 173.7 | 513 ms | 5.8 ms | 16.0 | 16 | 79,270 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,371.7 | 110.0 | 1,788 ms | 9.1 ms | 32.0 | 2 | 101,124 | 30.0 s |
| Sustained decode | 16K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 237.4 | 242.1 | 137 ms | 4.1 ms | 1.0 | 1 | 7,122 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 490.0 | 261.5 | 140 ms | 3.8 ms | 2.0 | 4 | 14,699 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 926.2 | 234.9 | 138 ms | 4.3 ms | 4.0 | 5 | 27,782 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,517.6 | 198.3 | 587 ms | 5.0 ms | 7.9 | 8 | 45,527 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,214.9 | 154.8 | 1,045 ms | 6.5 ms | 15.9 | 16 | 66,448 | 30.0 s |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 239.6 | 247.5 | 263 ms | 4.0 ms | 1.0 | 1 | 7,186 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 440.7 | 227.9 | 373 ms | 4.4 ms | 2.0 | 2 | 13,221 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 788.4 | 200.3 | 619 ms | 5.0 ms | 4.0 | 4 | 23,652 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,250.7 | 171.2 | 1,339 ms | 5.8 ms | 7.9 | 8 | 37,516 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,709.6 | 116.5 | 3,886 ms | 8.6 ms | 16.0 | 4 | 51,284 | 30.0 s |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 189.0 | 195.8 | 502 ms | 5.1 ms | 1.0 | 1 | 5,670 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 413.4 | 202.4 | 719 ms | 4.9 ms | 1.9 | 2 | 12,401 | 30.0 s |
| Sustained decode | 128K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 8 | Valid | 922.2 | 131.9 | 3,733 ms | 7.6 ms | 8.0 | 5 | 27,671 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,214.1 | 90.0 | 7,814 ms | 11.1 ms | 16.0 | 0 | 36,413 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 123.9 | 152.4 | 936 ms | 6.7 ms | 1.0 | 1 | 3,717 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 255.1 | 151.2 | 1,412 ms | 6.7 ms | 1.9 | 2 | 7,651 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 455.0 | 129.8 | 3,769 ms | 7.7 ms | 3.9 | 3 | 13,541 | 29.8 s |
| Sustained decode | 251K | 8 | Valid | 655.9 | 91.2 | 7,533 ms | 11.0 ms | 8.0 | 0 | 19,668 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.