Speed test matrix
Qwen3.6-35B-A3B-NVFP4 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9335+gdd15fb57b · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9335+gdd15fb57b
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
dd15fb57b5ef
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-nvidia-sglang/bin/sglang" serve --model-path nvidia/Qwen3.6-35B-A3B-NVFP4 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.859700 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context33.8K241 ms TTFT
16K context36.1K446 ms TTFT
32K context36.3K885 ms TTFT
64K context33.5K1,911 ms TTFT
128K context28.6K4,474 ms TTFT
251K context21.8K11,510 ms TTFT
256K context21.5K11,917 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 253.686 ms TTFT | 473.770 ms TTFT | 849.1128 ms TTFT | 1,396.3225 ms TTFT | 1,999.5367 ms TTFT | 2,583.5666 ms TTFT | 3,808.61,300 ms TTFT | 5,952.12,547 ms TTFT |
| 16K | 251.672 ms TTFT | 141.0136 ms TTFT | 809.7203 ms TTFT | 1,338.8345 ms TTFT | 1,795.1696 ms TTFT | 3,089.91,365 ms TTFT | 3,870.02,694 ms TTFT | 5,261.1Requests queued5,071 ms TTFT |
| 32K | 246.2137 ms TTFT | 102.1265 ms TTFT | 614.1453 ms TTFT | 1,146.3700 ms TTFT | 1,316.01,677 ms TTFT | 2,401.23,309 ms TTFT | 2,417.96,513 ms TTFT | 2,262.7Requests queued12,683 ms TTFT |
| 64K | 239.0291 ms TTFT | 62.8586 ms TTFT | 188.61,010 ms TTFT | 517.12,077 ms TTFT | 1,263.63,909 ms TTFT | 1,775.37,515 ms TTFT | 1,620.814,774 ms TTFT | Over capacity |
| 128K | 224.0518 ms TTFT | 51.31,042 ms TTFT | 105.61,957 ms TTFT | 214.53,869 ms TTFT | 636.07,725 ms TTFT | 901.115,430 ms TTFT | Over capacity | Over capacity |
| 251K | 199.71,048 ms TTFT | 55.72,065 ms TTFT | 115.64,066 ms TTFT | 279.88,048 ms TTFT | 398.016,003 ms TTFT | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 33.8K | — | 241 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 36.1K | — | 446 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 36.3K | — | 885 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 33.5K | — | 1,911 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 28.6K | — | 4,474 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 21.8K | — | 11,510 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 21.5K | — | 11,917 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 253.6 | 255.2 | 86 ms | 3.9 ms | 1.0 | 2 | 7,608 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 473.7 | 230.1 | 70 ms | 4.4 ms | 2.0 | 2 | 14,209 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 849.1 | 213.5 | 128 ms | 4.7 ms | 4.0 | 4 | 25,472 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,396.3 | 175.7 | 225 ms | 5.7 ms | 7.9 | 8 | 41,890 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,999.5 | 141.5 | 367 ms | 7.2 ms | 19.2 | 16 | 59,985 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,583.5 | 81.1 | 666 ms | 12.3 ms | 48.0 | 0 | 77,504 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,808.6 | 58.5 | 1,300 ms | 17.1 ms | 89.1 | 0 | 114,240 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,952.1 | 42.0 | 2,547 ms | 23.8 ms | 128.0 | 0 | 178,504 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 251.6 | 252.3 | 72 ms | 4.0 ms | 1.0 | 2 | 7,547 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 141.0 | 66.9 | 136 ms | 14.9 ms | 58.8 | 0 | 4,228 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 809.7 | 201.3 | 203 ms | 5.0 ms | 4.3 | 4 | 24,287 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,338.8 | 167.9 | 345 ms | 6.0 ms | 8.3 | 8 | 40,161 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,795.1 | 111.0 | 696 ms | 9.0 ms | 20.5 | 0 | 53,856 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,089.9 | 97.8 | 1,365 ms | 10.2 ms | 32.0 | 0 | 92,704 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,870.0 | 60.0 | 2,694 ms | 16.7 ms | 71.8 | 0 | 116,096 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 5,261.1 | 37.1 | 5,071 ms | 26.9 ms | 128.3 | 0 | 157,824 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 246.2 | 248.6 | 137 ms | 4.0 ms | 1.0 | 2 | 7,385 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 102.1 | 49.5 | 265 ms | 20.2 ms | 94.4 | 0 | 3,064 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 614.1 | 169.2 | 453 ms | 6.1 ms | 15.5 | 4 | 18,424 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,146.3 | 152.2 | 700 ms | 6.6 ms | 10.0 | 8 | 34,373 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,316.0 | 83.9 | 1,677 ms | 11.9 ms | 23.9 | 0 | 39,472 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,401.2 | 75.7 | 3,309 ms | 13.2 ms | 37.3 | 0 | 72,032 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,417.9 | 40.5 | 6,513 ms | 24.7 ms | 92.2 | 0 | 72,544 | 30.0 s |
| Sustained decode | 32K | 128 | Requests queued | 2,262.7 | 28.3 | 12,683 ms | 35.3 ms | 153.0 | 0 | 67,818 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 239.0 | 241.4 | 291 ms | 4.1 ms | 1.0 | 1 | 7,170 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 62.8 | 31.5 | 586 ms | 31.8 ms | 130.0 | 0 | 1,882 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 188.6 | 44.9 | 1,010 ms | 22.3 ms | 74.4 | 0 | 5,656 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 517.1 | 63.3 | 2,077 ms | 15.8 ms | 39.3 | 0 | 15,512 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,263.6 | 80.7 | 3,909 ms | 12.4 ms | 20.7 | 0 | 37,904 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,775.3 | 58.8 | 7,515 ms | 17.0 ms | 39.0 | 0 | 53,248 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 1,620.8 | 30.6 | 14,774 ms | 32.7 ms | 96.0 | 0 | 48,576 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 224.0 | 228.0 | 518 ms | 4.4 ms | 1.0 | 1 | 6,721 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 51.3 | 25.7 | 1,042 ms | 39.0 ms | 83.5 | 0 | 1,538 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 105.6 | 26.9 | 1,957 ms | 37.2 ms | 70.0 | 0 | 3,167 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 214.5 | 27.9 | 3,869 ms | 35.9 ms | 73.3 | 0 | 6,432 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 636.0 | 44.0 | 7,725 ms | 22.7 ms | 28.8 | 0 | 19,072 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 901.1 | 34.1 | 15,430 ms | 29.3 ms | 53.7 | 0 | 27,008 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 199.7 | 207.0 | 1,048 ms | 4.8 ms | 1.0 | 1 | 5,990 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 55.7 | 28.2 | 2,065 ms | 35.4 ms | 48.9 | 0 | 1,672 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 115.6 | 30.2 | 4,066 ms | 33.2 ms | 38.0 | 0 | 3,464 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 279.8 | 36.9 | 8,048 ms | 27.1 ms | 23.7 | 0 | 8,392 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 398.0 | 31.5 | 16,003 ms | 31.8 ms | 28.0 | 0 | 11,936 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.