Speed test matrix
Nanbeige4.2-3B on H200 NVL 141GB
sglang · 0.0.0.dev17517+g3e59d89e5 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.0.0.dev17517+g3e59d89e5
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/Nanbeige/sglang
- Build
- cuda-13.2
- Commit
3e59d89e5349
Run
- Served model
- nanbeige
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nanbeige-sglang/bin/sglang" serve --model-path Nanbeige/Nanbeige4.2-3B --served-model-name nanbeige --tp 1 --reasoning-parser nanbeige --tool-call-parser nanbeige --context-length 262144 --mem-fraction-static 0.855920 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-algorithm DSPARK --speculative-draft-model-path Nanbeige/Nanbeige4.2-3B-DSpark --speculative-dspark-block-size 7
Prefill
prompt tok/s by context length8K context26.6K308 ms TTFT
16K context22.8K712 ms TTFT
32K context17K1,900 ms TTFT
64K context11K5,868 ms TTFT
128K context6,36820,158 ms TTFT
251K context3,52571,347 ms TTFT
256K context3,48773,578 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 310.848 ms TTFT | 153.1126 ms TTFT | 997.884 ms TTFT | 1,823.672 ms TTFT | 2,153.4155 ms TTFT | 3,195.7371 ms TTFT | Over capacity | Over capacity |
| 16K | 291.374 ms TTFT | 318.095 ms TTFT | 908.599 ms TTFT | 1,515.1159 ms TTFT | 1,857.5365 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 267.3100 ms TTFT | 397.6124 ms TTFT | 794.8162 ms TTFT | 1,114.5256 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 218.7149 ms TTFT | 177.6275 ms TTFT | 566.8308 ms TTFT | 484.7754 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 182.9308 ms TTFT | 122.6561 ms TTFT | 251.51,034 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 120.7723 ms TTFT | 98.81,414 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 26.6K | — | 308 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 22.8K | — | 712 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 17K | — | 1,900 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11K | — | 5,868 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,368.0 | — | 20,158 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,525.0 | — | 71,347 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,487.0 | — | 73,578 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 310.8 | 307.7 | 48 ms | 3.2 ms | 1.0 | 2 | 9,322 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 153.1 | 77.3 | 126 ms | 13.0 ms | 3.0 | 0 | 4,592 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 997.8 | 233.9 | 84 ms | 4.3 ms | 4.5 | 4 | 29,927 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,823.6 | 228.8 | 72 ms | 4.4 ms | 8.0 | 11 | 54,709 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,153.4 | 149.7 | 155 ms | 6.7 ms | 17.4 | 16 | 64,584 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,195.7 | 95.0 | 371 ms | 10.5 ms | 33.8 | 30 | 95,850 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 291.3 | 299.1 | 74 ms | 3.3 ms | 1.0 | 2 | 8,736 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 318.0 | 215.3 | 95 ms | 5.5 ms | 19.5 | 2 | 9,540 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 908.5 | 228.6 | 99 ms | 4.4 ms | 4.5 | 4 | 27,251 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,515.1 | 189.0 | 159 ms | 5.3 ms | 7.9 | 8 | 45,443 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,857.5 | 112.0 | 365 ms | 8.9 ms | 18.1 | 3 | 55,711 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 267.3 | 268.4 | 100 ms | 3.7 ms | 1.0 | 2 | 8,016 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 397.6 | 204.4 | 124 ms | 5.0 ms | 3.5 | 2 | 11,926 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 794.8 | 193.3 | 162 ms | 5.2 ms | 4.3 | 4 | 23,837 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,114.5 | 143.1 | 256 ms | 7.0 ms | 8.7 | 8 | 33,435 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 218.7 | 221.2 | 149 ms | 4.5 ms | 1.0 | 1 | 6,561 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 177.6 | 90.6 | 275 ms | 11.0 ms | 9.4 | 0 | 5,328 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 566.8 | 148.8 | 308 ms | 6.8 ms | 4.3 | 4 | 17,000 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 484.7 | 61.7 | 754 ms | 16.2 ms | 12.0 | 0 | 14,529 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 182.9 | 183.0 | 308 ms | 5.5 ms | 1.0 | 1 | 5,486 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 122.6 | 61.9 | 561 ms | 16.2 ms | 10.0 | 0 | 3,676 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 251.5 | 63.3 | 1,034 ms | 15.8 ms | 5.4 | 0 | 7,540 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 120.7 | 174.2 | 723 ms | 6.2 ms | 1.0 | 1 | 3,620 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 98.8 | 51.8 | 1,414 ms | 19.4 ms | 5.9 | 0 | 2,963 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.