Speed test matrix
Nanbeige4.2-3B on H200 NVL 141GB
sglang · 0.0.0.dev17517+g3e59d89e5 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.0.0.dev17517+g3e59d89e5
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/Nanbeige/sglang
- Build
- cuda-13.2
- Commit
3e59d89e5349
Run
- Served model
- nanbeige
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nanbeige-sglang/bin/sglang" serve --model-path Nanbeige/Nanbeige4.2-3B --served-model-name nanbeige --tp 1 --reasoning-parser nanbeige --tool-call-parser nanbeige --context-length 262144 --mem-fraction-static 0.851935 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context27.9K294 ms TTFT
16K context23.1K701 ms TTFT
32K context17.1K1,881 ms TTFT
64K context11K5,822 ms TTFT
128K context6,40320,046 ms TTFT
251K context3,53671,128 ms TTFT
256K context3,49573,407 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 166.5Requests queued44 ms TTFT | 119.679 ms TTFT | 531.787 ms TTFT | 758.7127 ms TTFT | 1,476.0207 ms TTFT | 1,861.2364 ms TTFT | Over capacity | Over capacity |
| 16K | 156.465 ms TTFT | 89.8101 ms TTFT | 352.2124 ms TTFT | 745.0207 ms TTFT | 1,020.3362 ms TTFT | 997.8683 ms TTFT | Over capacity | Over capacity |
| 32K | 140.692 ms TTFT | 53.4153 ms TTFT | 119.5189 ms TTFT | 188.8380 ms TTFT | 429.9723 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 117.5139 ms TTFT | 49.1253 ms TTFT | 78.4405 ms TTFT | 118.3757 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 88.8284 ms TTFT | 28.9562 ms TTFT | 48.61,007 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 59.6720 ms TTFT | 21.51,467 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 27.9K | — | 294 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 23.1K | — | 701 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 17.1K | — | 1,881 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11K | — | 5,822 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,403.0 | — | 20,046 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,536.0 | — | 71,128 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,495.0 | — | 73,407 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Requests queued | 166.5 | 167.6 | 44 ms | 6.0 ms | 1.0 | 1 | 4,994 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 119.6 | 59.5 | 79 ms | 16.8 ms | 3.0 | 0 | 3,586 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 531.7 | 139.1 | 87 ms | 7.2 ms | 4.7 | 4 | 15,950 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 758.7 | 95.8 | 127 ms | 10.4 ms | 12.0 | 0 | 22,761 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,476.0 | 89.9 | 207 ms | 11.1 ms | 16.8 | 0 | 44,272 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,861.2 | 57.2 | 364 ms | 17.5 ms | 38.4 | 0 | 55,836 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 156.4 | 157.5 | 65 ms | 6.4 ms | 1.0 | 1 | 4,693 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 89.8 | 45.2 | 101 ms | 22.1 ms | 34.0 | 0 | 2,692 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 352.2 | 81.9 | 124 ms | 12.2 ms | 10.1 | 0 | 10,564 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 745.0 | 90.9 | 207 ms | 11.0 ms | 9.2 | 0 | 22,344 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,020.3 | 62.0 | 362 ms | 16.1 ms | 18.9 | 0 | 30,608 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 997.8 | 31.6 | 683 ms | 31.7 ms | 48.0 | 0 | 29,920 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 140.6 | 141.8 | 92 ms | 7.1 ms | 1.0 | 1 | 4,216 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 53.4 | 26.5 | 153 ms | 37.8 ms | 45.7 | 0 | 1,600 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 119.5 | 30.0 | 189 ms | 33.3 ms | 38.0 | 0 | 3,584 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 188.8 | 23.7 | 380 ms | 42.2 ms | 46.0 | 0 | 5,659 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 429.9 | 27.1 | 723 ms | 36.9 ms | 30.0 | 0 | 12,896 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 117.5 | 117.6 | 139 ms | 8.5 ms | 1.0 | 0 | 3,523 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 49.1 | 24.1 | 253 ms | 41.4 ms | 30.3 | 0 | 1,474 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 78.4 | 19.5 | 405 ms | 51.2 ms | 31.1 | 0 | 2,352 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 118.3 | 14.9 | 757 ms | 67.1 ms | 38.0 | 0 | 3,544 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 88.8 | 88.8 | 284 ms | 11.3 ms | 1.0 | 0 | 2,663 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 28.9 | 14.4 | 562 ms | 69.4 ms | 32.3 | 0 | 866 | 29.9 s |
| Sustained decode | 128K | 4 | Valid | 48.6 | 12.2 | 1,007 ms | 81.8 ms | 35.7 | 0 | 1,456 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 59.6 | 59.6 | 720 ms | 16.8 ms | 1.0 | 0 | 1,788 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 21.5 | 10.9 | 1,467 ms | 92.2 ms | 22.0 | 0 | 644 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.