Speed test matrix
NVIDIA-Nemotron-3-Super-120B-A12B-FP8 on H200 NVL 141GB ×2
sglang · 0.5.6.post3.dev9335+gdd15fb57b · FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9335+gdd15fb57b
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
dd15fb57b5ef
Run
- Served model
- nemotron
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-nvidia-sglang/bin/sglang" serve --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 --served-model-name nemotron --trust-remote-code --tp 2 --reasoning-parser nemotron_3 --tool-call-parser qwen3_coder --context-length 1048576 --mem-fraction-static 0.850546 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3
Prefill
prompt tok/s by context length8K context13.9K588 ms TTFT
16K context14.8K1,101 ms TTFT
32K context15.5K2,087 ms TTFT
64K context14.6K4,421 ms TTFT
128K context14.9K8,654 ms TTFT
256K context13.5K19,037 ms TTFT
512K context11.5K44,865 ms TTFT
1019K context8,726117,346 ms TTFT
1024K context8,632119,205 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | Looped | 284.3135 ms TTFT | Looped | Looped | Looped | Looped | 2,815.6736 ms TTFT | Looped |
| 16K | 147.2102 ms TTFT | Looped | Looped | Looped | Looped | Looped | Looped | Looped |
| 32K | 146.7127 ms TTFT | Looped | Looped | Looped | Looped | Looped | Looped | Looped |
| 64K | 145.9296 ms TTFT | 277.7225 ms TTFT | Looped | Looped | 1,083.51,430 ms TTFT | 1,597.12,889 ms TTFT | 2,300.25,593 ms TTFT | Looped |
| 128K | 143.8276 ms TTFT | Looped | Looped | 720.61,458 ms TTFT | Looped | 1,465.95,532 ms TTFT | 2,022.111,163 ms TTFT | Over capacity |
| 256K | 140.5479 ms TTFT | Looped | Looped | Looped | 942.75,486 ms TTFT | Looped | Over capacity | Over capacity |
| 512K | 133.7867 ms TTFT | 245.51,601 ms TTFT | Looped | Looped | 788.011,620 ms TTFT | Over capacity | Over capacity | Over capacity |
| 1019K | Looped | Looped | Looped | Looped | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 13.9K | — | 588 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14.8K | — | 1,101 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 15.5K | — | 2,087 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 14.6K | — | 4,421 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.9K | — | 8,654 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 13.5K | — | 19,037 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 11.5K | — | 44,865 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 8,726.0 | — | 117,346 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 8,632.0 | — | 119,205 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 2 | Valid | 284.3 | 143.1 | 135 ms | 7.0 ms | 2.0 | 2 | 8,528 | 30.0 s |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Valid | 2,815.6 | 44.8 | 736 ms | 22.3 ms | 64.0 | 0 | 84,477 | 30.0 s |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 147.2 | 147.8 | 102 ms | 6.8 ms | 1.0 | 1 | 4,416 | 30.0 s |
| Sustained decode | 16K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Valid | 146.7 | 147.4 | 127 ms | 6.8 ms | 1.0 | 1 | 4,402 | 30.0 s |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 1 | Valid | 145.9 | 146.7 | 296 ms | 6.8 ms | 1.0 | 1 | 4,375 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 277.7 | 140.3 | 225 ms | 7.1 ms | 1.9 | 2 | 8,331 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Valid | 1,083.5 | 68.7 | 1,430 ms | 14.6 ms | 16.0 | 0 | 32,496 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,597.1 | 51.2 | 2,889 ms | 19.5 ms | 32.0 | 0 | 47,897 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,300.2 | 37.6 | 5,593 ms | 26.6 ms | 64.0 | 0 | 68,989 | 30.0 s |
| Sustained decode | 64K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 1 | Valid | 143.8 | 145.4 | 276 ms | 6.9 ms | 1.0 | 1 | 4,315 | 30.0 s |
| Sustained decode | 128K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 8 | Valid | 720.6 | 91.5 | 1,458 ms | 10.9 ms | 8.0 | 0 | 21,616 | 30.0 s |
| Sustained decode | 128K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 32 | Valid | 1,465.9 | 48.7 | 5,532 ms | 20.5 ms | 32.0 | 0 | 43,961 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,022.1 | 36.0 | 11,163 ms | 27.8 ms | 64.0 | 0 | 60,666 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 140.5 | 142.9 | 479 ms | 7.0 ms | 1.0 | 1 | 4,216 | 30.0 s |
| Sustained decode | 256K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 256K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 256K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 256K | 16 | Valid | 942.7 | 63.4 | 5,486 ms | 15.8 ms | 16.0 | 0 | 28,270 | 30.0 s |
| Sustained decode | 256K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 133.7 | 137.6 | 867 ms | 7.3 ms | 1.0 | 1 | 4,011 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 245.5 | 128.4 | 1,601 ms | 7.8 ms | 2.0 | 2 | 7,366 | 30.0 s |
| Sustained decode | 512K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 512K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 512K | 16 | Valid | 788.0 | 57.6 | 11,620 ms | 17.3 ms | 16.0 | 0 | 23,630 | 30.0 s |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 1019K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 1019K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 1019K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.