Speed test matrix
NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9335+gdd15fb57b · NVFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9335+gdd15fb57b
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
dd15fb57b5ef
Run
- Served model
- nemotron
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-nvidia-sglang/bin/sglang" serve --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 --served-model-name nemotron --trust-remote-code --tp 1 --reasoning-parser nemotron_3 --tool-call-parser qwen3_coder --context-length 1048576 --mem-fraction-static 0.841880 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3 --mamba-ssm-dtype float16
Prefill
prompt tok/s by context length8K context9,705845 ms TTFT
16K context9,9891,626 ms TTFT
32K context9,8883,268 ms TTFT
64K context9,6936,650 ms TTFT
128K context9,16114,055 ms TTFT
256K context8,14231,604 ms TTFT
512K context6,65777,282 ms TTFT
1019K context4,876209,963 ms TTFT
1024K context4,862211,602 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 131.881 ms TTFT | 206.4136 ms TTFT | 350.0137 ms TTFT | 500.8165 ms TTFT | 643.3220 ms TTFT | 952.1359 ms TTFT | 476.6Requests queued719 ms TTFT | 476.3Requests queued1,295 ms TTFT |
| 16K | 131.296 ms TTFT | 64.8Requests queued168 ms TTFT | 217.6229 ms TTFT | 408.5272 ms TTFT | 626.1392 ms TTFT | 926.8705 ms TTFT | 725.4Requests queued49,429 ms TTFT | 466.7Requests queued2,584 ms TTFT |
| 32K | 130.6116 ms TTFT | 58.1191 ms TTFT | 118.0218 ms TTFT | 249.0377 ms TTFT | 531.1781 ms TTFT | 855.4Requests queued1,460 ms TTFT | 1,021.2Requests queued45,118 ms TTFT | 498.0Requests queued44,427 ms TTFT |
| 64K | 129.3171 ms TTFT | 55.3281 ms TTFT | 128.5395 ms TTFT | 299.9772 ms TTFT | 488.91,524 ms TTFT | 771.43,007 ms TTFT | 1,053.8Requests queued47,787 ms TTFT | Over capacity |
| 128K | 126.6257 ms TTFT | 52.9458 ms TTFT | 155.9743 ms TTFT | 300.41,445 ms TTFT | 425.73,044 ms TTFT | 658.85,674 ms TTFT | Over capacity | Over capacity |
| 256K | 122.0449 ms TTFT | 45.0855 ms TTFT | 133.81,514 ms TTFT | 227.53,055 ms TTFT | 347.36,223 ms TTFT | Over capacity | Over capacity | Over capacity |
| 512K | 117.0841 ms TTFT | 47.81,685 ms TTFT | 72.73,013 ms TTFT | 141.16,065 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 101.1Requests queued1,645 ms TTFT | 31.8Requests queued3,219 ms TTFT | 65.96,312 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,705.0 | — | 845 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,989.0 | — | 1,626 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,888.0 | — | 3,268 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,693.0 | — | 6,650 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,161.0 | — | 14,055 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,142.0 | — | 31,604 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 6,657.0 | — | 77,282 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 4,876.0 | — | 209,963 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 4,862.0 | — | 211,602 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 131.8 | 132.3 | 81 ms | 7.6 ms | 1.0 | 1 | 3,953 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 206.4 | 99.6 | 136 ms | 10.0 ms | 2.2 | 0 | 6,192 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 350.0 | 85.0 | 137 ms | 11.8 ms | 4.1 | 0 | 10,500 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 500.8 | 62.0 | 165 ms | 16.1 ms | 9.7 | 0 | 15,025 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 643.3 | 40.5 | 220 ms | 24.7 ms | 24.0 | 0 | 19,296 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 952.1 | 29.8 | 359 ms | 33.6 ms | 48.0 | 0 | 28,544 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 476.6 | 30.1 | 719 ms | 33.2 ms | 48.0 | 0 | 14,288 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 476.3 | 30.7 | 1,295 ms | 32.5 ms | 48.0 | 0 | 14,287 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 131.2 | 131.8 | 96 ms | 7.6 ms | 1.0 | 1 | 3,937 | 30.0 s |
| Sustained decode | 16K | 2 | Requests queued | 64.8 | 32.4 | 168 ms | 30.9 ms | 34.0 | 0 | 1,942 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 217.6 | 51.5 | 229 ms | 19.4 ms | 16.9 | 0 | 6,528 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 408.5 | 50.9 | 272 ms | 19.6 ms | 13.3 | 0 | 12,254 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 626.1 | 38.7 | 392 ms | 25.9 ms | 24.4 | 0 | 18,768 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 926.8 | 28.9 | 705 ms | 34.5 ms | 48.0 | 0 | 27,805 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 725.4 | 29.5 | 49,429 ms | 33.9 ms | 48.0 | 0 | 21,751 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 466.7 | 30.7 | 2,584 ms | 32.6 ms | 48.0 | 0 | 13,999 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 130.6 | 131.1 | 116 ms | 7.6 ms | 1.0 | 1 | 3,917 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 58.1 | 29.1 | 191 ms | 34.4 ms | 42.0 | 0 | 1,742 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 118.0 | 29.4 | 218 ms | 34.1 ms | 43.9 | 0 | 3,536 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 249.0 | 31.1 | 377 ms | 32.2 ms | 38.0 | 0 | 7,463 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 531.1 | 33.4 | 781 ms | 29.9 ms | 29.3 | 0 | 15,920 | 30.0 s |
| Sustained decode | 32K | 32 | Requests queued | 855.4 | 27.7 | 1,460 ms | 36.2 ms | 47.9 | 0 | 25,648 | 30.0 s |
| Sustained decode | 32K | 64 | Requests queued | 1,021.2 | 27.5 | 45,118 ms | 36.4 ms | 48.0 | 0 | 30,634 | 30.0 s |
| Sustained decode | 32K | 128 | Requests queued | 498.0 | 27.8 | 44,427 ms | 36.0 ms | 47.9 | 0 | 14,926 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 129.3 | 130.0 | 171 ms | 7.7 ms | 1.0 | 1 | 3,878 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 55.3 | 27.6 | 281 ms | 36.2 ms | 42.0 | 0 | 1,660 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 128.5 | 31.5 | 395 ms | 31.8 ms | 32.1 | 0 | 3,852 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 299.9 | 36.6 | 772 ms | 27.3 ms | 19.3 | 0 | 8,991 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 488.9 | 30.8 | 1,524 ms | 32.5 ms | 29.0 | 0 | 14,655 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 771.4 | 24.3 | 3,007 ms | 41.2 ms | 48.0 | 0 | 23,136 | 30.0 s |
| Sustained decode | 64K | 64 | Requests queued | 1,053.8 | 24.1 | 47,787 ms | 41.5 ms | 47.9 | 0 | 31,610 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 126.6 | 127.5 | 257 ms | 7.8 ms | 1.0 | 1 | 3,798 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 52.9 | 26.5 | 458 ms | 37.7 ms | 34.0 | 0 | 1,586 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 155.9 | 37.5 | 743 ms | 26.6 ms | 17.2 | 0 | 4,674 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 300.4 | 37.3 | 1,445 ms | 26.8 ms | 15.9 | 0 | 9,008 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 425.7 | 27.3 | 3,044 ms | 36.6 ms | 28.6 | 0 | 12,751 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 658.8 | 21.1 | 5,674 ms | 47.4 ms | 48.0 | 0 | 19,742 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 122.0 | 124.8 | 449 ms | 8.0 ms | 1.0 | 1 | 3,660 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 45.0 | 22.4 | 855 ms | 44.6 ms | 34.0 | 0 | 1,349 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 133.8 | 32.6 | 1,514 ms | 30.7 ms | 19.5 | 0 | 4,012 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 227.5 | 29.1 | 3,055 ms | 34.3 ms | 19.2 | 0 | 6,824 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 347.3 | 23.4 | 6,223 ms | 42.7 ms | 30.0 | 0 | 10,415 | 30.0 s |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 117.0 | 117.0 | 841 ms | 8.5 ms | 1.0 | 0 | 3,508 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 47.8 | 24.1 | 1,685 ms | 41.6 ms | 30.0 | 0 | 1,432 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 72.7 | 18.8 | 3,013 ms | 53.3 ms | 30.0 | 0 | 2,180 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 141.1 | 18.7 | 6,065 ms | 53.4 ms | 34.4 | 0 | 4,231 | 30.0 s |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Requests queued | 101.1 | 105.1 | 1,645 ms | 9.5 ms | 1.0 | 0 | 3,032 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 31.8 | 17.4 | 3,219 ms | 57.4 ms | 31.9 | 0 | 952 | 30.0 s |
| Sustained decode | 1019K | 4 | Valid | 65.9 | 17.1 | 6,312 ms | 58.4 ms | 23.9 | 0 | 1,976 | 30.0 s |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.