Speed test matrix
Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9335+gdd15fb57b · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9335+gdd15fb57b
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
dd15fb57b5ef
Run
- Served model
- nemotron
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nvidia-sglang/bin/sglang" serve --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 --served-model-name nemotron --trust-remote-code --tp 1 --reasoning-parser nemotron_3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.871010 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3
Prefill
prompt tok/s by context length8K context34K241 ms TTFT
16K context45.6K356 ms TTFT
32K context44.5K726 ms TTFT
64K context40.6K1,588 ms TTFT
128K context34.3K3,756 ms TTFT
251K context25.5K9,878 ms TTFT
256K context25.4K10,115 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 367.8102 ms TTFT | 648.6152 ms TTFT | 1,209.7205 ms TTFT | 1,147.0248 ms TTFT | 2,467.6405 ms TTFT | 4,355.1702 ms TTFT | 2,654.81,344 ms TTFT | 3,393.22,556 ms TTFT |
| 16K | 386.4224 ms TTFT | 584.1309 ms TTFT | 1,021.5442 ms TTFT | 1,311.7457 ms TTFT | 2,320.1725 ms TTFT | 3,992.31,348 ms TTFT | 2,621.02,748 ms TTFT | 4,136.25,600 ms TTFT |
| 32K | 358.0279 ms TTFT | 614.7461 ms TTFT | 741.7489 ms TTFT | 1,411.4799 ms TTFT | 2,447.0909 ms TTFT | 2,475.43,026 ms TTFT | 2,854.16,334 ms TTFT | 4,029.612,422 ms TTFT |
| 64K | 349.1399 ms TTFT | 399.6530 ms TTFT | 611.6806 ms TTFT | 1,362.31,549 ms TTFT | 1,933.23,000 ms TTFT | 1,769.95,927 ms TTFT | 3,598.411,848 ms TTFT | 4,377.623,338 ms TTFT |
| 128K | 330.4680 ms TTFT | 43.8992 ms TTFT | 840.31,339 ms TTFT | 1,205.82,842 ms TTFT | 1,494.65,841 ms TTFT | 1,375.711,744 ms TTFT | 2,770.923,724 ms TTFT | Over capacity |
| 251K | 301.21,192 ms TTFT | 349.91,639 ms TTFT | 657.52,721 ms TTFT | 914.46,209 ms TTFT | 1,016.312,264 ms TTFT | 1,445.225,013 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 34K | — | 241 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 45.6K | — | 356 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 44.5K | — | 726 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 40.6K | — | 1,588 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 34.3K | — | 3,756 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 25.5K | — | 9,878 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 25.4K | — | 10,115 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 367.8 | 3548.8 | 102 ms | 0.3 ms | 1.0 | 3 | 11,008 | 29.9 s |
| Sustained decode | 8K | 2 | Valid | 648.6 | 514.6 | 152 ms | 2.3 ms | 2.0 | 4 | 16,033 | 24.7 s |
| Sustained decode | 8K | 4 | Valid | 1,209.7 | 2011.6 | 205 ms | 0.5 ms | 4.0 | 8 | 32,984 | 27.3 s |
| Sustained decode | 8K | 8 | Valid | 1,147.0 | 216.2 | 248 ms | 4.6 ms | 8.0 | 8 | 34,411 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,467.6 | 176.0 | 405 ms | 5.7 ms | 15.7 | 16 | 74,025 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,355.1 | 132.3 | 702 ms | 7.6 ms | 31.7 | 32 | 130,638 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,654.8 | 91.7 | 1,344 ms | 10.9 ms | 64.0 | 0 | 79,626 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,393.2 | 65.7 | 2,556 ms | 15.2 ms | 128.0 | 0 | 101,798 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 386.4 | 368.3 | 224 ms | 2.7 ms | 1.0 | 2 | 8,248 | 21.3 s |
| Sustained decode | 16K | 2 | Valid | 584.1 | 321.7 | 309 ms | 3.1 ms | 1.9 | 4 | 17,523 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,021.5 | 257.6 | 442 ms | 3.9 ms | 3.9 | 8 | 30,645 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,311.7 | 199.2 | 457 ms | 5.0 ms | 10.0 | 8 | 39,347 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,320.1 | 152.0 | 725 ms | 6.6 ms | 16.0 | 16 | 69,594 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,992.3 | 127.3 | 1,348 ms | 7.9 ms | 31.2 | 32 | 119,739 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,621.0 | 66.7 | 2,748 ms | 15.0 ms | 96.0 | 0 | 78,618 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,136.2 | 60.3 | 5,600 ms | 16.6 ms | 128.0 | 0 | 124,090 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 358.0 | 364.5 | 279 ms | 2.7 ms | 1.0 | 2 | 10,740 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 614.7 | 317.5 | 461 ms | 3.1 ms | 2.0 | 4 | 18,441 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 741.7 | 243.3 | 489 ms | 4.1 ms | 3.9 | 4 | 22,250 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,411.4 | 194.5 | 799 ms | 5.1 ms | 7.8 | 8 | 42,342 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,447.0 | 158.4 | 909 ms | 6.3 ms | 15.9 | 16 | 73,406 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,475.4 | 108.7 | 3,026 ms | 9.2 ms | 32.0 | 0 | 74,262 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,854.1 | 82.6 | 6,334 ms | 12.1 ms | 64.0 | 0 | 85,626 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 4,029.6 | 57.6 | 12,422 ms | 17.4 ms | 128.0 | 0 | 120,889 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 349.1 | 358.9 | 399 ms | 2.8 ms | 1.0 | 2 | 10,474 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 399.6 | 235.4 | 530 ms | 4.4 ms | 18.0 | 2 | 11,989 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 611.6 | 228.9 | 806 ms | 4.4 ms | 3.9 | 4 | 18,346 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,362.3 | 180.4 | 1,549 ms | 5.5 ms | 7.8 | 8 | 40,865 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,933.2 | 562.9 | 3,000 ms | 1.8 ms | 15.3 | 16 | 58,587 | 30.3 s |
| Sustained decode | 64K | 32 | Valid | 1,769.9 | 67.6 | 5,927 ms | 14.8 ms | 48.0 | 0 | 53,098 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,598.4 | 62.6 | 11,848 ms | 16.0 ms | 64.0 | 0 | 107,964 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 4,377.6 | 41.0 | 23,338 ms | 24.4 ms | 128.0 | 0 | 131,309 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 330.4 | 347.1 | 680 ms | 2.9 ms | 0.9 | 2 | 9,912 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 43.8 | 37.7 | 992 ms | 26.5 ms | 130.0 | 0 | 1,314 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 840.3 | 217.4 | 1,339 ms | 4.6 ms | 3.9 | 4 | 25,207 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,205.8 | 161.4 | 2,842 ms | 6.2 ms | 7.7 | 8 | 36,167 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,494.6 | 116.0 | 5,841 ms | 8.6 ms | 16.0 | 2 | 44,690 | 29.9 s |
| Sustained decode | 128K | 32 | Valid | 1,375.7 | 57.0 | 11,744 ms | 17.6 ms | 34.0 | 0 | 41,272 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,770.9 | 55.0 | 23,724 ms | 18.2 ms | 64.0 | 0 | 83,136 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 301.2 | 326.7 | 1,192 ms | 3.1 ms | 0.9 | 2 | 9,035 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 349.9 | 210.6 | 1,639 ms | 5.2 ms | 18.9 | 2 | 10,496 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 657.5 | 180.0 | 2,721 ms | 5.6 ms | 4.4 | 4 | 19,725 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 914.4 | 119.9 | 6,209 ms | 8.3 ms | 9.4 | 4 | 28,128 | 30.8 s |
| Sustained decode | 251K | 16 | Valid | 1,016.3 | 80.0 | 12,264 ms | 12.5 ms | 20.0 | 0 | 30,491 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,445.2 | 67.4 | 25,013 ms | 14.8 ms | 32.0 | 0 | 43,344 | 30.0 s |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.