Speed test matrix
NVIDIA-Nemotron-3-Super-120B-A12B-BF16 on H200 NVL 141GB ×2
vllm · 0.26.1rc1.dev638+g52be12cfa · BF168 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- nemotron
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 "$HOME/env_h200-nvidia-vllm/bin/vllm" serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 --served-model-name nemotron --trust-remote-code --tensor-parallel-size 2 --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 1048576 --gpu-memory-utilization 0.875184 --host 0.0.0.0 --port 8000 --api-key <redacted> --async-scheduling --dtype auto --kv-cache-dtype fp8 --mamba-ssm-cache-dtype float32 --max-num-seqs 256 --max-cudagraph-capture-size 128 --enable-chunked-prefill
Prefill
prompt tok/s by context length8K context24.1K340 ms TTFT
16K context23.5K691 ms TTFT
32K context23K1,404 ms TTFT
64K context22.6K2,847 ms TTFT
128K context20.9K6,151 ms TTFT
256K context18.2K14,103 ms TTFT
512K context14.5K35,499 ms TTFT
1019K context10.2K100,059 ms TTFT
1024K context10.2K100,455 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 137.4329 ms TTFT | 230.4616 ms TTFT | 357.8806 ms TTFT | 566.31,431 ms TTFT | 833.72,705 ms TTFT | 1,274.95,570 ms TTFT | 1,611.9Requests queued9,559 ms TTFT | 1,557.5Requests queued9,617 ms TTFT |
| 16K | 137.0362 ms TTFT | 230.0512 ms TTFT | 357.2817 ms TTFT | 587.91,493 ms TTFT | 829.12,799 ms TTFT | 1,263.25,396 ms TTFT | 1,514.0Requests queued9,291 ms TTFT | 1,577.5Requests queued9,316 ms TTFT |
| 32K | 136.1382 ms TTFT | 230.4538 ms TTFT | 354.9854 ms TTFT | 559.71,574 ms TTFT | 827.52,915 ms TTFT | 1,245.65,292 ms TTFT | 1,491.6Requests queued9,036 ms TTFT | Over capacity |
| 64K | 134.5414 ms TTFT | 226.9542 ms TTFT | 352.4959 ms TTFT | 554.51,746 ms TTFT | 807.62,854 ms TTFT | 1,214.05,056 ms TTFT | Over capacity | Over capacity |
| 128K | 131.2453 ms TTFT | 222.7554 ms TTFT | 347.3785 ms TTFT | 542.81,369 ms TTFT | 778.72,228 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 123.11,005 ms TTFT | 213.71,229 ms TTFT | 335.91,764 ms TTFT | 512.02,756 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 116.81,685 ms TTFT | 199.42,023 ms TTFT | 315.23,035 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 100.73,971 ms TTFT | 175.44,166 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 24.1K | — | 340 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 23.5K | — | 691 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 23K | — | 1,404 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 22.6K | — | 2,847 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 20.9K | — | 6,151 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 18.2K | — | 14,103 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 14.5K | — | 35,499 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 10.2K | — | 100,059 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 10.2K | — | 100,455 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 137.4 | 139.3 | 329 ms | 7.2 ms | 1.0 | 1 | 4,122 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 230.4 | 114.5 | 616 ms | 8.7 ms | 2.0 | 0 | 6,912 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 357.8 | 88.3 | 806 ms | 11.3 ms | 4.0 | 0 | 10,732 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 566.3 | 69.0 | 1,431 ms | 14.5 ms | 8.0 | 0 | 16,983 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 833.7 | 49.4 | 2,705 ms | 20.3 ms | 16.0 | 0 | 25,005 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,274.9 | 35.4 | 5,570 ms | 28.3 ms | 32.0 | 0 | 38,239 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 1,611.9 | 29.1 | 9,559 ms | 34.4 ms | 48.0 | 0 | 48,332 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 1,557.5 | 28.7 | 9,617 ms | 34.8 ms | 48.0 | 0 | 46,705 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 137.0 | 138.9 | 362 ms | 7.2 ms | 1.0 | 1 | 4,109 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 230.0 | 114.6 | 512 ms | 8.7 ms | 2.0 | 0 | 6,898 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 357.2 | 88.2 | 817 ms | 11.3 ms | 4.0 | 0 | 10,717 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 587.9 | 71.5 | 1,493 ms | 14.0 ms | 8.0 | 0 | 17,639 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 829.1 | 49.3 | 2,799 ms | 20.3 ms | 16.0 | 0 | 24,863 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,263.2 | 35.5 | 5,396 ms | 28.2 ms | 32.0 | 0 | 37,888 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 1,514.0 | 28.7 | 9,291 ms | 34.8 ms | 48.0 | 0 | 45,404 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 1,577.5 | 29.2 | 9,316 ms | 34.2 ms | 48.0 | 0 | 47,322 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 136.1 | 138.5 | 382 ms | 7.2 ms | 1.0 | 1 | 4,082 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 230.4 | 114.8 | 538 ms | 8.7 ms | 2.0 | 0 | 6,914 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 354.9 | 87.8 | 854 ms | 11.4 ms | 4.0 | 0 | 10,646 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 559.7 | 68.6 | 1,574 ms | 14.6 ms | 8.0 | 0 | 16,792 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 827.5 | 49.5 | 2,915 ms | 20.2 ms | 16.0 | 0 | 24,830 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,245.6 | 35.2 | 5,292 ms | 28.4 ms | 32.0 | 0 | 37,347 | 30.0 s |
| Sustained decode | 32K | 64 | Requests queued | 1,491.6 | 27.6 | 9,036 ms | 36.2 ms | 48.0 | 0 | 44,740 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 134.5 | 137.3 | 414 ms | 7.3 ms | 1.0 | 1 | 4,037 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 226.9 | 113.1 | 542 ms | 8.8 ms | 2.0 | 0 | 6,806 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 352.4 | 87.5 | 959 ms | 11.4 ms | 4.0 | 0 | 10,572 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 554.5 | 68.6 | 1,746 ms | 14.6 ms | 8.0 | 0 | 16,640 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 807.6 | 48.7 | 2,854 ms | 20.5 ms | 16.0 | 0 | 24,223 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,214.0 | 34.7 | 5,056 ms | 28.8 ms | 32.0 | 0 | 36,418 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 131.2 | 135.5 | 453 ms | 7.4 ms | 1.0 | 1 | 3,937 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 222.7 | 111.2 | 554 ms | 9.0 ms | 2.0 | 0 | 6,682 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 347.3 | 86.2 | 785 ms | 11.6 ms | 4.0 | 0 | 10,416 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 542.8 | 67.3 | 1,369 ms | 14.9 ms | 8.0 | 0 | 16,287 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 778.7 | 47.4 | 2,228 ms | 21.1 ms | 16.0 | 0 | 23,361 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 123.1 | 142.2 | 1,005 ms | 7.1 ms | 1.0 | 1 | 3,694 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 213.7 | 106.4 | 1,229 ms | 9.4 ms | 2.0 | 0 | 6,410 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 335.9 | 82.5 | 1,764 ms | 12.1 ms | 4.0 | 0 | 10,079 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 512.0 | 61.5 | 2,756 ms | 16.3 ms | 8.0 | 0 | 15,360 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 116.8 | 117.4 | 1,685 ms | 8.5 ms | 1.0 | 0 | 3,504 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 199.4 | 99.3 | 2,023 ms | 10.1 ms | 2.0 | 0 | 5,984 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 315.2 | 77.1 | 3,035 ms | 13.0 ms | 4.0 | 0 | 9,453 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 100.7 | 101.7 | 3,971 ms | 9.8 ms | 1.0 | 0 | 3,020 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 175.4 | 87.1 | 4,166 ms | 11.5 ms | 2.0 | 0 | 5,260 | 30.0 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.