Speed test matrix
Nex-N2-mini on H200 SXM 141GB ×4
vllm · 0.29.0 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.29.0
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- nex-n2-mini
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nex-n2-vllm-v0290/bin/vllm" serve nex-agi/Nex-N2-mini --served-model-name nex-n2-mini --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --mamba-ssm-cache-dtype float32
Prefill
prompt tok/s by context length8K context42.1K195 ms TTFT
16K context49.9K325 ms TTFT
32K context52.2K619 ms TTFT
64K context63.1K1,021 ms TTFT
128K context61.8K2,083 ms TTFT
251K context51.6K4,890 ms TTFT
256K context48.4K5,316 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 313.972 ms TTFT | 567.0121 ms TTFT | 1,001.2210 ms TTFT | 1,725.3340 ms TTFT | 2,889.0432 ms TTFT | 4,549.2611 ms TTFT | 7,195.2870 ms TTFT | 11.5K1,421 ms TTFT |
| 16K | 308.096 ms TTFT | 557.2155 ms TTFT | 983.2229 ms TTFT | 1,678.8351 ms TTFT | 2,785.6512 ms TTFT | 4,167.4780 ms TTFT | 6,688.11,275 ms TTFT | 9,925.62,075 ms TTFT |
| 32K | 296.9146 ms TTFT | 537.9209 ms TTFT | 952.7241 ms TTFT | 1,603.5379 ms TTFT | 2,586.6532 ms TTFT | 3,825.9691 ms TTFT | 5,619.71,932 ms TTFT | 7,730.93,693 ms TTFT |
| 64K | 278.0245 ms TTFT | 506.1329 ms TTFT | 908.7383 ms TTFT | 1,459.4347 ms TTFT | 2,224.6348 ms TTFT | 3,213.22,176 ms TTFT | 4,334.84,082 ms TTFT | 5,159.78,549 ms TTFT |
| 128K | 249.1492 ms TTFT | 459.4540 ms TTFT | 810.8648 ms TTFT | 1,236.8522 ms TTFT | 1,808.12,042 ms TTFT | 2,363.24,004 ms TTFT | 2,799.97,949 ms TTFT | Over capacity |
| 251K | 204.6948 ms TTFT | 381.21,023 ms TTFT | 674.7996 ms TTFT | 956.41,862 ms TTFT | 1,303.04,621 ms TTFT | 1,492.38,815 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 42.1K | — | 195 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 49.9K | — | 325 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 52.2K | — | 619 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 63.1K | — | 1,021 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 61.8K | — | 2,083 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 51.6K | — | 4,890 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 48.4K | — | 5,316 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 313.9 | 315.5 | 72 ms | 3.2 ms | 1.0 | 2 | 9,418 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 567.0 | 285.9 | 121 ms | 3.5 ms | 2.0 | 4 | 17,012 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,001.2 | 254.1 | 210 ms | 3.9 ms | 4.0 | 8 | 30,035 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,725.3 | 218.3 | 340 ms | 4.6 ms | 7.7 | 8 | 51,759 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,889.0 | 183.2 | 432 ms | 5.5 ms | 16.0 | 16 | 86,672 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,549.2 | 145.5 | 611 ms | 6.9 ms | 31.5 | 32 | 136,494 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 7,195.2 | 112.3 | 870 ms | 8.9 ms | 64.0 | 0 | 215,936 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 11.5K | 88.8 | 1,421 ms | 11.3 ms | 128.0 | 0 | 343,808 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 308.0 | 310.1 | 96 ms | 3.2 ms | 1.0 | 2 | 9,238 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 557.2 | 281.8 | 155 ms | 3.5 ms | 1.9 | 4 | 16,716 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 983.2 | 249.8 | 229 ms | 4.0 ms | 4.0 | 8 | 29,496 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,678.8 | 212.0 | 351 ms | 4.7 ms | 8.0 | 8 | 50,361 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,785.6 | 177.0 | 512 ms | 5.6 ms | 15.9 | 16 | 83,569 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 4,167.4 | 133.2 | 780 ms | 7.5 ms | 32.0 | 32 | 125,026 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 6,688.1 | 103.1 | 1,275 ms | 9.7 ms | 64.0 | 0 | 200,640 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 9,925.6 | 75.0 | 2,075 ms | 13.3 ms | 128.0 | 0 | 297,728 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 296.9 | 300.1 | 146 ms | 3.3 ms | 1.0 | 2 | 8,907 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 537.9 | 273.1 | 209 ms | 3.7 ms | 2.0 | 4 | 16,138 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 952.7 | 242.6 | 241 ms | 4.1 ms | 3.9 | 8 | 28,562 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,603.5 | 202.8 | 379 ms | 4.9 ms | 8.0 | 8 | 48,112 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,586.6 | 163.7 | 532 ms | 6.1 ms | 16.0 | 16 | 77,601 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,825.9 | 119.6 | 691 ms | 8.4 ms | 32.0 | 23 | 114,678 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 5,619.7 | 86.1 | 1,932 ms | 11.6 ms | 64.0 | 0 | 168,576 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 7,730.9 | 57.4 | 3,693 ms | 17.4 ms | 128.0 | 0 | 231,936 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 278.0 | 282.7 | 245 ms | 3.5 ms | 1.0 | 2 | 8,339 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 506.1 | 258.4 | 329 ms | 3.9 ms | 2.0 | 4 | 15,184 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 908.7 | 229.9 | 383 ms | 4.4 ms | 4.0 | 4 | 27,257 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,459.4 | 184.2 | 347 ms | 5.4 ms | 7.9 | 8 | 43,782 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,224.6 | 140.4 | 348 ms | 7.1 ms | 15.9 | 16 | 66,732 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 3,213.2 | 98.9 | 2,176 ms | 10.1 ms | 32.0 | 0 | 96,544 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 4,334.8 | 66.2 | 4,082 ms | 15.1 ms | 64.0 | 0 | 130,048 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 5,159.7 | 40.0 | 8,549 ms | 25.0 ms | 128.0 | 0 | 154,880 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 249.1 | 253.9 | 492 ms | 3.9 ms | 1.0 | 2 | 7,390 | 29.7 s |
| Sustained decode | 128K | 2 | Valid | 459.4 | 234.4 | 540 ms | 4.3 ms | 2.0 | 2 | 13,778 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 810.8 | 206.9 | 648 ms | 4.8 ms | 3.9 | 4 | 24,324 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,236.8 | 157.1 | 522 ms | 6.4 ms | 7.9 | 8 | 37,102 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,808.1 | 112.8 | 2,042 ms | 8.9 ms | 16.0 | 3 | 54,119 | 29.9 s |
| Sustained decode | 128K | 32 | Valid | 2,363.2 | 73.0 | 4,004 ms | 13.7 ms | 32.0 | 0 | 70,880 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,799.9 | 43.0 | 7,949 ms | 23.3 ms | 64.0 | 0 | 83,968 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 204.6 | 212.2 | 948 ms | 4.7 ms | 1.0 | 1 | 6,139 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 381.2 | 197.5 | 1,023 ms | 5.1 ms | 2.0 | 2 | 11,435 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 674.7 | 174.3 | 996 ms | 5.7 ms | 3.9 | 4 | 20,241 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 956.4 | 122.2 | 1,862 ms | 8.2 ms | 7.8 | 6 | 28,595 | 29.9 s |
| Sustained decode | 251K | 16 | Valid | 1,303.0 | 81.4 | 4,621 ms | 12.3 ms | 16.0 | 0 | 39,090 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,492.3 | 46.5 | 8,815 ms | 21.5 ms | 32.0 | 0 | 44,770 | 30.0 s |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.