Speed test matrix
Nanbeige4.2-3B on H200 NVL 141GB
vllm · 0.1.dev20429+g62f6de733 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev20429+g62f6de733
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/Nanbeige/vllm
- Build
- cuda-13.2
- Commit
62f6de733d7a
Run
- Served model
- nanbeige
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nanbeige-vllm/bin/vllm" serve Nanbeige/Nanbeige4.2-3B --served-model-name nanbeige --tensor-parallel-size 1 --reasoning-parser nanbeige --enable-auto-tool-choice --tool-call-parser nanbeige --max-model-len 262144 --gpu-memory-utilization 0.868238 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context29.5K179 ms TTFT
16K context24.2K432 ms TTFT
32K context20K1,034 ms TTFT
64K context14.3K2,874 ms TTFT
128K context9,0739,061 ms TTFT
251K context5,23230,778 ms TTFT
256K context5,20731,543 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 164.648 ms TTFT | 316.976 ms TTFT | 603.792 ms TTFT | 1,065.1110 ms TTFT | 1,658.8202 ms TTFT | 2,476.2324 ms TTFT | Over capacity | Over capacity |
| 16K | 157.783 ms TTFT | 300.397 ms TTFT | 557.782 ms TTFT | 921.5254 ms TTFT | 1,385.0291 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 146.4116 ms TTFT | 271.3111 ms TTFT | 477.5178 ms TTFT | 741.7325 ms TTFT | 1,015.0534 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 128.7148 ms TTFT | 229.9210 ms TTFT | 378.2306 ms TTFT | 519.8440 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 104.5255 ms TTFT | 176.8303 ms TTFT | 255.9512 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 75.9562 ms TTFT | 120.6763 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 29.5K | — | 179 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 24.2K | — | 432 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 20K | — | 1,034 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 14.3K | — | 2,874 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,073.0 | — | 9,061 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,232.0 | — | 30,778 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,207.0 | — | 31,543 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 164.6 | 165.7 | 48 ms | 6.0 ms | 1.0 | 1 | 4,939 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 316.9 | 160.3 | 76 ms | 6.2 ms | 1.9 | 2 | 9,506 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 603.7 | 153.8 | 92 ms | 6.5 ms | 4.0 | 4 | 18,110 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,065.1 | 138.1 | 110 ms | 7.2 ms | 8.0 | 8 | 31,952 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,658.8 | 105.8 | 202 ms | 9.4 ms | 16.0 | 0 | 49,760 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,476.2 | 79.1 | 324 ms | 12.6 ms | 32.0 | 0 | 74,267 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 157.7 | 159.0 | 83 ms | 6.3 ms | 1.0 | 1 | 4,731 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 300.3 | 151.9 | 97 ms | 6.6 ms | 2.0 | 2 | 9,013 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 557.7 | 141.8 | 82 ms | 7.1 ms | 4.0 | 4 | 16,730 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 921.5 | 116.3 | 254 ms | 8.6 ms | 8.0 | 0 | 27,637 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,385.0 | 87.7 | 291 ms | 11.4 ms | 16.0 | 0 | 41,552 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 146.4 | 148.3 | 116 ms | 6.7 ms | 1.0 | 1 | 4,393 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 271.3 | 137.9 | 111 ms | 7.3 ms | 2.0 | 2 | 8,138 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 477.5 | 127.8 | 178 ms | 7.9 ms | 4.0 | 4 | 14,323 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 741.7 | 93.2 | 325 ms | 10.7 ms | 8.0 | 0 | 22,248 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,015.0 | 64.1 | 534 ms | 15.6 ms | 16.0 | 0 | 30,444 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 128.7 | 132.1 | 148 ms | 7.6 ms | 1.0 | 1 | 3,860 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 229.9 | 115.3 | 210 ms | 8.7 ms | 2.0 | 0 | 6,896 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 378.2 | 94.9 | 306 ms | 10.5 ms | 4.0 | 0 | 11,344 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 519.8 | 65.2 | 440 ms | 15.3 ms | 8.0 | 0 | 15,592 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 104.5 | 104.9 | 255 ms | 9.5 ms | 1.0 | 0 | 3,135 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 176.8 | 88.6 | 303 ms | 11.3 ms | 2.0 | 0 | 5,306 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 255.9 | 64.2 | 512 ms | 15.6 ms | 4.0 | 0 | 7,676 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 75.9 | 76.2 | 562 ms | 13.1 ms | 1.0 | 0 | 2,277 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 120.6 | 60.6 | 763 ms | 16.5 ms | 2.0 | 0 | 3,618 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.