Speed test matrix
Nanbeige4.2-3B on H200 NVL 141GB
vllm · 0.1.dev20429+g62f6de733 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev20429+g62f6de733
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/Nanbeige/vllm
- Build
- cuda-13.2
- Commit
62f6de733d7a
Run
- Served model
- nanbeige
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nanbeige-vllm/bin/vllm" serve Nanbeige/Nanbeige4.2-3B --served-model-name nanbeige --tensor-parallel-size 1 --reasoning-parser nanbeige --enable-auto-tool-choice --tool-call-parser nanbeige --max-model-len 262144 --gpu-memory-utilization 0.860223 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"dspark","model":"Nanbeige/Nanbeige4.2-3B-DSpark","num_speculative_tokens":7}'Prefill
prompt tok/s by context length8K context28.6K186 ms TTFT
16K context23.8K437 ms TTFT
32K context19.9K1,039 ms TTFT
64K context14.4K2,866 ms TTFT
128K context9,0259,109 ms TTFT
251K context5,21430,886 ms TTFT
256K context5,17731,722 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 318.866 ms TTFT | 623.061 ms TTFT | 1,165.167 ms TTFT | 2,047.073 ms TTFT | 2,964.4106 ms TTFT | 3,776.1292 ms TTFT | Over capacity | Over capacity |
| 16K | 363.182 ms TTFT | 593.397 ms TTFT | 1,072.8101 ms TTFT | 1,825.4111 ms TTFT | 2,509.2134 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 299.2119 ms TTFT | 583.8140 ms TTFT | 983.9130 ms TTFT | 1,421.0160 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 326.9187 ms TTFT | 491.1187 ms TTFT | 761.7219 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 214.4268 ms TTFT | 367.2336 ms TTFT | 521.0328 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 154.5575 ms TTFT | 250.1644 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 28.6K | — | 186 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 23.8K | — | 437 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 19.9K | — | 1,039 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 14.4K | — | 2,866 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,025.0 | — | 9,109 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,214.0 | — | 30,886 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,177.0 | — | 31,722 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 318.8 | 316.9 | 66 ms | 3.2 ms | 1.0 | 2 | 9,565 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 623.0 | 311.8 | 61 ms | 3.2 ms | 2.0 | 4 | 18,690 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,165.1 | 291.4 | 67 ms | 3.4 ms | 4.0 | 8 | 34,947 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,047.0 | 254.7 | 73 ms | 3.9 ms | 7.9 | 16 | 61,407 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,964.4 | 189.1 | 106 ms | 5.3 ms | 16.0 | 16 | 88,919 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,776.1 | 120.4 | 292 ms | 8.3 ms | 31.9 | 14 | 113,238 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 363.1 | 323.9 | 82 ms | 3.1 ms | 1.0 | 2 | 10,893 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 593.3 | 296.3 | 97 ms | 3.4 ms | 2.0 | 4 | 17,798 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,072.8 | 272.3 | 101 ms | 3.7 ms | 3.9 | 8 | 32,176 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,825.4 | 233.3 | 111 ms | 4.3 ms | 8.0 | 10 | 54,762 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,509.2 | 160.8 | 134 ms | 6.2 ms | 16.0 | 16 | 75,246 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 299.2 | 308.1 | 119 ms | 3.2 ms | 1.0 | 2 | 8,975 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 583.8 | 281.9 | 140 ms | 3.5 ms | 2.0 | 4 | 17,511 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 983.9 | 232.5 | 130 ms | 4.3 ms | 4.0 | 7 | 29,507 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,421.0 | 181.4 | 160 ms | 5.5 ms | 8.0 | 8 | 42,629 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 326.9 | 279.4 | 187 ms | 3.6 ms | 0.9 | 2 | 9,808 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 491.1 | 248.9 | 187 ms | 4.0 ms | 2.0 | 3 | 14,733 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 761.7 | 194.1 | 219 ms | 5.2 ms | 3.9 | 4 | 22,849 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 214.4 | 218.0 | 268 ms | 4.6 ms | 1.0 | 1 | 6,431 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 367.2 | 199.8 | 336 ms | 5.0 ms | 2.0 | 2 | 11,013 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 521.0 | 135.0 | 328 ms | 7.4 ms | 4.0 | 4 | 15,625 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 154.5 | 159.6 | 575 ms | 6.3 ms | 1.0 | 1 | 4,638 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 250.1 | 145.5 | 644 ms | 6.9 ms | 2.0 | 2 | 7,499 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.