Speed test matrix
Nanbeige4.2-3B-Base on H200 NVL 141GB
vllm · 0.1.dev20429+g62f6de733 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev20429+g62f6de733
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/Nanbeige/vllm
- Build
- cuda-13.2
- Commit
62f6de733d7a
Run
- Served model
- nanbeige
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nanbeige-vllm/bin/vllm" serve Nanbeige/Nanbeige4.2-3B-Base --served-model-name nanbeige --tensor-parallel-size 1 --max-model-len 131072 --gpu-memory-utilization 0.872774 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context30.8K171 ms TTFT
16K context24.2K428 ms TTFT
32K context20.2K1,022 ms TTFT
64K context14.5K2,835 ms TTFT
123K context9,3828,417 ms TTFT
128K context9,2158,917 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 169.349 ms TTFT | 325.864 ms TTFT | 621.786 ms TTFT | 1,094.9125 ms TTFT | 1,691.1168 ms TTFT | 2,511.7397 ms TTFT | Over capacity | Over capacity |
| 16K | 162.281 ms TTFT | 308.370 ms TTFT | 572.789 ms TTFT | 941.8222 ms TTFT | 1,409.2308 ms TTFT | 1,898.6455 ms TTFT | Over capacity | Over capacity |
| 32K | 150.5109 ms TTFT | 279.2106 ms TTFT | 489.0166 ms TTFT | 755.5260 ms TTFT | 1,027.6501 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 131.7154 ms TTFT | 235.8221 ms TTFT | 384.6336 ms TTFT | 524.8484 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 123K | 108.2247 ms TTFT | 185.4309 ms TTFT | 264.6515 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 30.8K | — | 171 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 24.2K | — | 428 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 20.2K | — | 1,022 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 14.5K | — | 2,835 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 9,382.0 | — | 8,417 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,215.0 | — | 8,917 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 169.3 | 170.4 | 49 ms | 5.9 ms | 1.0 | 1 | 5,080 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 325.8 | 164.7 | 64 ms | 6.1 ms | 2.0 | 2 | 9,775 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 621.7 | 158.1 | 86 ms | 6.3 ms | 4.0 | 4 | 18,655 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,094.9 | 141.8 | 125 ms | 7.1 ms | 7.8 | 8 | 32,847 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,691.1 | 108.0 | 168 ms | 9.3 ms | 16.0 | 0 | 50,736 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,511.7 | 80.3 | 397 ms | 12.4 ms | 32.0 | 0 | 75,360 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 162.2 | 163.4 | 81 ms | 6.1 ms | 1.0 | 1 | 4,864 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 308.3 | 155.7 | 70 ms | 6.4 ms | 2.0 | 2 | 9,248 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 572.7 | 145.7 | 89 ms | 6.9 ms | 4.0 | 4 | 17,181 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 941.8 | 118.9 | 222 ms | 8.4 ms | 8.0 | 0 | 28,248 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,409.2 | 89.3 | 308 ms | 11.2 ms | 16.0 | 0 | 42,272 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,898.6 | 60.2 | 455 ms | 16.6 ms | 32.0 | 0 | 56,961 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 150.5 | 152.2 | 109 ms | 6.6 ms | 1.0 | 1 | 4,513 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 279.2 | 141.9 | 106 ms | 7.0 ms | 2.0 | 2 | 8,377 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 489.0 | 125.9 | 166 ms | 7.9 ms | 4.0 | 4 | 14,669 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 755.5 | 94.9 | 260 ms | 10.5 ms | 8.0 | 0 | 22,665 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,027.6 | 64.9 | 501 ms | 15.4 ms | 16.0 | 0 | 30,848 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 131.7 | 134.6 | 154 ms | 7.4 ms | 1.0 | 1 | 3,949 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 235.8 | 118.3 | 221 ms | 8.5 ms | 2.0 | 0 | 7,072 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 384.6 | 96.5 | 336 ms | 10.4 ms | 4.0 | 0 | 11,536 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 524.8 | 65.8 | 484 ms | 15.2 ms | 8.0 | 0 | 15,744 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 108.2 | 108.6 | 247 ms | 9.2 ms | 1.0 | 0 | 3,247 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 185.4 | 93.0 | 309 ms | 10.8 ms | 2.0 | 0 | 5,560 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 264.6 | 66.4 | 515 ms | 15.1 ms | 4.0 | 0 | 7,940 | 30.0 s |
| Sustained decode | 123K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.