Speed test matrix
limite-1b-base-soup on H200 NVL 141GB
vllm · 0.26.0 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.0
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- limite-1b-base-soup
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_PLUGINS=limite "$HOME/env_h200-paradigma-inc-vllm-v0260/bin/vllm" serve paradigma-inc/limite-1b-base-soup --served-model-name limite-1b-base-soup --tensor-parallel-size 1 --max-model-len 131072 --gpu-memory-utilization 0.86 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN
Prefill
prompt tok/s by context length8K context98.3K83 ms TTFT
16K context94.3K172 ms TTFT
32K context94.8K340 ms TTFT
64K context88.4K729 ms TTFT
123K context69.4K1,781 ms TTFT
128K context69.2K1,858 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 300.748 ms TTFT | 587.964 ms TTFT | 1,154.656 ms TTFT | 2,181.180 ms TTFT | 3,992.1180 ms TTFT | Looped | 11K273 ms TTFT | 15.3K822 ms TTFT |
| 16K | 297.285 ms TTFT | 580.365 ms TTFT | 1,127.8111 ms TTFT | 2,104.2147 ms TTFT | Looped | 6,261.5205 ms TTFT | Looped | Looped |
| 32K | 290.9114 ms TTFT | 563.6139 ms TTFT | Looped | Looped | Looped | Looped | Looped | Looped |
| 64K | 279.2170 ms TTFT | 535.3219 ms TTFT | 998.8251 ms TTFT | Looped | Looped | Looped | Looped | 5,999.35,649 ms TTFT |
| 123K | 255.0312 ms TTFT | Looped | 879.0364 ms TTFT | Looped | 2,058.2380 ms TTFT | Looped | Looped | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 98.3K | — | 83 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 94.3K | — | 172 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 94.8K | — | 340 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 88.4K | — | 729 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 69.4K | — | 1,781 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 69.2K | — | 1,858 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 300.7 | 301.9 | 48 ms | 3.3 ms | 1.0 | 2 | 9,020 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 587.9 | 295.5 | 64 ms | 3.4 ms | 1.9 | 4 | 17,637 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,154.6 | 289.6 | 56 ms | 3.5 ms | 4.0 | 8 | 34,638 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,181.1 | 274.5 | 80 ms | 3.6 ms | 8.0 | 16 | 65,432 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,992.1 | 252.6 | 180 ms | 4.0 ms | 15.9 | 32 | 119,760 | 30.0 s |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Valid | 11K | 175.6 | 273 ms | 5.7 ms | 64.0 | 64 | 331,393 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 15.3K | 120.0 | 822 ms | 8.3 ms | 128.0 | 17 | 458,147 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 297.2 | 299.0 | 85 ms | 3.3 ms | 1.0 | 2 | 8,917 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 580.3 | 291.6 | 65 ms | 3.4 ms | 2.0 | 4 | 17,408 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,127.8 | 284.2 | 111 ms | 3.5 ms | 4.0 | 8 | 33,833 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,104.2 | 265.5 | 147 ms | 3.8 ms | 7.7 | 16 | 63,121 | 30.0 s |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Valid | 6,261.5 | 196.5 | 205 ms | 5.1 ms | 32.0 | 32 | 187,850 | 30.0 s |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Valid | 290.9 | 293.2 | 114 ms | 3.4 ms | 1.0 | 2 | 8,726 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 563.6 | 284.6 | 139 ms | 3.5 ms | 2.0 | 4 | 16,908 | 30.0 s |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 1 | Valid | 279.2 | 282.6 | 170 ms | 3.5 ms | 1.0 | 2 | 8,374 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 535.3 | 272.2 | 219 ms | 3.7 ms | 2.0 | 4 | 16,057 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 998.8 | 253.8 | 251 ms | 3.9 ms | 3.9 | 8 | 29,963 | 30.0 s |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 128 | Valid | 5,999.3 | 46.8 | 5,649 ms | 21.3 ms | 128.0 | 0 | 179,968 | 30.0 s |
| Sustained decode | 123K | 1 | Valid | 255.0 | 260.4 | 312 ms | 3.8 ms | 1.0 | 2 | 7,649 | 30.0 s |
| Sustained decode | 123K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 4 | Valid | 879.0 | 222.1 | 364 ms | 4.5 ms | 4.0 | 4 | 26,369 | 30.0 s |
| Sustained decode | 123K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 16 | Valid | 2,058.2 | 130.5 | 380 ms | 7.7 ms | 15.8 | 16 | 61,784 | 30.0 s |
| Sustained decode | 123K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.