Speed test matrix
limite-1b-violetto on H200 NVL 141GB
vllm · 0.26.0 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.0
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- limite-1b-violetto
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_PLUGINS=limite "$HOME/env_h200-paradigma-inc-vllm-v0260/bin/vllm" serve paradigma-inc/limite-1b-violetto --served-model-name limite-1b-violetto --tensor-parallel-size 1 --reasoning-parser deepseek_r1 --max-model-len 131072 --gpu-memory-utilization 0.86 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN
Prefill
prompt tok/s by context length8K context15.9K515 ms TTFT
16K context31.4K516 ms TTFT
32K context62.3K517 ms TTFT
64K context88.9K723 ms TTFT
123K context69.9K1,768 ms TTFT
128K context67.2K1,913 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | Looped | 578.776 ms TTFT | Looped | Looped | Looped | Looped | Looped | Looped |
| 16K | Looped | Looped | Looped | Looped | Looped | Looped | Looped | Looped |
| 32K | Looped | Looped | Looped | Looped | Looped | Looped | Looped | Looped |
| 64K | Looped | 527.3243 ms TTFT | 986.2236 ms TTFT | Looped | Looped | Looped | Looped | Looped |
| 123K | 252.4309 ms TTFT | 477.2375 ms TTFT | Looped | Looped | Looped | Looped | Looped | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 15.9K | — | 515 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 31.4K | — | 516 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 62.3K | — | 517 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 88.9K | — | 723 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 69.9K | — | 1,768 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 67.2K | — | 1,913 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 2 | Valid | 578.7 | 291.1 | 76 ms | 3.4 ms | 2.0 | 4 | 17,362 | 30.0 s |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 2 | Valid | 527.3 | 268.2 | 243 ms | 3.7 ms | 2.0 | 4 | 15,818 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 986.2 | 250.7 | 236 ms | 4.0 ms | 3.9 | 8 | 29,597 | 30.0 s |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 1 | Valid | 252.4 | 257.7 | 309 ms | 3.9 ms | 1.0 | 2 | 7,573 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 477.2 | 245.2 | 375 ms | 4.1 ms | 2.0 | 4 | 14,316 | 30.0 s |
| Sustained decode | 123K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.