Speed test matrix
Laguna-XS-2.1-NVFP4 on H200 NVL 141GB
vllm · 0.26.0 · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.0
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-poolside-laguna-xs-vllm/bin/vllm" serve poolside/Laguna-XS-2.1-NVFP4 --served-model-name poolside --trust-remote-code --tensor-parallel-size 1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.881415 --host 0.0.0.0 --port 8000 --api-key <redacted> --reasoning-parser poolside_v1 --max-num-batched-tokens 32768 --speculative-config '{"model":"poolside/Laguna-XS-2.1-DFlash-NVFP4","num_speculative_tokens":15,"method":"dflash"}'Prefill
prompt tok/s by context length8K context33.2K247 ms TTFT
16K context32.5K498 ms TTFT
32K context30.3K1,062 ms TTFT
64K context26K2,470 ms TTFT
128K context19.5K6,586 ms TTFT
251K context12.9K19,490 ms TTFT
256K context12.7K20,235 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 124.652 ms TTFT | 224.0421 ms TTFT | 342.1125 ms TTFT | 572.6200 ms TTFT | 921.6259 ms TTFT | 1,152.4353 ms TTFT | 1,395.0533 ms TTFT | 1,618.0847 ms TTFT |
| 16K | 113.682 ms TTFT | 194.5103 ms TTFT | 298.0167 ms TTFT | 484.9184 ms TTFT | 760.2286 ms TTFT | 932.9485 ms TTFT | 1,100.5843 ms TTFT | 1,254.81,365 ms TTFT |
| 32K | 110.4106 ms TTFT | 185.8112 ms TTFT | 276.4201 ms TTFT | 435.3305 ms TTFT | 644.7474 ms TTFT | 766.8758 ms TTFT | 874.21,339 ms TTFT | Over capacity |
| 64K | 102.9173 ms TTFT | 170.9225 ms TTFT | 247.3321 ms TTFT | 362.2549 ms TTFT | 501.1784 ms TTFT | 568.51,449 ms TTFT | Over capacity | Over capacity |
| 128K | 94.5289 ms TTFT | 147.5359 ms TTFT | 202.2541 ms TTFT | 273.2861 ms TTFT | 341.71,648 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 80.7544 ms TTFT | 117.0793 ms TTFT | 147.51,243 ms TTFT | 183.61,894 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 33.2K | — | 247 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 32.5K | — | 498 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30.3K | — | 1,062 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 26K | — | 2,470 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 19.5K | — | 6,586 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 12.9K | — | 19,490 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.7K | — | 20,235 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 124.6 | 127.2 | 52 ms | 7.9 ms | 1.0 | 1 | 3,739 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 224.0 | 112.1 | 421 ms | 8.9 ms | 2.0 | 0 | 6,719 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 342.1 | 85.3 | 125 ms | 11.7 ms | 4.0 | 0 | 10,259 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 572.6 | 70.8 | 200 ms | 14.1 ms | 8.0 | 0 | 17,175 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 921.6 | 57.0 | 259 ms | 17.6 ms | 16.0 | 0 | 27,643 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,152.4 | 35.9 | 353 ms | 27.9 ms | 32.0 | 0 | 34,577 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,395.0 | 21.8 | 533 ms | 45.8 ms | 64.0 | 0 | 41,792 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 1,618.0 | 12.7 | 847 ms | 78.5 ms | 128.0 | 0 | 48,506 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 113.6 | 113.9 | 82 ms | 8.8 ms | 1.0 | 0 | 3,408 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 194.5 | 97.4 | 103 ms | 10.3 ms | 2.0 | 0 | 5,835 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 298.0 | 74.7 | 167 ms | 13.4 ms | 4.0 | 0 | 8,941 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 484.9 | 60.8 | 184 ms | 16.4 ms | 8.0 | 0 | 14,545 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 760.2 | 47.7 | 286 ms | 21.0 ms | 16.0 | 0 | 22,804 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 932.9 | 29.3 | 485 ms | 34.2 ms | 32.0 | 0 | 27,960 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,100.5 | 17.3 | 843 ms | 57.8 ms | 64.0 | 0 | 32,963 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 1,254.8 | 9.9 | 1,365 ms | 101.1 ms | 128.0 | 0 | 37,653 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 110.4 | 110.5 | 106 ms | 9.1 ms | 1.0 | 0 | 3,313 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 185.8 | 92.9 | 112 ms | 10.8 ms | 2.0 | 0 | 5,572 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 276.4 | 69.4 | 201 ms | 14.4 ms | 4.0 | 0 | 8,289 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 435.3 | 54.5 | 305 ms | 18.4 ms | 8.0 | 0 | 13,055 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 644.7 | 40.4 | 474 ms | 24.8 ms | 16.0 | 0 | 19,337 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 766.8 | 24.0 | 758 ms | 41.6 ms | 32.0 | 0 | 22,983 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 874.2 | 13.7 | 1,339 ms | 72.7 ms | 64.0 | 0 | 26,227 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 102.9 | 103.3 | 173 ms | 9.7 ms | 1.0 | 0 | 3,087 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 170.9 | 85.6 | 225 ms | 11.7 ms | 2.0 | 0 | 5,126 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 247.3 | 61.9 | 321 ms | 16.2 ms | 4.0 | 0 | 7,418 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 362.2 | 45.3 | 549 ms | 22.1 ms | 8.0 | 0 | 10,861 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 501.1 | 31.3 | 784 ms | 31.9 ms | 16.0 | 0 | 15,018 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 568.5 | 17.8 | 1,449 ms | 56.1 ms | 32.0 | 0 | 17,034 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 94.5 | 94.7 | 289 ms | 10.6 ms | 1.0 | 0 | 2,834 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 147.5 | 73.9 | 359 ms | 13.5 ms | 2.0 | 0 | 4,425 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 202.2 | 50.6 | 541 ms | 19.7 ms | 4.0 | 0 | 6,062 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 273.2 | 34.3 | 861 ms | 29.2 ms | 8.0 | 0 | 8,190 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 341.7 | 21.5 | 1,648 ms | 46.6 ms | 16.0 | 0 | 10,242 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 80.7 | 80.8 | 544 ms | 12.4 ms | 1.0 | 0 | 2,421 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 117.0 | 58.8 | 793 ms | 17.0 ms | 2.0 | 0 | 3,508 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 147.5 | 37.1 | 1,243 ms | 27.0 ms | 4.0 | 0 | 4,424 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 183.6 | 23.1 | 1,894 ms | 43.3 ms | 8.0 | 0 | 5,503 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.