Speed test matrix
Laguna-M.1-NVFP4 on H200 NVL 141GB ×2
vllm · 0.27.1 · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.1
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-poolside-vllm/bin/vllm" serve poolside/Laguna-M.1-NVFP4 --served-model-name poolside --trust-remote-code --tensor-parallel-size 2 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.863994 --host 0.0.0.0 --port 8000 --api-key <redacted> --reasoning-parser poolside_v1
Prefill
prompt tok/s by context length8K context8,929915 ms TTFT
16K context8,4111,923 ms TTFT
32K context7,5414,269 ms TTFT
64K context6,21710,330 ms TTFT
128K context4,49828,516 ms TTFT
251K context2,94485,375 ms TTFT
256K context2,89588,548 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 79.157 ms TTFT | 140.983 ms TTFT | 240.6136 ms TTFT | 374.4182 ms TTFT | 570.7281 ms TTFT | 872.0383 ms TTFT | Over capacity | Over capacity |
| 16K | 77.8109 ms TTFT | 137.4110 ms TTFT | 226.6169 ms TTFT | 358.0214 ms TTFT | 523.8310 ms TTFT | 752.8585 ms TTFT | Over capacity | Over capacity |
| 32K | 75.2129 ms TTFT | 130.5149 ms TTFT | 210.3218 ms TTFT | 309.1350 ms TTFT | 430.8516 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 70.3191 ms TTFT | 117.2249 ms TTFT | 180.4296 ms TTFT | 250.0496 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 62.3291 ms TTFT | 97.6380 ms TTFT | 138.2579 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 51.3567 ms TTFT | 73.9800 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 8,929.0 | — | 915 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 8,411.0 | — | 1,923 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 7,541.0 | — | 4,269 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 6,217.0 | — | 10,330 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,498.0 | — | 28,516 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 2,944.0 | — | 85,375 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 2,895.0 | — | 88,548 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 79.1 | 79.2 | 57 ms | 12.6 ms | 1.0 | 0 | 2,373 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 140.9 | 70.5 | 83 ms | 14.2 ms | 2.0 | 0 | 4,224 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 240.6 | 60.4 | 136 ms | 16.6 ms | 4.0 | 0 | 7,216 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 374.4 | 47.1 | 182 ms | 21.2 ms | 8.0 | 0 | 11,224 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 570.7 | 36.0 | 281 ms | 27.8 ms | 16.0 | 0 | 17,121 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 872.0 | 27.4 | 383 ms | 36.5 ms | 32.0 | 0 | 26,144 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 77.8 | 77.8 | 109 ms | 12.8 ms | 1.0 | 0 | 2,334 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 137.4 | 68.8 | 110 ms | 14.5 ms | 2.0 | 0 | 4,122 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 226.6 | 56.7 | 169 ms | 17.6 ms | 4.0 | 0 | 6,796 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 358.0 | 45.0 | 214 ms | 22.2 ms | 8.0 | 0 | 10,736 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 523.8 | 32.9 | 310 ms | 30.4 ms | 16.0 | 0 | 15,714 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 752.8 | 23.6 | 585 ms | 42.3 ms | 32.0 | 0 | 22,560 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 75.2 | 75.3 | 129 ms | 13.3 ms | 1.0 | 0 | 2,255 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 130.5 | 65.4 | 149 ms | 15.3 ms | 2.0 | 0 | 3,914 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 210.3 | 52.7 | 218 ms | 19.0 ms | 4.0 | 0 | 6,308 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 309.1 | 38.8 | 350 ms | 25.8 ms | 8.0 | 0 | 9,272 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 430.8 | 27.0 | 516 ms | 37.0 ms | 16.0 | 0 | 12,912 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 70.3 | 70.4 | 191 ms | 14.2 ms | 1.0 | 0 | 2,108 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 117.2 | 58.7 | 249 ms | 17.0 ms | 2.0 | 0 | 3,516 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 180.4 | 45.2 | 296 ms | 22.1 ms | 4.0 | 0 | 5,412 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 250.0 | 31.3 | 496 ms | 31.9 ms | 8.0 | 0 | 7,496 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 62.3 | 62.4 | 291 ms | 16.0 ms | 1.0 | 0 | 1,869 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 97.6 | 48.9 | 380 ms | 20.4 ms | 2.0 | 0 | 2,928 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 138.2 | 34.7 | 579 ms | 28.9 ms | 4.0 | 0 | 4,144 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 51.3 | 51.5 | 567 ms | 19.4 ms | 1.0 | 0 | 1,539 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 73.9 | 37.1 | 800 ms | 26.9 ms | 2.0 | 0 | 2,218 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.