Speed test matrix
Laguna-XS-2.1 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9269+g834400705 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9269+g834400705
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
834400705f2d
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-poolside-sglang/bin/sglang" serve --model-path poolside/Laguna-XS-2.1 --served-model-name poolside --trust-remote-code --tp 1 --reasoning-parser poolside_v1 --tool-call-parser poolside_v1 --context-length 262144 --mem-fraction-static 0.858089 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --page-size 1 --attention-backend fa3 --speculative-algorithm DFLASH --speculative-draft-model-path poolside/Laguna-XS-2.1-DFlash
Prefill
prompt tok/s by context length8K context33K248 ms TTFT
16K context34.2K473 ms TTFT
32K context32K1,007 ms TTFT
64K context26.1K2,460 ms TTFT
128K context18.9K6,775 ms TTFT
251K context12.3K20,440 ms TTFT
256K context12.2K21,042 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 263.769 ms TTFT | 292.096 ms TTFT | 537.682 ms TTFT | 847.292 ms TTFT | 1,728.0200 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 290.483 ms TTFT | 325.4122 ms TTFT | 744.3109 ms TTFT | 1,215.5191 ms TTFT | 1,432.1330 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 215.6111 ms TTFT | 216.2141 ms TTFT | 713.7122 ms TTFT | 887.6353 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 228.7163 ms TTFT | 231.4239 ms TTFT | 496.1393 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 245.2272 ms TTFT | 232.9420 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 167.3471 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 33K | — | 248 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 34.2K | — | 473 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 32K | — | 1,007 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 26.1K | — | 2,460 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 18.9K | — | 6,775 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 12.3K | — | 20,440 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.2K | — | 21,042 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 263.7 | 211.2 | 69 ms | 4.7 ms | 1.0 | 2 | 7,911 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 292.0 | 165.3 | 96 ms | 6.1 ms | 2.6 | 2 | 8,760 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 537.6 | 134.1 | 82 ms | 7.5 ms | 4.4 | 4 | 16,122 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 847.2 | 100.1 | 92 ms | 10.0 ms | 11.4 | 2 | 25,400 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,728.0 | 78.7 | 200 ms | 12.7 ms | 18.3 | 7 | 51,807 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 290.4 | 221.3 | 83 ms | 4.5 ms | 0.9 | 2 | 8,710 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 325.4 | 144.3 | 122 ms | 7.0 ms | 11.4 | 2 | 9,758 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 744.3 | 191.4 | 109 ms | 5.2 ms | 5.1 | 4 | 22,327 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,215.5 | 137.4 | 191 ms | 7.3 ms | 9.5 | 6 | 36,453 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,432.1 | 74.0 | 330 ms | 13.5 ms | 19.5 | 2 | 42,964 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 215.6 | 221.1 | 111 ms | 4.6 ms | 1.0 | 1 | 6,467 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 216.2 | 107.2 | 141 ms | 9.4 ms | 8.3 | 0 | 6,485 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 713.7 | 139.9 | 122 ms | 7.1 ms | 3.9 | 5 | 21,406 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 887.6 | 84.5 | 353 ms | 11.8 ms | 11.2 | 3 | 26,622 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 228.7 | 226.9 | 163 ms | 4.4 ms | 1.0 | 1 | 6,860 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 231.4 | 112.9 | 239 ms | 8.9 ms | 5.9 | 0 | 6,941 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 496.1 | 122.8 | 393 ms | 8.1 ms | 4.0 | 3 | 14,877 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 245.2 | 197.6 | 272 ms | 5.1 ms | 0.9 | 2 | 7,353 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 232.9 | 120.2 | 420 ms | 8.3 ms | 4.6 | 1 | 6,984 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 167.3 | 171.4 | 471 ms | 5.8 ms | 1.0 | 1 | 5,017 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.