Speed test matrix
INTELLECT-2 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9335+gdd15fb57b · FP164 contexts (8K–35K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 35K, 40K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9335+gdd15fb57b
- Quantization
- FP16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
dd15fb57b5ef
Run
- Served model
- intellect
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-primeintellect-sglang/bin/sglang" serve --model-path PrimeIntellect/INTELLECT-2 --served-model-name intellect --tp 1 --reasoning-parser deepseek-r1 --tool-call-parser hermes --context-length 40960 --mem-fraction-static 0.859667 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context7,9641,032 ms TTFT
16K context7,4152,194 ms TTFT
32K context6,4485,024 ms TTFT
35K context6,3255,600 ms TTFT
40K context6,1616,569 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 49.4Requests queued55 ms TTFT | 83.393 ms TTFT | 156.1111 ms TTFT | 304.4140 ms TTFT | 481.2188 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 48.170 ms TTFT | 57.1136 ms TTFT | 106.8157 ms TTFT | 192.5181 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 45.8Requests queued97 ms TTFT | 48.6174 ms TTFT | 92.0187 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 35K | 45.4Requests queued120 ms TTFT | 44.0170 ms TTFT | 82.0205 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 7,964.0 | — | 1,032 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 7,415.0 | — | 2,194 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,448.0 | — | 5,024 ms | — | — | 1 | — | — |
| Prefill | 35K | 1 | Valid | 6,325.0 | — | 5,600 ms | — | — | 1 | — | — |
| Prefill | 40K | 1 | Valid | 6,161.0 | — | 6,569 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Requests queued | 49.4 | 49.4 | 55 ms | 20.2 ms | 1.0 | 0 | 1,481 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 83.3 | 41.7 | 93 ms | 24.0 ms | 3.0 | 0 | 2,498 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 156.1 | 38.9 | 111 ms | 25.7 ms | 6.8 | 0 | 4,680 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 304.4 | 38.1 | 140 ms | 26.3 ms | 13.7 | 0 | 9,128 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 481.2 | 30.1 | 188 ms | 33.2 ms | 28.0 | 0 | 14,432 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 48.1 | 48.1 | 70 ms | 20.8 ms | 1.0 | 0 | 1,442 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 57.1 | 28.5 | 136 ms | 35.1 ms | 26.1 | 0 | 1,714 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 106.8 | 26.5 | 157 ms | 37.7 ms | 26.3 | 0 | 3,204 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 192.5 | 24.1 | 181 ms | 41.5 ms | 30.0 | 0 | 5,768 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Requests queued | 45.8 | 45.8 | 97 ms | 21.8 ms | 1.0 | 0 | 1,372 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 48.6 | 24.1 | 174 ms | 41.6 ms | 16.0 | 0 | 1,456 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 92.0 | 22.9 | 187 ms | 43.6 ms | 18.6 | 0 | 2,756 | 30.0 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 35K | 1 | Requests queued | 45.4 | 45.4 | 120 ms | 22.0 ms | 1.0 | 0 | 1,362 | 30.0 s |
| Sustained decode | 35K | 2 | Valid | 44.0 | 21.9 | 170 ms | 45.6 ms | 18.8 | 0 | 1,318 | 30.0 s |
| Sustained decode | 35K | 4 | Valid | 82.0 | 20.5 | 205 ms | 48.8 ms | 20.0 | 0 | 2,460 | 30.0 s |
| Sustained decode | 35K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 35K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 35K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 35K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 35K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.