Speed test matrix
Step-3.7-Flash-FP8 on H200 NVL 141GB ×2
vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- step
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-stepfun-vllm/bin/vllm" serve stepfun-ai/Step-3.7-Flash-FP8 --served-model-name step --trust-remote-code --tensor-parallel-size 2 --reasoning-parser step3p5 --enable-auto-tool-choice --tool-call-parser step3p5 --max-model-len 262144 --gpu-memory-utilization 0.863334 --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-expert-parallel --disable-cascade-attn
Prefill
prompt tok/s by context length8K context24.6K332 ms TTFT
16K context23.4K694 ms TTFT
32K context22.6K1,431 ms TTFT
64K context20.4K3,153 ms TTFT
128K context16.4K7,831 ms TTFT
251K context11.7K21,568 ms TTFT
256K context11.6K22,217 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 112.048 ms TTFT | 191.590 ms TTFT | 343.5112 ms TTFT | 593.2162 ms TTFT | 972.0232 ms TTFT | 1,519.8342 ms TTFT | 2,443.8529 ms TTFT | Over capacity |
| 16K | 111.387 ms TTFT | 190.3108 ms TTFT | 339.8143 ms TTFT | 580.6177 ms TTFT | 941.6309 ms TTFT | 1,460.7477 ms TTFT | Over capacity | Over capacity |
| 32K | 109.9113 ms TTFT | 187.2142 ms TTFT | 333.5176 ms TTFT | 560.7304 ms TTFT | 900.3477 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 107.8164 ms TTFT | 181.7224 ms TTFT | 319.7352 ms TTFT | 524.0583 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 103.3300 ms TTFT | 172.2434 ms TTFT | 297.5589 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 95.9574 ms TTFT | 156.0913 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 24.6K | — | 332 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 23.4K | — | 694 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 22.6K | — | 1,431 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 20.4K | — | 3,153 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 16.4K | — | 7,831 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 11.7K | — | 21,568 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 11.6K | — | 22,217 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 112.0 | 112.0 | 48 ms | 8.9 ms | 1.0 | 0 | 3,358 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 191.5 | 95.8 | 90 ms | 10.4 ms | 2.0 | 0 | 5,744 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 343.5 | 85.9 | 112 ms | 11.6 ms | 4.0 | 0 | 10,304 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 593.2 | 74.3 | 162 ms | 13.5 ms | 8.0 | 0 | 17,792 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 972.0 | 61.1 | 232 ms | 16.4 ms | 16.0 | 0 | 29,152 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,519.8 | 47.9 | 342 ms | 20.9 ms | 32.0 | 0 | 45,600 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,443.8 | 38.6 | 529 ms | 25.9 ms | 64.0 | 0 | 73,280 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 111.3 | 111.3 | 87 ms | 9.0 ms | 1.0 | 0 | 3,337 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 190.3 | 95.2 | 108 ms | 10.5 ms | 2.0 | 0 | 5,708 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 339.8 | 85.1 | 143 ms | 11.8 ms | 4.0 | 0 | 10,192 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 580.6 | 72.8 | 177 ms | 13.7 ms | 8.0 | 0 | 17,416 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 941.6 | 59.1 | 309 ms | 16.9 ms | 16.0 | 0 | 28,248 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,460.7 | 46.0 | 477 ms | 21.7 ms | 32.0 | 0 | 43,808 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 109.9 | 109.9 | 113 ms | 9.1 ms | 1.0 | 0 | 3,296 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 187.2 | 93.7 | 142 ms | 10.7 ms | 2.0 | 0 | 5,616 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 333.5 | 83.4 | 176 ms | 12.0 ms | 4.0 | 0 | 10,000 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 560.7 | 70.2 | 304 ms | 14.3 ms | 8.0 | 0 | 16,819 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 900.3 | 56.4 | 477 ms | 17.7 ms | 16.0 | 0 | 27,008 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 107.8 | 107.9 | 164 ms | 9.3 ms | 1.0 | 0 | 3,232 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 181.7 | 91.0 | 224 ms | 11.0 ms | 2.0 | 0 | 5,452 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 319.7 | 80.1 | 352 ms | 12.5 ms | 4.0 | 0 | 9,592 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 524.0 | 65.7 | 583 ms | 15.2 ms | 8.0 | 0 | 15,720 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 103.3 | 103.5 | 300 ms | 9.7 ms | 1.0 | 0 | 3,100 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 172.2 | 86.3 | 434 ms | 11.6 ms | 2.0 | 0 | 5,164 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 297.5 | 74.5 | 589 ms | 13.4 ms | 4.0 | 0 | 8,924 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 95.9 | 96.2 | 574 ms | 10.4 ms | 1.0 | 0 | 2,878 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 156.0 | 78.3 | 913 ms | 12.8 ms | 2.0 | 0 | 4,678 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.