Speed test matrix
DeepSeek-V4-Flash-DSpark on H200 NVL 141GB ×2
sglang · 0.5.16 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.16
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
Run
- Served model
- deepseek
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-deepseek-sglang/bin/sglang" serve --model-path deepseek-ai/DeepSeek-V4-Flash-DSpark --served-model-name deepseek --trust-remote-code --tp 2 --reasoning-parser deepseek-v4 --tool-call-parser deepseekv4 --context-length 1048576 --mem-fraction-static 0.808234 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --moe-runner-backend marlin --speculative-algorithm DSPARK
Prefill
prompt tok/s by context length8K context10.2K803 ms TTFT
16K context14.8K1,096 ms TTFT
32K context15K2,153 ms TTFT
64K context14.7K4,382 ms TTFT
128K context13.8K9,342 ms TTFT
256K context12K21,422 ms TTFT
512K context9,54454,018 ms TTFT
1019K context6,762151,709 ms TTFT
1024K context6,686154,185 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 228.4108 ms TTFT | 628.5133 ms TTFT | 681.5176 ms TTFT | 1,108.6249 ms TTFT | 1,878.4263 ms TTFT | 2,544.2842 ms TTFT | 3,714.5726 ms TTFT | 4,695.41,317 ms TTFT |
| 16K | 301.6135 ms TTFT | 492.3160 ms TTFT | 735.5195 ms TTFT | 1,175.5185 ms TTFT | 1,570.9392 ms TTFT | 2,320.2755 ms TTFT | 3,589.01,757 ms TTFT | 4,348.22,533 ms TTFT |
| 32K | 338.1158 ms TTFT | 524.7185 ms TTFT | 691.6216 ms TTFT | 1,055.8434 ms TTFT | 1,625.8765 ms TTFT | 2,264.21,463 ms TTFT | 3,505.02,873 ms TTFT | Over capacity |
| 64K | 245.1209 ms TTFT | 530.0245 ms TTFT | 643.8259 ms TTFT | 1,017.5771 ms TTFT | 1,765.81,583 ms TTFT | 2,306.03,091 ms TTFT | Over capacity | Over capacity |
| 128K | 272.0307 ms TTFT | 383.8510 ms TTFT | 761.4696 ms TTFT | 939.01,616 ms TTFT | 1,672.93,146 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 235.7531 ms TTFT | 417.6696 ms TTFT | 612.51,071 ms TTFT | 1,057.83,049 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 218.61,026 ms TTFT | 438.81,310 ms TTFT | 513.33,154 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 185.42,056 ms TTFT | 353.9Requests queued2,699 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.2K | — | 803 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14.8K | — | 1,096 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 15K | — | 2,153 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 14.7K | — | 4,382 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 13.8K | — | 9,342 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12K | — | 21,422 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 9,544.0 | — | 54,018 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 6,762.0 | — | 151,709 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 6,686.0 | — | 154,185 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 228.4 | 230.1 | 108 ms | 4.4 ms | 1.0 | 1 | 6,851 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 628.5 | 340.5 | 133 ms | 2.9 ms | 2.0 | 4 | 18,852 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 681.5 | 166.2 | 176 ms | 6.0 ms | 4.0 | 3 | 20,439 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,108.6 | 120.2 | 249 ms | 8.3 ms | 7.9 | 5 | 33,242 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,878.4 | 101.6 | 263 ms | 9.8 ms | 15.9 | 9 | 56,331 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,544.2 | 78.9 | 842 ms | 12.7 ms | 32.0 | 0 | 76,248 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,714.5 | 50.4 | 726 ms | 19.8 ms | 64.0 | 0 | 111,334 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,695.4 | 33.0 | 1,317 ms | 30.3 ms | 128.0 | 0 | 140,770 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 301.6 | 256.0 | 135 ms | 3.9 ms | 1.0 | 2 | 9,048 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 492.3 | 216.3 | 160 ms | 4.6 ms | 2.0 | 3 | 14,767 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 735.5 | 175.0 | 195 ms | 5.7 ms | 4.0 | 4 | 22,063 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,175.5 | 134.5 | 185 ms | 7.4 ms | 7.9 | 8 | 35,262 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,570.9 | 89.0 | 392 ms | 11.2 ms | 16.0 | 3 | 47,074 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,320.2 | 62.2 | 755 ms | 16.1 ms | 32.0 | 0 | 69,626 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,589.0 | 49.9 | 1,757 ms | 20.1 ms | 64.0 | 0 | 107,671 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,348.2 | 30.5 | 2,533 ms | 32.8 ms | 128.0 | 0 | 130,230 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 338.1 | 386.6 | 158 ms | 2.6 ms | 0.9 | 2 | 10,141 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 524.7 | 221.2 | 185 ms | 4.5 ms | 2.0 | 4 | 15,723 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 691.6 | 157.2 | 216 ms | 6.4 ms | 4.0 | 4 | 20,739 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,055.8 | 116.8 | 434 ms | 8.6 ms | 7.9 | 6 | 31,671 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,625.8 | 87.2 | 765 ms | 11.5 ms | 15.9 | 5 | 48,760 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,264.2 | 68.4 | 1,463 ms | 14.6 ms | 32.0 | 0 | 67,899 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,505.0 | 50.1 | 2,873 ms | 20.0 ms | 64.0 | 0 | 105,162 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 245.1 | 244.9 | 209 ms | 4.1 ms | 1.0 | 2 | 7,354 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 530.0 | 301.8 | 245 ms | 3.3 ms | 2.0 | 3 | 15,891 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 643.8 | 155.9 | 259 ms | 6.4 ms | 3.9 | 5 | 19,304 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,017.5 | 122.0 | 771 ms | 8.2 ms | 7.9 | 5 | 30,518 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,765.8 | 99.2 | 1,583 ms | 10.1 ms | 16.0 | 6 | 52,957 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,306.0 | 68.6 | 3,091 ms | 14.6 ms | 32.0 | 0 | 69,154 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 272.0 | 232.1 | 307 ms | 4.3 ms | 1.0 | 2 | 8,160 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 383.8 | 190.9 | 510 ms | 5.3 ms | 2.0 | 2 | 11,509 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 761.4 | 226.0 | 696 ms | 4.4 ms | 3.9 | 4 | 22,834 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 939.0 | 111.0 | 1,616 ms | 9.0 ms | 8.0 | 2 | 28,167 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,672.9 | 88.5 | 3,146 ms | 11.3 ms | 15.9 | 7 | 50,128 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 235.7 | 238.3 | 531 ms | 4.2 ms | 1.0 | 1 | 7,068 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 417.6 | 189.5 | 696 ms | 5.3 ms | 2.0 | 2 | 12,526 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 612.5 | 158.4 | 1,071 ms | 6.3 ms | 3.9 | 4 | 18,366 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 1,057.8 | 130.9 | 3,049 ms | 7.6 ms | 7.9 | 5 | 31,734 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 218.6 | 224.5 | 1,026 ms | 4.5 ms | 1.0 | 1 | 6,556 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 438.8 | 239.8 | 1,310 ms | 4.3 ms | 1.9 | 2 | 13,157 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 513.3 | 137.9 | 3,154 ms | 7.3 ms | 3.9 | 3 | 15,391 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 185.4 | 201.5 | 2,056 ms | 5.0 ms | 1.0 | 1 | 5,561 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 353.9 | 183.3 | 2,699 ms | 5.5 ms | 1.9 | 2 | 10,615 | 30.0 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.