InferCrane
Serving release View files ↗

Recorded H200 qualification

Same weights.
1.72× more useful work.

A qualified serving recipe for Qwen3.8-27B-FP8. InferCrane searched the runtime path, measured it against a paired control, and kept the winner only after output and API checks passed.

1× H200 SGLang 0.5.20 3,919 → 1,377 tokens Streaming

Paired comparison

Concurrency 12

recorded, not live
Selected1,393.6
aggregate output tok/s
Control810.4
same image, model, workload, GPU
Per-request speed
+85.3%
Median TTFT
−53.6%
GPU output cost
−41.9%

Throughput frontier

Choose the load, inspect the trade-off.

Aggregate throughput rises with concurrency. Per-request speed and tail latency tell us where productive utilization becomes queueing.

1,393.6 aggregate output tok/s
Per request
141.0 tok/s
TTFT p50
739 ms
TTFT p95
2,472 ms
ITL p95
8.67 ms

Release path

The artifact is more than a command line.

Weights are unchanged. The release pins the surrounding execution system and keeps the evidence boundary explicit.

  1. 01
    Pin identity

    Checkpoint revision, runtime digest, GPU and workload.

  2. 02
    Preserve correctness

    Apply the fail-closed GDN FP32-state patch to both candidates.

  3. 03
    Search execution

    FP8 KV, NEXTN/MTP, bounded graphs and runtime-selected kernels.

  4. 04
    Measure and reject

    Streaming usage, token parity, latency, cost and SLO gates.

Reproduce

One pinned container. Two comparable paths.

Run selected and control from the same image, then generate a machine-readable benchmark receipt.

H200 host
git clone https://huggingface.co/infercrane/qwen3.8-27b-h200-serving-recipe
cd qwen3.8-27b-h200-serving-recipe
bash serve.sh selected
bash benchmark.sh --recipe-label selected
Evidence boundary

These are loopback measurements from a controlled Modal H200 run. They do not include public-network latency and are not an OpenRouter or cross-provider leaderboard result. Changing the model revision, runtime, GPU or workload requires requalification.