> For the complete documentation index, see [llms.txt](https://seekspeed.gitbook.io/seekspeed-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://seekspeed.gitbook.io/seekspeed-docs/spec-lab/how-we-benchmark-deepspec.md).

# How We Benchmark DeepSpec

This page is the long-form companion to [DeepSpec Integration](/seekspeed-docs/spec-lab/deepspec.md) and [Bottleneck Analysis](/seekspeed-docs/spec-lab/bottlenecks.md). It documents *exactly* what SeekSpeed reads off a [DeepSpec](https://github.com/deepseek-ai/DeepSpec/tree/main) endpoint, how the acceptance breakeven math is derived, and how to read the Welch's *t*-test output the runner produces.

If a number appears in the Spec Lab UI, it came from one of the three stages below.

***

## Stage 1 — Counter probe (`/metrics`)

Every DeepSpec / DSpark run starts with a probe against vLLM's Prometheus `/metrics` endpoint. Three counter families are the only ones that matter for speculative decoding:

```
vllm:spec_decode_num_draft_tokens_total
vllm:spec_decode_num_accepted_tokens_total
vllm:spec_decode_num_emitted_tokens_total
```

The parser lives in `src/lib/spec-lab.functions.ts` and is intentionally strict — a missing counter means the server is not actually running speculative decoding, and we surface that instead of silently returning `0`.

```ts
// src/lib/spec-lab.functions.ts (paraphrased)
const get = (k: string) => {
  const m = text.match(new RegExp(`^${k}\\s+([0-9eE+\\-.]+)`, "m"));
  return m ? Number(m[1]) : NaN;
};

const drafted  = get("vllm:spec_decode_num_draft_tokens_total");
const accepted = get("vllm:spec_decode_num_accepted_tokens_total");
const emitted  = get("vllm:spec_decode_num_emitted_tokens_total");

const acceptanceRate = drafted  > 0 ? accepted / drafted  : undefined;
const tokensPerStep  = accepted > 0 ? emitted  / accepted : undefined;
```

### What each counter means

| Counter                     | What it counts                                                                                      | Why we read it                                                                      |
| --------------------------- | --------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| `num_draft_tokens_total`    | Every token the draft model *proposed*                                                              | Denominator of acceptance rate. Also the draft-side compute cost.                   |
| `num_accepted_tokens_total` | Draft tokens the target model *accepted* verbatim                                                   | Numerator of acceptance rate. The direct measure of draft quality.                  |
| `num_emitted_tokens_total`  | Total tokens actually returned to the user (accepted + bonus tokens from target's own forward pass) | Divided by accepted, gives tokens-per-target-step — the real throughput multiplier. |

### Derived quantities

```
α (alpha)       = accepted / drafted                     # acceptance rate
tokensPerStep   = emitted  / accepted                    # avg tokens per target forward-pass
observedSpeedup = specThroughput / baselineThroughput    # end-to-end, measured
```

These four numbers — `α`, `tokensPerStep`, `observedSpeedup`, and TTFT (measured client-side, see stage 2) — are the entire input to the Bottleneck Panel.

***

## Stage 2 — Two-endpoint A/B run

DeepSpec's marketing number ("60–85% faster") is a *ratio*. Ratios need two measurements. SeekSpeed refuses to claim a speedup without a `baselineEndpoint` on the job:

```ts
if (!data.baselineEndpoint) {
  log("warn", "No baseline endpoint — speedup will not be measurable. Running spec endpoint only.");
}
```

When both endpoints are configured, we run the same prompt set through both and record per-iteration latency distributions. TTFT is captured by parsing the SSE stream and stopping the clock on the first `data: {"choices":[{"delta":{"content":"..."}}]}` frame; total latency is `end - start` on the whole stream.

Recommended defaults:

* **Sample count**: 32 minimum (the `n ≥ 5` gate refuses to draw a significance badge below that).
* **Max tokens**: 256 (long enough that decode dominates, short enough that variance is manageable).
* **Concurrency**: 1 first, then rerun at 4 to populate the batch-pressure card.
* **Same prompts on both endpoints, same order.**

The output of stage 2 is two arrays: `baselineLatencies` and `specLatencies`, each of length `n`.

***

## Stage 3 — Statistical rigor

We never report a mean speedup on its own. Every A/B run in `src/lib/spec-lab.functions.ts` calls `welchTTest` from `src/lib/stats.ts`:

```ts
export interface WelchResult {
  meanA: number;      // baseline mean latency (ms)
  meanB: number;      // spec mean latency (ms)
  delta: number;      // meanB - meanA
  pctChange: number;  // (meanB - meanA) / meanA
  t: number;          // Welch's t statistic
  df: number;         // Welch-Satterthwaite degrees of freedom
  pValue: number;     // two-tailed p, via regularized incomplete beta
  significant: boolean; // pValue < 0.05
}
```

### How to read the output

Pattern-match against these four cases in order:

1. **`n < 5` per side** → the UI omits the badge entirely. Rerun with more iterations. Do not eyeball the difference.
2. **`pValue >= 0.05`** → the observed speedup is indistinguishable from noise at 95% confidence. Even if `pctChange` looks large, treat it as "no effect measured" and increase `n` before drawing conclusions.
3. **`pValue < 0.05` and `pctChange < 0`** → spec is *faster* on the mean (lower latency). This is what "DeepSpec worked" looks like. Also check the p95 delta — a mean-only win with a p99 regression is a bad trade for real-time agents.
4. **`pValue < 0.05` and `pctChange > 0`** → spec is *slower*. Common when α < 0.4 or prefill share > 60%. Cross-reference the Bottleneck Panel; the acceptance card or prefill card will usually be red.

### Worked example

```
baseline (n=32): mean 1420 ms, σ 180 ms
spec     (n=32): mean 1180 ms, σ 150 ms

t   ≈ (1420 − 1180) / sqrt(180²/32 + 150²/32)  ≈ 5.79
df  ≈ 60.1
p   ≈ 2.3e-7
Δ   = −240 ms  (spec is 16.9% faster on the mean)
```

Because `p < 0.05` and `pctChange < 0`, the badge renders green. `Δ` is saved on the report row so the [A/B Comparisons](/seekspeed-docs/reports/ab-comparisons.md) view can chart it against later runs.

***

## Stage 4 — Acceptance breakeven

Even a statistically significant speedup can be a bad configuration choice if it is *below the theoretical ceiling*. The breakeven curve tells you where you are on that ceiling.

We use the standard speculative-decoding speedup approximation (Leviathan et al. 2023), adjusted for draft cost:

```
ceiling(α, k)   = 1 + k · α
speedup(α, k, c) = (1 + k · α) − c · (1 + k)

  α = acceptance rate            (from /metrics)
  k = num_speculative_tokens     (from the DSpark connector config)
  c = draft cost / target cost   (default 0.15 for DSpark's FP8 draft)
```

Interpretation:

* **`speedup(α) = 1.0`** — breakeven. You are paying draft compute for no wall-clock gain.
* **`speedup(α) < 1.0`** — spec decoding is *slower* than vanilla vLLM. Turn it off.
* **`speedup(α) > 1.0`** — spec decoding pays for itself. Compare against the *observed* speedup from stage 2:
  * If `observed / speedup(α) < 0.6` you are leaving >40% of the theoretical gain on the floor (usually prefill share or batch pressure — see the corresponding cards).
  * If `observed / speedup(α) ≈ 1` you are running at the ceiling. Any further improvement requires a better draft (higher α) or a smaller draft (lower c), not a config tweak.

The chart is rendered by `src/components/spec-lab/BottleneckPanel.tsx` using Recharts, with the observed `(α, observedSpeedup)` plotted as a single dot against the theoretical curve so the gap is visible at a glance.

***

## Reproducing our reference DSpark numbers

If you want to reproduce the numbers we quote in [DSpark Integration](/seekspeed-docs/spec-lab/dspark.md):

```bash
# Spec endpoint
python -m vllm.entrypoints.openai.api_server \
  --model deepseek-ai/DeepSeek-V4-Pro-DSpark \
  --speculative-model deepseek-ai/DeepSpec-Draft-V3 \
  --num-speculative-tokens 5 \
  --use-v2-block-manager \
  --port 8000

# Baseline endpoint (same target, no draft)
python -m vllm.entrypoints.openai.api_server \
  --model deepseek-ai/DeepSeek-V4-Pro-DSpark \
  --port 8001
```

Then in Spec Lab: adapter `vllm`, endpoint `http://…:8000/v1`, baseline `http://…:8001/v1`, `n = 32`, `max_tokens = 256`, concurrency `1` and `4`. Save the report — the JSON export contains every raw latency, every counter reading, and the full Welch's output, which is what downstream regression alerts diff against.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://seekspeed.gitbook.io/seekspeed-docs/spec-lab/how-we-benchmark-deepspec.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.