> For the complete documentation index, see [llms.txt](https://seekspeed.gitbook.io/seekspeed-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://seekspeed.gitbook.io/seekspeed-docs/benchmarking/statistics.md).

# Statistical Rigor

Most "we made the agent 30% faster" claims fall apart under a *t*-test. SeekSpeed bakes in three checks so they don't.

## 1. Welch's *t*-test on per-iteration distributions

We use Welch's because baseline and variant rarely have equal variance — a faster variant is often also more consistent.

```ts
// src/lib/stats.ts
export function welchTTest(a: number[], b: number[]) {
  const ma = mean(a), mb = mean(b);
  const va = variance(a), vb = variance(b);
  const se = Math.sqrt(va / a.length + vb / b.length);
  const t  = (ma - mb) / se;
  const df = Math.pow(va/a.length + vb/b.length, 2) /
             (Math.pow(va/a.length, 2)/(a.length-1) + Math.pow(vb/b.length, 2)/(b.length-1));
  return { t, df, pValue: studentT2Tailed(t, df) };
}
```

## 2. Minimum sample size

The UI refuses to display a significance badge under `n < 5` per group. A "significant" result from 3 iterations is almost always a fluke.

## 3. Percentile, not mean, for tail-sensitive workloads

The optimization engine generates separate recommendations for "mean too high" and "p95 too high". A mean-only fix can pass a *t*-test while making the worst case worse — that's a regression for voice and real-time agents.

## Worked example

Baseline run: 30 iterations, mean total 1,420 ms, σ = 180. Variant (slimmer system prompt): 30 iterations, mean total 1,180 ms, σ = 150.

```
t ≈ (1420 − 1180) / sqrt(180²/30 + 150²/30) ≈ 5.62
df ≈ 56
p ≈ 0.0000004  → significant
```

The variant is accepted, the parent agent is archived under `prev-versions`, and the recommendation moves to `accepted` status with `delta_ms = -240` saved on the row.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://seekspeed.gitbook.io/seekspeed-docs/benchmarking/statistics.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.