> For the complete documentation index, see [llms.txt](https://seekspeed.gitbook.io/seekspeed-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://seekspeed.gitbook.io/seekspeed-docs/benchmarking/metrics.md).

# Metrics: TTFT, Latency, Throughput

| Metric                  | Definition                                                                      | Why it matters                                                                              |
| ----------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| **TTFT**                | ms from request send to first non-empty token chunk in the SSE stream           | "Are you still there?" gap. Single biggest driver of perceived latency for interactive UIs. |
| **Inter-token latency** | mean ms between successive SSE chunks during decode                             | Stutter between words. Voice pipelines need this < 40 ms.                                   |
| **Total latency**       | ms from request send to `[DONE]`                                                | What batch / offline workloads actually care about.                                         |
| **Throughput (tok/s)**  | `tokens_out / total_seconds`                                                    | Capacity metric. Independent of TTFT.                                                       |
| **p50 / p95 / p99**     | percentile cuts across all iterations                                           | Tail latency. p99 under load is what trips SLAs.                                            |
| **Cost**                | `tokens_in × input_price + tokens_out × output_price` from `src/lib/pricing.ts` | USD per benchmark; used for cost/perf optimization rules.                                   |
| **Error rate**          | non-200 + timeout + JSON-parse failures / total                                 | Rejected requests still cost user time; rules track this separately.                        |

## Reading p-values

A run report shows the *t* statistic, degrees of freedom, and *p* value for any baseline-vs-variant comparison. The convention used throughout SeekSpeed:

* `p < 0.01` → **strong** evidence the change moved the mean.
* `p < 0.05` → **significant**, threshold for auto-accept eligibility.
* `p < 0.10` → **suggestive**, retry with more iterations.
* `p ≥ 0.10` → **noise**, do not promote.

## The dishonest metric

`tokens/second averaged over a single request` is a misleading composite. A model with 60 ms TTFT and 40 ms per token over 500 tokens produces the *same* tok/s as a model with 400 ms TTFT and 33 ms per token. They feel completely different. SeekSpeed never reports a single "tok/s" number without TTFT alongside it.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://seekspeed.gitbook.io/seekspeed-docs/benchmarking/metrics.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.