diff --git a/fern/assistants/model-intelligence/metrics-methodology.mdx b/fern/assistants/model-intelligence/metrics-methodology.mdx index 2405268c9..e7ae90077 100644 --- a/fern/assistants/model-intelligence/metrics-methodology.mdx +++ b/fern/assistants/model-intelligence/metrics-methodology.mdx @@ -30,7 +30,7 @@ Vapi reports the median response time for each component based on production cal The displayed total is simply the sum of the three component medians, so treat it as a rough estimate rather than an exact figure. -The total excludes [endpointing](/customization/speech-configuration) and transport time. +The total excludes [endpointing](/customization/speech-configuration) and transport time. It isn't a call's turn latency. See [per-turn latency on a call](/assistants/model-intelligence/understanding-latency#per-turn-latency-on-a-call) for how each call measures latency. See [how latency works](/assistants/model-intelligence/understanding-latency) for help understanding latency and why more capable models tend to be slower. diff --git a/fern/assistants/model-intelligence/understanding-latency.mdx b/fern/assistants/model-intelligence/understanding-latency.mdx index bdbd4107b..4e7c2c9da 100644 --- a/fern/assistants/model-intelligence/understanding-latency.mdx +++ b/fern/assistants/model-intelligence/understanding-latency.mdx @@ -1,7 +1,7 @@ --- title: How voice-to-voice latency works in voice agents subtitle: Learn what voice-to-voice latency measures and why capable models respond more slowly -description: Understand voice-to-voice latency, the components included in Vapi performance metrics, and how model capability, endpointing, and transport affect response time. +description: Understand voice-to-voice latency, the per-turn latency fields each call records, Vapi performance metrics, and how model capability, endpointing, and transport affect response time. slug: assistants/model-intelligence/understanding-latency --- @@ -9,15 +9,36 @@ Voice-to-voice latency is the gap between when a caller finishes speaking and wh ## Voice-to-voice latency sequence -Voice-to-voice latency combines five stages that run in sequence. Each stage begins after the previous one. +Voice-to-voice latency combines five stages. After the caller stops speaking, each stage begins when the previous one ends. 1. **Transport:** The caller's audio travels over the network for processing. -2. **Endpointing:** Vapi detects that the caller has finished speaking. -3. **Transcription:** The transcriber converts the caller's speech to text. +2. **Transcription:** The transcriber finishes converting the caller's last words to text. +3. **Endpointing:** Vapi decides that the caller has finished their turn and sends the transcript to the model. 4. **Model response:** The model reads the transcript and begins generating a reply. Vapi measures this stage as time to first token. 5. **Speech synthesis:** The voice provider turns the reply into audio. Vapi measures this stage as time to first audio. -A caller's total voice-to-voice latency is roughly the sum of all five stages. +A caller's total voice-to-voice latency is roughly the sum of all five stages, plus the time the reply takes to reach the caller. + +## Per-turn latency on a call + +Each call records one latency entry per turn in `artifact.performanceMetrics.turnLatencies`. The call's **Latency Summary** tab, the [**Traffic** tab](/assistants/versioning/traffic-tab), and [simulation latency limits](/observability/simulations-advanced#set-latency-limits) are all built from these entries. [GPT-Live](/gpt-live/overview) assistants don't record these entries yet; measure their latency from the call's timestamps, as described in [testing GPT-Live](/gpt-live/testing#latency). + +| Field | Starts when | Ends when | +| --- | --- | --- | +| `transcriberLatency` | The caller's last voice activity, normally when they stop speaking | The transcriber returns the words of that speech | +| `endpointingLatency` | The transcriber returns the words | Vapi decides the turn is over and sends it to the model | +| `modelLatency` | The turn's model stage starts, before knowledge base retrieval and request routing | The model returns its first output token | +| `voiceLatency` | The model returns its first output token | The first reply audio reaches Vapi's turn-taking buffer, after any sample-rate conversion | +| `turnLatency` | The caller's last voice activity | Vapi starts sending the reply audio to the caller | + +Keep these points in mind when you read the numbers: + +- **The fields don't add up to `turnLatency`.** `turnLatency` is `transcriberLatency` plus `endpointingLatency` plus the time from that decision to the reply starting. Model and voice latency happen inside that last stretch, alongside other work such as handing the turn to the model and buffering the reply audio before it plays. +- **Model and voice latency include more than the provider.** `modelLatency` includes knowledge base retrieval and request routing before the provider request. `voiceLatency` includes collecting enough of the reply to send the first voice request, as set by `voice.chunkPlan` (`minCharacters` and punctuation boundaries), and any sample-rate conversion after the provider returns audio. +- **Vapi measures on its servers.** None of the fields include network time to or from the caller, or playback on the caller's device. +- **Only replies to the caller are recorded.** Turns where the assistant didn't speak, for example because the caller interrupted first, have no entry. When the assistant speaks first, its first message has no entry either. When it waits for the caller (`firstMessageMode` set to `assistant-waits-for-user`), the first message is recorded as a reply, with `endpointingLatency` as `0`. +- **`0` can mean "not measured".** A stage that didn't run or wasn't measured on a turn is recorded as `0` rather than left out. For example, when a turn's first audio is a static tool message rather than model output, `modelLatency` and `voiceLatency` are `0`. When `transcriberLatency` is `0` because it wasn't measured, `turnLatency` leaves that interval out too. +- **Averages count every turn.** The call's `modelLatencyAverage`, `voiceLatencyAverage`, and other averages divide by every entry, including those zeros. Simulation latency limits ignore a `0` for `modelLatency` and `voiceLatency`, so a simulation's model or voice result can be higher than the call's average for the same call. ## What the displayed latency measures @@ -27,6 +48,8 @@ Vapi calculates each median from production calls, which reflect a typical call The displayed total excludes [endpointing](/customization/speech-configuration) and transport time. Both contribute to the delay a caller experiences. Endpointing can account for a meaningful share of that delay. Improving endpointing can make an assistant feel more responsive even though it is not included in the displayed total. +The displayed total is a sum of component medians across many calls, so it isn't comparable to a call's `turnLatency`, which is measured end to end on each turn and includes endpointing. See [per-turn latency on a call](#per-turn-latency-on-a-call). + ## Model capability and latency More capable models usually respond more slowly. A model's time to first token is often the largest component of voice-to-voice latency. More capable models tend to be larger and take longer to begin generating a response. diff --git a/fern/assistants/versioning/traffic-tab.mdx b/fern/assistants/versioning/traffic-tab.mdx index 69333bf67..15ffcbd92 100644 --- a/fern/assistants/versioning/traffic-tab.mdx +++ b/fern/assistants/versioning/traffic-tab.mdx @@ -168,7 +168,7 @@ This table lists one row per version, with these columns: - **Duration**: how long calls lasted. - **Cost per minute**: the average cost of a minute of call. Expand it to see the parts: **LLM**, **STT**, **TTS**, **Vapi**, and **Transport**. -- **Turn latency**: how long the assistant took to start responding after the caller finished speaking, averaged over each call. Expand it to see the **Transcriber**, **Model**, and **Voice** stages. +- **Turn latency**: how long the assistant took to start responding after the caller finished speaking, averaged over each call. Expand it to see the **Transcriber**, **Model**, and **Voice** stages. The stages don't add up to the total: endpointing isn't shown, and model and voice latency happen inside the turn rather than after it. See [per-turn latency on a call](/assistants/model-intelligence/understanding-latency#per-turn-latency-on-a-call). Use the **p50** and **p95** toggle to switch duration and latency between the typical call and the slow tail. Sort any column to rank the versions by it. Each bar shows the value against the largest value in its column, so the slowest or most expensive version is easy to spot. diff --git a/fern/glossary.mdx b/fern/glossary.mdx index c22df80ce..4d7ea2fed 100644 --- a/fern/glossary.mdx +++ b/fern/glossary.mdx @@ -104,7 +104,7 @@ An abbreviation used for "Text-to-speech". The process of converting raw text in ### Voice-to-Voice -"Voice-to-voice" is often a term brought up in discussing voice AI system latency — the time it takes to go from a user finishing their speech (however that endpoint is computed) → to the AI agent's first speech chunk/byte being played back on a client’s device. +"Voice-to-voice" is often a term brought up in discussing voice AI system latency — the time it takes to go from a user finishing their speech (however that endpoint is computed) → to the AI agent's first speech chunk/byte being played back on a client’s device. A call's `turnLatency` measures the part of voice-to-voice latency that happens on Vapi's servers; see [per-turn latency on a call](/assistants/model-intelligence/understanding-latency#per-turn-latency-on-a-call). Ideally, this process should happen in \<1s, better if closer to 500-700ms (responding too quickly can be an issue as well). Voice AI applications must closely watch this metric to ensure their applications stay responsive & usable. diff --git a/fern/observability/logs/call-logs.mdx b/fern/observability/logs/call-logs.mdx index 3b95789b5..451e590b4 100644 --- a/fern/observability/logs/call-logs.mdx +++ b/fern/observability/logs/call-logs.mdx @@ -109,7 +109,7 @@ Categories can include speech-to-text (STT), large language model (LLM), text-to The **Latency Summary** tab shows where time was spent during the call. It includes the turn count, average turn latency, and a breakdown by stage. Stages include **Transcriber**, **Endpointing**, **LLM**, **Voice**, and transport in each direction. -Use the per-turn table to find which stage made the call feel slow. It shows each turn's total and per-stage milliseconds. The values are averages and per-turn measurements, not percentiles. +Use the per-turn table to find which stage made the call feel slow. It shows each turn's total and per-stage milliseconds. The values are averages and per-turn measurements, not percentiles. See [per-turn latency on a call](/assistants/model-intelligence/understanding-latency#per-turn-latency-on-a-call) for what each stage measures. For the full object behind these views, see the [call object](/api-reference/calls/get). Recordings and artifacts are also covered in [Call recording](/assistants/call-recording) and [Retrieve call artifacts](/assistants/retrieve-call-artifacts). diff --git a/fern/observability/simulations-advanced.mdx b/fern/observability/simulations-advanced.mdx index 3298e8a26..f55b7e242 100644 --- a/fern/observability/simulations-advanced.mdx +++ b/fern/observability/simulations-advanced.mdx @@ -323,11 +323,17 @@ curl -X PATCH "https://api.vapi.ai/eval/simulation/scenario/" \ ### Metrics and aggregations -| Metric | What it measures | -| --- | --- | -| `turn` | Total time from the end of the AI tester's speech to the start of the assistant's reply | -| `model` | Time for the model to return its first token | -| `voice` | Time for the voice provider to return the first audio | +Each metric reads one field of the call's per-turn latency, measured the same way as on a live call. See [per-turn latency on a call](/assistants/model-intelligence/understanding-latency#per-turn-latency-on-a-call) for exactly where each one starts and ends. + +| Metric | Field | What it measures | +| --- | --- | --- | +| `turn` | `turnLatency` | Time from the end of the AI tester's speech to the assistant starting its reply | +| `model` | `modelLatency` | Time from the start of the turn's model stage, including knowledge base retrieval, to the model's first output token | +| `voice` | `voiceLatency` | Time from the model's first output token to the first reply audio being ready, including the voice provider and any sample-rate conversion | + +Model and voice latency happen inside turn latency, so the `model` and `voice` metrics don't add up to `turn`. A turn that recorded `0` for `model` or `voice` didn't measure that stage, so the metric leaves it out. + +The AI tester speaks with clean synthetic audio, so transcription, endpointing, and turn latency can run faster than with real callers. The aggregation reduces the call's turns to one value: `mean`, `median`, `p95`, or `max`. `p95` uses the nearest-rank method, so on a call with fewer than 20 turns it equals `max`. Measured values are rounded to the nearest millisecond before they are compared with the limit.