Latency

In a live conversation the number that matters is time to first audio: the gap between someone finishing their sentence and hearing the first sound back. Everything else — tokens per second, total generation time — is invisible to the person on the call. On EESI Cloud that gap is around two seconds at the median, and this page is about which parts of it you control.

Where the time goes

A turn is four stages, and they are not equal.

Stage	What happens	Share of the budget
End-of-turn detection	Server-side VAD waits out the pause and decides you have stopped	Small, and set by the server
Understanding	The model hears the turn and decides what to say	Small
Synthesis	The reply is turned into audio	Most of it
Network	Frames reach your client	Your distance to the region

Synthesis dominates, and it dominates for a structural reason worth understanding before you try to tune anything else.

Synthesis is not incremental

The first audio frame arrives when the whole utterance is finished. Speech synthesis here is not streamed token-by-token — a reply is generated as one piece, so a long reply is slow to start, not just slow to finish.

This inverts the intuition people bring from text streaming, where a long answer costs nothing extra up front because the first token arrives immediately either way. Here, reply length is the single biggest lever you have on perceived latency, and it is a lever you pull from the prompt:

{
  "type": "session.update",
  "session": {
    "instructions": "Answer in one or two sentences. Never list more than three items aloud. If a full answer would take more than about fifteen seconds to say, give the short answer and offer to go deeper."
  }
}

That instruction is worth more than any infrastructure change you can make. Two sentences instead of two paragraphs is the difference between a conversation and a lecture, and it happens to also be the difference between a fast turn and a slow one. Three things follow:

Ask for brevity explicitly, and give a limit the model can apply. “Be concise” is advice; “one or two sentences” is a rule.
Don’t let the model read structure aloud. Bulleted lists, headings and step-by-step numbering are long by construction. If your agent needs to convey a list, have it say the first item and ask whether to continue.
Push long content out of the voice channel. “I’ve sent that to your email” finishes in a second; reading a policy paragraph does not.

Pace your audio in real time

Send ~20 ms of 16 kHz mono PCM16 per message (640 bytes), at the rate it was recorded. Server-side VAD uses arrival timing to find the end of a turn, so pushing a whole file into the socket at once does not make the answer come faster — it makes endpointing wrong, because everything looks like one continuous burst of speech. A microphone feeding the socket in real time is the case the detector is tuned for.

Both directions are 16 kHz, and nothing on the wire checks it. Audio at another rate is resampled as though it were already 16 kHz — 24 kHz input plays back 1.5× fast and transcription accuracy collapses. It presents as a bad model rather than as a configuration error. See Realtime.

Cut playback on `speech_started`, don’t drain

When someone interrupts, generation cancels server-side and queued audio is dropped. Your client has to do the same thing locally:

ws.addEventListener("message", (event) => {
  const message = JSON.parse(event.data);
  if (message.type === "input_audio_buffer.speech_started") {
    // Stop now. Anything already buffered is a reply to a question
    // the user has moved on from.
    player.stop();
    player.clear();
  }
});

Draining the buffer you already have is the most common cause of an agent that “feels laggy” despite fast server numbers: the server cancelled in 50 ms and your audio element kept talking for another two seconds. Barge-in is a client responsibility as much as a server one.

Keep sockets short

A realtime session is billed for the time it is connected and it holds a slot against your plan’s concurrency ceiling, so an idle open socket costs money and capacity without doing anything. Open one when a conversation starts and close it when the conversation ends — not when the page unloads. Reconnecting is cheap. Holding thirty sockets open for thirty users who are mostly reading is not.

What doesn’t help

Tempting	Why it doesn’t move the number
Sending audio faster than real time	Breaks end-of-turn detection; the reply starts later, not sooner
Smaller audio chunks	20 ms is already well under every other term in the budget
Pre-warming with an empty `response.create`	Produces a real turn the user hears

Batch synthesis is a different problem

None of this applies to POST /v1/audio/speech. There you are not racing a human, so the thing to optimize is throughput: run requests concurrently, split long text into paragraph-sized pieces so a single failure doesn’t cost you the whole job, and write straight to disk as the response arrives. See Streaming to a file.

Get started

Speech

Realtime

Telephony

Care

Platform

Where the time goes

Synthesis is not incremental

Pace your audio in real time

Cut playback on `speech_started`, don’t drain

Keep sockets short

What doesn’t help

Batch synthesis is a different problem

Going further

Realtime

Pricing and rate limits

​Where the time goes

​Synthesis is not incremental

​Pace your audio in real time

​Cut playback on speech_started, don’t drain

​Keep sockets short

​What doesn’t help

​Batch synthesis is a different problem

​Going further

Realtime

Pricing and rate limits

Where the time goes

Synthesis is not incremental

Pace your audio in real time

Cut playback on `speech_started`, don’t drain

Keep sockets short

What doesn’t help

Batch synthesis is a different problem

Going further