Audio in, audio out, one model — no transcribe, then think, then speak. The round trip that costs you is gone, and the things that make live conversation hard are handled rather than left to you. /v1/realtime speaks the OpenAI Realtime protocol: session.update, input_audio_buffer.append, response.create, response.output_audio.delta. Existing realtime clients work against it.

​
When to use it

Live conversation

A person talks, the agent answers, either can interrupt. This is what the endpoint is for.

Live transcription

Transcripts of both sides arrive on the same socket as the audio, so you get captions without a second service.

Phone and browser alike

The same session backs a telephony call and a WebRTC one — the agent does not know or care which.

Not for recorded files

Have the audio already? Use transcription — it is cheaper and it does not need a socket held open.

​
Connect

Browsers can’t set headers on a WebSocket. Mint a short-lived token server-side and pass it as a query parameter instead of shipping your API key to the client.

​
Configure the session

The voice must go in session.audio.output.voice. The serving backend reads nowhere else, and a session without one drifts between voices — see Voices.
Do not set audio.input.format. The wire is 16 kHz PCM16 by default, and that default is the only way to get it: the format schema is a discriminated union whose PCM member only admits rate: 24000, so { "type": "audio/pcm", "rate": 16000 } is rejected — and because the union is discriminated on type, the whole session.update is dropped with it, voice and instructions included, while the socket stays open and healthy. An untagged { "rate": 16000 } is worse: it validates as μ-law, and your PCM is decoded as noise. Omit the field and send 16 kHz. If you genuinely have 24 kHz audio, declare { "type": "audio/pcm", "rate": 24000 } and the server resamples. session.created lists the rates it takes (below).

​
Protocol version

/v1/realtime speaks protocol version 1 of EESI’s dialect. The first event on every socket, session.created, declares it beside the OpenAI-shaped session:
  • audio_input_rates — PCM16 rates the server accepts. The first is the default when audio.input.format is omitted.
  • audio_formats — pcm16 (audio/pcm), g711_ulaw (audio/pcmu, 8 kHz) and g711_alaw (audio/pcma, 8 kHz).
  • events — which optional events the speech server emits or honours. Version 1 emits eesi.recording_disclosure (below). resume is false here because resumption is a gateway feature, not a model-server one — see Resume a lost session; ping is reserved.
Both fields are additive. An OpenAI-compatible client that ignores them is still correct; an EESI client reads them instead of guessing what the far end supports. Absent fields mean a gateway that predates versioning — treat that as version 0, never as "".

​
Resume a lost session

Every session’s greeting carries a gateway-private eesi.session event with its resumable identity:
Keep both. If the socket drops — the phone changes networks, a deploy rolls the gateway, a worker dies — reconnect with them:
The gateway checks the token and the organization before it spends a concurrency slot, opens a fresh pipeline, and replays the conversation into it — the last session.update you sent and every completed turn — before your first frame is relayed. The model keeps its memory; at most the turn that was in flight is lost. The greeting of a resumed session repeats the same session_id and token (so it can be resumed again) and adds "resumed": { "replayed_items": N }. A checkpoint lives for 15 minutes after the last turn; an intentional close is not resumable. @eesi/realtime decodes the event as sessionHandle, and the console, iPhone and Mac clients send it back on every redial automatically.

​
Send audio

16 kHz mono PCM16, base64-encoded, in ~20 ms chunks (640 bytes). Pace the sends in real time — server-side VAD uses arrival timing to find turn boundaries, so dumping a whole file at once produces bad endpointing.

​
Receive

Two of these read the opposite way to how they look, and both failures are silent — you get a plausible transcript that is wrong:
  • conversation.item.input_audio_transcription.delta is cumulative. Each delta carries the whole utterance so far, so it replaces the open entry. Append them and you get “hello hello there hello there world”.
  • response.output_audio_transcript.done fires per segment, not once per turn. Keep only the last and every multi-segment reply is truncated — append the deltas instead.
Both are pinned by the conformance corpus in spec/realtime/v1/, which every EESI client replays.
Older OpenAI variants spell three of these without output_: response.audio.delta, response.audio_transcript.delta, and response.audio_transcript.done. This server emits only the canonical spellings above. If your client also talks to a backend that uses the legacy names, handle both — handling one is how a client goes silent against exactly one backend.

​
Barge-in

Interruption is handled server-side. When the user starts speaking over the assistant, generation cancels and queued audio is dropped. Stop playback on input_audio_buffer.speech_started rather than draining your buffer; the cancelled turn then closes with response.done carrying response.status: "cancelled".
Both directions are 16 kHz. Nothing on the wire validates this: the serving backend pins its pipeline rate and resamples whatever arrives as if it were already 16 kHz, so sending 24 kHz audio raises no error — it plays back 1.5× fast and transcription accuracy collapses. It presents as a bad model.

​
Language

Nur is natively multilingual and code-switches mid-sentence without configuration. If you want it pinned to one language, say so in instructions. To interpret rather than converse — output the same words in another language and nothing else — use Live translation. Instructions alone won’t do it: every conversational session carries a persona that tells the model to introduce itself and not to translate unasked.

​
Agent-to-agent

Two realtime sessions can be cross-piped so agents talk to each other — useful for adversarial evaluation and load testing. This needs the serving backend to run at least two sessions concurrently.

​
Connecting from a browser

A browser cannot put an Authorization header on a WebSocket handshake, so the credential would have to ride the URL — and URLs get logged. Mint a ticket over HTTPS instead, where the header is not logged, and hand the socket something that dies on first use.
The ticket goes in token, not ticket — the gateway reads exactly three credential channels (the Authorization header, ?api_key=, ?token=), and a ticket under any other name is a 401 Missing authentication. Tickets are single-use and expire in 30 seconds — mint one per connection, immediately before opening the socket, and mint a fresh one for every reconnect. Clients that can set headers (the OpenAI SDKs, native apps) never need this and should send the key directly. If the handshake is refused, the close code says why: 1008 is a limit or a rejection (auth, quota, an unknown model, the concurrent-session cap) — capacity frees, so “try again in a moment” is honest; 1011 means the realtime backend is unavailable, and retrying changes nothing until it is back. A drop after session.created (1006, or 1011 on a backend restart) is worth redialing with backoff; 1000 from the server is a deliberate end of session.

​
Going further

Latency

Where a turn’s two seconds go, and which knobs actually move them.

Voices

Speech API