# Live conversation

A spoken conversation in real time, over WebRTC in the browser or a WebSocket from your server.

Nur holds a conversation. You stream audio in, Nur streams audio back,
and the server decides when you have finished speaking, lets you interrupt, and
keeps the conversation's history. The model is `nur-v1`.

There are two ways to connect, and both speak the OpenAI Realtime protocol:

| | Use it from | Audio travels as |
| --- | --- | --- |
| **WebRTC**, `POST /v1/realtime/calls` | Browsers and mobile apps | Opus over a media track, with built-in echo cancellation |
| **WebSocket**, `wss://api.eesi.ai/v1/realtime` | Servers, telephony, anything headless | Base64 PCM in JSON events |

Events, session settings and replies are the same on both.

## WebRTC in the browser

The browser captures the microphone and plays Nur through a normal audio
element. Your server only relays two requests, so the key stays private.

```js server.js
// Node 18+ with Express. Two routes the browser calls.
import express from "express";

const app = express();
const KEY = process.env.EESI_API_KEY;
const API = "https://api.eesi.ai/v1";

app.get("/nur/ice", async (_req, res) => {
  const r = await fetch(`${API}/realtime/ice-servers`, {
    headers: { Authorization: `Bearer ${KEY}` },
  });
  res.json(await r.json());
});

app.post("/nur/call", express.text({ type: "application/sdp" }), async (req, res) => {
  const r = await fetch(`${API}/realtime/calls?model=nur-v1`, {
    method: "POST",
    headers: { Authorization: `Bearer ${KEY}`, "Content-Type": "application/sdp" },
    body: req.body,
  });
  res.status(r.status).type("application/sdp").send(await r.text());
});

app.listen(3000);
```

```js browser.js
const { ice_servers } = await fetch("/nur/ice").then((r) => r.json());
const pc = new RTCPeerConnection({ iceServers: ice_servers });

// Nur's voice
const speaker = new Audio();
speaker.autoplay = true;
pc.ontrack = (event) => { speaker.srcObject = event.streams[0]; };

// Your voice
const mic = await navigator.mediaDevices.getUserMedia({
  audio: { echoCancellation: true, noiseSuppression: true, autoGainControl: true },
});
pc.addTrack(mic.getAudioTracks()[0], mic);

// Events travel on a data channel with this exact name
const events = pc.createDataChannel("oai-events");
events.onopen = () => {
  events.send(JSON.stringify({
    type: "session.update",
    session: {
      type: "realtime",
      instructions: "You are a friendly concierge. Keep answers short.",
      audio: { output: { voice: "ev_d116d96b" } },
    },
  }));
  events.send(JSON.stringify({ type: "response.create" })); // Nur speaks first
};
events.onmessage = (event) => console.log(JSON.parse(event.data).type);

// Send the offer once candidates are gathered
await pc.setLocalDescription(await pc.createOffer());
await new Promise((resolve) => {
  if (pc.iceGatheringState === "complete") return resolve();
  pc.addEventListener("icegatheringstatechange", () => {
    if (pc.iceGatheringState === "complete") resolve();
  });
  setTimeout(resolve, 2000);
});
const answer = await fetch("/nur/call", {
  method: "POST",
  headers: { "Content-Type": "application/sdp" },
  body: pc.localDescription.sdp,
}).then((r) => r.text());
await pc.setRemoteDescription({ type: "answer", sdp: answer });
```

To hang up, close the peer connection. The OpenAI SDKs can create the call
too: `client.realtime.calls.create(sdp=..., session=...)` posts the same
request.

## WebSocket from a server

```python
import asyncio, json, os

import websockets

URL = "wss://api.eesi.ai/v1/realtime?model=nur-v1"


async def main():
    async with websockets.connect(
        URL, additional_headers={"Authorization": f"Bearer {os.environ['EESI_API_KEY']}"}
    ) as ws:
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {"type": "realtime", "instructions": "Be brief and kind."},
        }))
        async for raw in ws:
            event = json.loads(raw)
            print(event["type"])


asyncio.run(main())
```

A browser can also use the WebSocket by connecting with a single-use
[ticket](/develop/authentication#websocket-a-ticket) instead of a key.

## Configure the session

Send `session.update` at any time. Later updates merge with earlier ones, so a
frame that changes only the voice keeps your instructions.

```json
{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "instructions": "You are the front desk of a small hotel. Answer in one or two sentences.",
    "audio": {
      "output": { "voice": "ev_d116d96b", "speed": 1.0 },
      "input": { "turn_detection": { "type": "server_vad", "silence_duration_ms": 600 } }
    },
    "tools": []
  }
}
```

| Field | What it does |
| --- | --- |
| `instructions` | Who the assistant is and how it behaves. Up to 64 KB. |
| `audio.output.voice` | A voice id from [Voices](/speech/voices). Cloned voices work. |
| `audio.output.speed` | Speaking rate from 0.5 to 2.0. |
| `audio.input.turn_detection` | `server_vad` with `silence_duration_ms` (how long a pause ends a turn) and `threshold`. Set `interrupt_response: false` if people should not be able to cut Nur off. |
| `output_modalities` | `["text"]` for text replies only, which skips speech and answers faster. |
| `tools` | Function tools in the OpenAI format. Tool calls arrive as `response.function_call_arguments.done`. |

Put today's date and anything else the assistant should know in
`instructions`.

## Audio on a WebSocket

WebRTC handles audio formats for you. On a WebSocket, audio is 16-bit mono PCM,
base64 encoded:

- **Input** defaults to 16 kHz. Send about 20 ms per message (640 bytes) at the
  pace it was recorded. Turn detection listens to the timing, so sending a file
  all at once makes for poor turn-taking, not a faster reply.
- **Output** defaults to 16 kHz. For 24 kHz, set
  `"output": {"format": {"type": "audio/pcm", "rate": 24000}}`.
- For telephony, `{"type": "audio/pcmu"}` and `{"type": "audio/pcma"}` select
  G.711 at 8 kHz.

To keep 16 kHz, leave `format` out. Don't write `"rate": 16000`, which the
schema rejects.

```json
{ "type": "input_audio_buffer.append", "audio": "<base64 PCM16>" }
```

You can type into the same conversation. Text does not end a turn by itself, so
follow it with `response.create`:

```json
{ "type": "conversation.item.create", "item": { "type": "message", "role": "user", "content": [{ "type": "input_text", "text": "What time is checkout?" }] } }
```

## Interruptions

People talk over each other, and Nur is built for it. When someone starts
speaking over a reply, the server stops generating and tells you with
`input_audio_buffer.speech_started`. On WebRTC the gateway also stops the
audio it already sent. On a WebSocket, stop your own playback when that event
arrives, or Nur will seem to keep talking.

```python
if event["type"] == "input_audio_buffer.speech_started":
    player.clear()  # drop queued audio right away
```

## Events you will use

| Event | When |
| --- | --- |
| `session.created` | The session is ready. |
| `input_audio_buffer.speech_started` / `speech_stopped` | The person started or stopped speaking. |
| `conversation.item.input_audio_transcription.completed` | What the person said, as text. |
| `response.created` | Nur is starting a reply. |
| `response.output_audio.delta` | A chunk of Nur's voice (WebSocket only). |
| `response.output_audio_transcript.delta` | The words Nur is saying, as they come. |
| `response.done` | The reply is complete. Audio may still be playing. |
| `error` | Something was refused. The session stays open unless it closes. |

The server also sends a few `eesi.*` events. `eesi.session` carries the
`session_id` and `resume_token` for reconnecting, `eesi.recording_disclosure`
has wording to show people when sessions are recorded, and
`eesi.synthetic_audio` declares the audio machine-generated. You can ignore
them if you don't need them.

## Reconnecting

If a connection drops, reconnect within 15 minutes with
`?resume=<session_id>&resume_token=<resume_token>` from the `eesi.session`
event, and the conversation continues where it left off. The new
`eesi.session` event then includes `resumed`; if it doesn't, you got a fresh
session.

## Languages

Nur understands and speaks more than a hundred languages and follows the person
when they switch. To keep it in one language, say so in `instructions`. For
interpreting rather than conversing, use [Live translation](/realtime/translate).

## Limits

- A session can last up to three hours. A WebSocket with no client traffic for
  five minutes closes; send `{"type": "eesi.keepalive"}` while a person is
  reading or thinking.
- Usage is billed per minute of session time.
- If your organization runs out of credits during a call, the session ends with
  a message saying why before it closes.