eesi-sdk) and npm (@eesi/sdk) packages are on the way; the code you
import is the same.
The Python audio extra adds this computer’s microphone and speaker. Leave it
out on a server that streams audio some other way. The TypeScript package runs
on Node 22 or later, Deno, Bun and in browsers; @eesi/sdk/browser has the
page’s microphone and speakers, @eesi/sdk/node file memory and WAV files.
The examples below are Python; the TypeScript SDK has the same parts under
camelCase names, and its README walks through each one.
A character in a few lines
start() opens a session, connects your microphone and speaker, prints the
conversation, and runs until you press Ctrl+C. Kael greets you first. Talk
over him and he stops.
You can also try a character without writing code:
chat lets you type instead of speak.
Who the character is
name. The SDK composes them into the
character’s instructions, adds the rules every spoken character needs (short
turns, no lists or stage directions), and sends them when a session starts.
kael.instructions_text() shows exactly what was composed.
A character keeps its own name and stays in role. When someone sincerely asks
whether they are talking to a person, it says it is an AI.
Presets
Presets are starting points with sensible defaults. Every option still works on top of them.Voices
voice takes a built-in voice name ("Atlas", "Nova", "Nur"), any voice
id from your library ("ev_…"), or a voice described in words:
python -m eesi voices lists
what you have. To hear takes before keeping one, use
voice design directly.
Save and share a character
Sessions
A character is defined once. Each conversation is a session, usually one per player:
Without
audio, send microphone audio with await session.send_audio(pcm)
(16-bit mono PCM, 16 kHz by default, at the pace it was recorded) and play the
character’s voice from audio.output.chunk events. That is how a game server
relays audio from a player’s machine.
You can also type. await session.send_text("Where is the mill?") adds the line
and asks for a reply; reply = await session.ask("Where is the mill?") also
waits for the character’s answer to that line and returns it.
Events
Callbacks can be plain functions or
async functions. on("tool.*") matches
every tool event and on("*") every event. A character’s own on(...) hears
every one of its sessions.
Who speaks when
turn_taking on the character decides who starts replies:
"natural"(the default). Nur decides, with its full conversational timing: it backchannels, waits through pauses, and yields when cut off. It also takes part on its own: when a conversation goes quiet after its reply it may follow up, and it may chime in briefly when someone pauses mid-thought. A session that sets"eesi_proactive": falsedoes neither."app". The SDK asks for a reply about half a second after each turn ends, once fresh state and memories are in place. Tool use is most reliable here."director". The character only speaks when your code says so, withsession.respond()orsession.cue(). Use it for scripted scenes and several characters in one room.
session.say(line)
asks the character to say a line in its voice, and session.interrupt()
stops it mid-sentence.
Tools
ToolContext receives the calling session and player
instead of a model argument:
For work that takes a while, return
Started:
allow or in
your own game code, and keep your game server the authority on what happened.
Live context
Context is what is true right now, beside the conversation. Send it whenever it changes:- Updates merge field by field.
Noneremoves a field. ttl=30makes fields expire.- Pydantic models and dataclasses are accepted, so typed game state works as is.
- Fields are grouped into named blocks. The character’s own context is the
worldblock and a session’s is thesceneblock; passblock="combat"to start another. - Update as often as you like. Each block is sent at most a few times a second, and only when its text changed.
session.inspect() shows the instructions, every block as the character reads
it, the tools and the memories in use.
await session.set_mood("annoyed", reason="the player insulted the king")
colours how the character speaks for a while without changing who it is.
Memory
memory=True
the file is .eesi/memory/memory.db.
What happens in each session:
- When it starts, the memories that matter most for this player are
chosen by relevance to the current situation, importance and recency, and
sent as the
memoryblock. A new player is introduced as one: the character knows it has not met them. - While it runs, what the player says is searched against their memories
and anything relevant is added to a small
recallblock. This happens off the audio path and usually reaches the next reply. - When it ends, what was worth keeping is learned: facts about the player, their preferences, events, promises, how the relationship changed, and a summary of the conversation. Corrections replace the memory they correct, and the old one is kept as history.
nur-v1) in text, with your key. Pass
MemoryPolicy(extractor=...) to use your own model, or
MemoryPolicy(extract=False) to keep only what your code writes. Learning
updates the memory file. It does not train or fine-tune any model.
Write memories yourself:
python -m eesi memory list --character kael.
MemoryPolicy sets what is kept: retention_days (episodes expire),
max_records_per_player, redact (a function applied to every text before it
is stored) and the memory block’s size. memory="read" on connect recalls
without learning, for a guest or a player who has not agreed to be remembered.
Preferences
persist=False adapts only this session. pace also sets the voice’s speed.
personalize=False on the character stops it from learning preferences by
itself.
Several players
One character, many players. Every player gets their own session from the same character. Their memories never mix. Withmultiplayer_npc (or
party=True), each session also sees a short log of what the character said to
the other players, so its story stays consistent.
Several people on one microphone. Nur labels each voice Speaker 1,
Speaker 2 once it has heard more than one. Name the players and the SDK binds
the labels to them from introductions (“I’m Layla”) and the order they were
heard:
transcript.final events then carry each part’s player_id, and tools know
who asked. It is a best effort and inherits every mistake the voice labels
make.
A room. When your server already receives every player’s microphone (your
game’s voice chat runs through it), a room mixes them into one conversation
with the character and knows who spoke from who was loudest while each turn
was spoken, not from introductions:
language and the character answers each in their own.
Languages
A character follows the player by default and switches when they do. Pin one language (a code or a name) and the greeting and every reply are in it;ar-EG is Egyptian Arabic, and language_name names any variety:
Speaking first
A character can speak without being spoken to. Remarks wait for the floor, so they never talk over anyone:Browsers and game engines
Never put your API key in a game client. Your server keeps it and hands each client a client secret instead:secret.url as a WebSocket and is talking to Kael at once:
the session already has his instructions, voice, tools, the player’s memories
and the context you passed. It streams audio and answers tool calls by
forwarding them to your server, which runs them. A secret opens one session and
expires after a minute unless you set expires_in (up to ten minutes).
lock decides what the client may not change. By default it is the
instructions and the tools.
session.update is left out, and the rest of the update
applies. A locked event (live state, a system message, a deleted message) is
refused with an error event whose code is session_locked.
This works from any engine that has a WebSocket: browsers, Unity, Unreal,
Godot. The session speaks the protocol described in Nur.
In a browser, the TypeScript SDK opens it for you. Send the client
secret.join (in either SDK): the URL, and who greets, runs tools and asks for
replies. It never holds your key.
Steer the session from your server
Mint the secret withcontrol=True and your server can act in the session the
client opened, without relaying its audio: live state, tool calls, cues,
remarks and memory, as in a session of its own.
wss://api.eesi.ai/v1/realtime/control/{id}, authenticated with
your key. Tool calls are the server’s to answer; control="client" leaves
them to the client. What the server sends is not held to the locks, and only
the organization that minted the secret can open its channel.
When something goes wrong
- A dropped connection is resumed with the conversation intact
(
reconnecting, thenreconnected). If it cannot be resumed, the session ends withreason="connection_lost". - A refused key raises
AuthenticationErrorfromconnect. - An unknown voice raises
ConfigurationErrorlisting the voices you have. - Context over a limit raises
ContextErrorsaying which block and why. - Errors inside your event handlers are logged and never end the session.
Trace(path, content=True) while
you debug.
Limits
- Python has no echo canceller. On open speakers,
nur.audio.local()keeps the microphone from hearing the character, and you cut in by speaking up. Use headphones andnur.audio.local(echo="off")for the most natural barge-in. Browsers and phones cancel echo themselves. - Latency measured in the SDK is from the end of a turn to the first audio of the reply reaching your machine. It does not include your speaker’s own buffering.
- Everything else (session length, idle timeout, billing) is as described in Nur.