Robots

• A live conversation through the robot’s own microphone and speaker. Nur hears people while it speaks and stops the moment someone cuts in.
• Events your body code follows: someone started talking, the robot started a reply, the robot was cut off.
• Actions as tools. Each one answers “started” at once and reports back when the motors finish, so the conversation never waits on a wheel or an arm.
• Voices told apart. With more than one person in the room, transcripts label each voice Speaker 1, Speaker 2, so the robot can turn to whoever spoke.

​

Run it

pip install "eesi-sdk[audio] @ git+https://github.com/eesi-ai/eesi-sdk-python"
export EESI_API_KEY=sk-eesi-...
python robot.py

import asyncio

from eesi import NurClient, Started


class Body:
    """Swap these for your robot's own motion calls."""

    def look_at_speaker(self):
        print("(turns to whoever is talking)")

    def stop_gestures(self):
        print("(stops moving)")

    async def wave(self):
        await asyncio.sleep(2)  # the arm moving
        return "Waved."


body = Body()
nur = NurClient()
robot = nur.character(name="Nova", persona="A friendly robot at the front desk of our lab.")


@robot.tool
async def wave() -> Started:
    """Wave at the person you are talking with."""
    return Started("Waving now.", body.wave())


async def main():
    async with robot.connect("visitor", audio=nur.audio.local()) as session:
        session.on("audio.input.started", lambda event: body.look_at_speaker())
        session.on("speech.interrupted", lambda event: body.stop_gestures())
        await session.run()


asyncio.run(main())

​

On a real robot

• Audio. nur.audio.local() takes the computer’s default microphone and speaker. Pick the robot’s own with input_device and output_device. If its audio already cancels echo, pass echo="off". Otherwise the SDK keeps the microphone from hearing the robot, and people cut in by speaking up.
• What it sees and does. Send perception as live state, not as conversation: await session.context.update({"seen": {"blue mug": "on the counter"}, "holding": "nothing"}). Nur reads the latest state before each reply, and an update never starts a reply by itself.
• Who is talking. transcript.final events carry turns, each with its speaker number. Bind a number to the direction your microphone array heard it from, and turn the head there.
• Your own audio pipeline. Leave audio out and stream 16 kHz mono PCM with await session.send_audio(pcm); play the audio.output.chunk events.

​

Next