Audio intelligence

Audio intelligence is speech-to-text with speakers, word timing and acoustics built in. One pass returns the transcript, anonymous speaker labels, segment and word timestamps, per-word confidence, and an audibility reading for every region — there is no separate diarization stage to pay for, no forced-alignment service to bolt on, and no chunk-and-normalize pipeline to maintain. The model handles up to ~90 minutes in a single pass; the serving hardware sets the practical ceiling per deployment — on the current production GPU that is roughly 20 minutes per file (the model’s full context needs a larger card). Split longer recordings at silence boundaries before uploading.

When to use it

Meetings and interviews

More than one voice, and you need to know which is which.

Call review

Both sides of a call, with the timings to jump a player to any line.

Field and shoot audio

The audibility read tells you which takes are usable before anyone tries to edit them.

One voice only?

Plain transcription is cheaper and returns the same words.

Getting a diarized transcript

It rides the same endpoint as plain transcription. Pick a model that reports supports_diarization from GET /v1/audio/models and ask for verbose_json:

curl https://api.eesi.ai/v1/audio/transcriptions \
  -H "Authorization: Bearer sk-eesi-..." \
  -F file=@interview.mp3 \
  -F model=nur-stt-v1-pro \
  -F response_format=verbose_json

{
  "task": "transcribe",
  "language": "en",
  "language_probability": 1.0,
  "duration": 9.19,
  "noise_floor_db": -54.9,
  "speech_seconds": 8.99,
  "silence_seconds": 0.2,
  "text": "The property has been fully renovated... Can you move a little to the left, please?",
  "segments": [
    { "id": 0, "start": 0.07, "end": 3.61, "speaker": "S01", "text": "The property has been fully renovated with new hardwood floors throughout.",
      "pace_wpm": 186, "energy_db": -13.5, "level_db": -13.5, "snr_db": 41.4, "audibility": "on_mic", "confidence": 0.962 },
    { "id": 1, "start": 3.65, "end": 5.55, "speaker": "S02", "text": "Can you move a little to the left, please?",
      "pace_wpm": 284, "energy_db": -19.8, "level_db": -51.6, "snr_db": 3.3, "audibility": "faint", "confidence": 0.943 }
  ],
  "words": [
    { "word": "The", "start": 0.02, "end": 0.08, "confidence": 0.979, "segment_id": 0, "speaker": "S01", "level_db": -16.8, "snr_db": 38.1, "audibility": "on_mic" },
    { "word": "property", "start": 0.16, "end": 0.58, "confidence": 0.984, "segment_id": 0, "speaker": "S01", "level_db": -13.7, "snr_db": 41.2, "audibility": "on_mic" }
  ],
  "gaps": [],
  "disfluencies": { "filler": 0, "repetition": 0, "false_start": 0 }
}

Word-level output

words is a flat, OpenAI-shaped array — each entry carries segment_id and speaker so you can regroup it into turns without a second pass.

Field	Meaning
`start`, `end`	Word boundaries in seconds, from CTC forced alignment against the decoded transcript. Clamped inside the turn the word belongs to.
`confidence`	How strongly the audio supports that word at the position it was aligned to, 0–1. `null` when the word could not be aligned (a numeral, a symbol) — its timing is then interpolated between its neighbours.
`filler`, `disfluency`	`filler` marks the vendor-standard set (`uh`, `um`, `mm-hmm`, …). `disfluency` is `filler`, `repetition` (the same word twice back to back) or `false_start` (an immediate restart). Discourse markers like “like” and “right” are not flagged — by lexicon they produce false positives on ordinary speech.
`level_db`, `snr_db`, `audibility`	See below.

Word timing is on by default. Send timestamp_granularities[]=segment to skip the alignment pass — worth a few seconds of GPU on a long file.

confidence is acoustic support, not probability-of-correctness. Word timing and confidence come from force-aligning the finished transcript back to the audio, so the number says how well the audio matches the word where it was placed — it is not the recognizer’s own belief. The two come apart on hard audio: on a brown-noise-mixed clip the model transcribed every word correctly while this score collapsed. Read it as “the audio is difficult here, give it a listen”, not “this word is probably wrong”.Measured on our clip set (149 words): ECE 0.145, Brier 0.097. Measure it on your own material before gating on it — python -m evals.stt.word_confidence clip.wav prints a reliability table plus ECE, Brier and AUROC.

Audibility: on-mic dialogue vs background talk

Faint off-mic speech — crew direction bleeding into a take, a question from behind the camera — sits 30 dB or more under the dialogue. Whole-file loudness normalization does not recover it, because integrated loudness is dominated by the loud material. This service normalizes adaptively in short frames instead, which does recover it: a −35 dB utterance transcribes word-perfect with the correct speaker. That leaves a problem: after normalization every level looks the same, and the transcript cannot tell you which lines were actually on mic. So the level is measured on the pre-normalization signal, alongside the file’s own noise floor:

Field	Meaning
`noise_floor_db`	The file’s room tone, dBFS. Top level.
`level_db`	Speech level of this word/turn before normalization, dBFS.
`snr_db`	`level_db` minus the noise floor — how far the speech stands above the room.
`audibility`	`on_mic` (≥ 25 dB), `off_mic` (12–25 dB), `faint` (< 12 dB).
`energy_db`	Delivery energy after normalization. Ranks loud/flat delivery inside a take; it is not an audibility signal.

In the example above the presenter reads 41.4 dB SNR (on_mic) and the off-camera request reads 3.3 dB (faint) — both fully transcribed, and distinguishable afterwards.

Silence, pace and language

Field	Meaning
`gaps`	Every silence of 0.5 s or longer, including the lead-in and the tail: `{start, end, duration}`. Computed over merged turns, so overlapping speakers don’t hide a gap.
`speech_seconds`, `silence_seconds`	The same split as two totals.
`pace_wpm`	Words per minute for the turn.
`language`, `language_probability`	Identified from the transcript when the caller doesn’t pass `language`; a supplied `language` is echoed untouched.
`disfluencies`	Whole-file counts of `filler` / `repetition` / `false_start`.

Speaker labels (S01, S02, …) are consistent within one file but anonymous — they identify voices, not people, and reset between files. Two things also happen service-side before decoding: adaptive loudness normalization (above), and a voice-activity gate that answers silent uploads (b-roll, room tone) immediately with "text": "", empty segments, and "no_speech": true — without occupying the GPU.

Sound-event timeline

Alongside the transcript, the whole file is swept with an AudioSet tagger and returned as scene — what sound occurs when, for the entire duration, not one dominant label per asset.

{
  "scene": [
    { "start": 0.0, "end": 42.0, "label": "speech", "raw_label": "Speech", "score": 0.71 },
    { "start": 0.0, "end": 42.0, "label": "music", "raw_label": "Music", "score": 0.28, "role": "bed" },
    { "start": 42.0, "end": 58.0, "label": "music", "raw_label": "Theme music", "score": 0.66, "role": "featured" },
    { "start": 44.0, "end": 58.0, "label": "room_tone", "raw_label": "Silence", "score": 0.21 }
  ],
  "scene_summary": { "speech": 0.72, "music": 0.41, "room_tone": 0.24, "dominant": "speech" }
}

Field	Notes
`label`	`music`, `speech`, `crowd`, `ambience`, `room_tone`, `sfx`.
`raw_label`	The strongest underlying AudioSet class in that region — `Theme music`, `Ocean`, `Door`.
`score`	Mean confidence across the region.
`role`	Music only: `bed` when dialogue runs across it, `featured` otherwise.
`scene_summary`	Share of the file each bucket is present for, plus the dominant bucket. Shares overlap, so they sum above 1.

Events overlap on purpose. Dialogue over a music bed in a room with traffic outside is three simultaneous answers, and a timeline that reports one winner per moment throws two of them away — we measured a speech clip mixed with broadband noise coming back 100% ambience, the speech erased. So each bucket gets its own run of events, and music + speech covering the same seconds is the normal case, not a conflict. Resolution is 2 s, and each slot is classified with a full 10 s of surrounding context. There is no duration cap — the sweep runs over the whole file.

Streaming progress

Long recordings decode for minutes. With stream=true the response is SSE: standard transcript.text.delta / transcript.text.done events, interleaved with a transcript.segment event per finished diarized segment. Each segment’s end timestamp tells you how far into the audio the model has gotten, so a progress bar is end / duration.

import httpx, json

with httpx.Client(base_url="https://api.eesi.ai", timeout=None) as http:
    with http.stream(
        "POST",
        "/v1/audio/transcriptions",
        headers={"Authorization": "Bearer sk-eesi-..."},
        files={"file": ("interview.mp3", open("interview.mp3", "rb"), "audio/mpeg")},
        data={"model": "nur-stt-v1-pro", "stream": "true"},
    ) as response:
        for line in response.iter_lines():
            if not line.startswith("data:"):
                continue
            event = json.loads(line[5:])
            if event["type"] == "transcript.segment":
                seg = event["segment"]
                print(f"[{seg['start']:.1f}s] {seg['speaker']}: {seg['text']}")

Streamed segments carry speaker, pace and audibility as they land. Words need the finished transcript to align against, so they arrive once at the end: the final transcript.text.done event carries the complete verbose_json payload in a verbose field, words included, so a streaming client never needs a second request for the authoritative result.

Limits

Field	Notes
`file`	Audio up to 1 GB. WAV, MP3, FLAC, OGG, M4A, and video containers with an audio track. Practical duration ceiling is set by the serving deployment (~20 minutes on the current production GPU; the model itself handles ~90 on larger cards).
`model`	A model with `supports_diarization` from `GET /v1/audio/models`.
`response_format`	`verbose_json` for segments, words and acoustics; `json`/`text` return the plain transcript.
`timestamp_granularities[]`	`word` (default) or `segment` to skip alignment.
`stream`	`true` streams SSE while the model decodes.
`session_id`	Optional continuity key. Speaker labels are normally per-request (the first voice heard is `S01` in that upload); requests sharing a `session_id` keep one numbering, so `S01` is the same person across every chunk or turn of a conversation. Adds `session_speakers` — the number of distinct voices heard under that key so far — to the verbose payload.

Going further

Speech to text

Plain batch transcription without speaker labels.

Get started

Speech

Realtime

Telephony

Care

Platform

Audio intelligence

When to use it

Meetings and interviews

Call review

Field and shoot audio

One voice only?

Getting a diarized transcript

Word-level output

Audibility: on-mic dialogue vs background talk

Silence, pace and language

Sound-event timeline

Streaming progress

Limits

Going further

Speech to text

​When to use it

Meetings and interviews

Call review

Field and shoot audio

One voice only?

​Getting a diarized transcript

​Word-level output

​Audibility: on-mic dialogue vs background talk

​Silence, pace and language

​Sound-event timeline

​Streaming progress

​Limits

​Going further

Speech to text

When to use it

Getting a diarized transcript

Word-level output

Audibility: on-mic dialogue vs background talk

Silence, pace and language

Sound-event timeline

Streaming progress

Limits

Going further