an.audio

Audio pipeline — TTS and lip-sync providers + orchestration.

Phase 3 ships:

• Protocols: TTSProvider, LipSyncProvider (Phase 1).

• Default offline provider: OfflineTTS (silent WAV). The lip-sync providers (OfflineLipSync deterministic char-to-viseme, RhubarbLipSync, WhisperLipSync) moved to cutan with the cut-out genre (an#225); the old names still resolve from here, with a warning.

• Real providers: ElevenLabsTTS (needs ELEVEN_API_KEY).

• Orchestration: produce_audio_for_dialogue / produce_audio_for_scene walk a SceneIR, synthesize, persist to mall, stamp viseme tracks back onto the IR.

The defaults are intentionally offline so the entire an pipeline works without external services.

Functions

word_timings_to_visemes(words, *, ...[, ...])	Distribute viseme keyframes across word boundaries.
default_tts()	The default TTS provider: OfflineTTS.
default_lipsync()	The default lip-sync provider: the loaded genre's offline one (the deterministic char-to-viseme provider of cutan), else NullLipSync.
produce_audio_for_dialogue(dialogue[, mall, ...])	Synthesize audio + visemes for one dialogue line.
produce_audio_for_scene(scene[, mall, tts, ...])	Walk every dialogue line, synthesize, and stamp viseme tracks back.
make_tts(name)	Instantiate a TTS provider by name.
make_lipsync(name, *[, language])	Instantiate a LipSync provider by name.
known_tts_names()	Return the registered TTS provider names.
known_lipsync_names()	Return the registered LipSync provider names.

Classes

TTSProvider(*args, **kwargs)	Text-to-speech provider.
AudioClip([path, bytes_, duration, ...])	A rendered audio clip, on disk or in memory.
VoiceMeta(voice_id, name, provider[, ...])	Metadata for a TTS voice as exposed by a provider.
LipSyncProvider(*args, **kwargs)	Audio + transcript → aligned viseme track.
Viseme(time, code[, intensity])	A single mouth-shape keyframe.
VisemeTrack([visemes, convention, duration, ...])	Aligned viseme sequence produced by a LipSyncProvider.
WordTimingProvider(*args, **kwargs)	Audio → [(word, start_s, end_s), ...].
NullLipSync()	The lip-sync provider of a core without a genre: no mouth shapes at all.
OfflineTTS(*[, sample_rate, channels, ...])	Default TTS provider: silent WAV of length proportional to text.
ElevenLabsTTS(*[, api_key, model_id, ...])	ElevenLabs-backed TTSProvider.
MacSayTTS(*[, default_voice_id, ...])	macOS say-backed TTSProvider.

class an.audio.AudioClip(path=None, bytes_=None, duration=0.0, sample_rate=44100, channels=1, voice_id=None, transcript=None)

Bases: object

A rendered audio clip, on disk or in memory.

class an.audio.ElevenLabsTTS(*, api_key=None, model_id='eleven_turbo_v2_5', output_format='mp3_44100_128', audio_tag_model_prefixes=('eleven_v3', 'eleven_v4'), client_factory=None)

Bases: object

ElevenLabs-backed TTSProvider. Constructor takes an optional api_key (falls back to ELEVEN_API_KEY / ELEVENLABS_API_KEY).

Implements the TTSProvider protocol, plus the optional synthesis_options hook the audio pipeline reads (an#209).

billed: bool = True

what may never run in memory).

Type:

Every request is charged (an#311

billed_characters(text, *, audio_tags=None, **_options)

Characters one request bills: the text as sent, audio tags included.

Return type:

int

>>> ElevenLabsTTS(api_key="unused").billed_characters("Hi!", audio_tags=["excited"])
13

check_available()

Raise ElevenLabsUnavailableError when a request could not be sent (no API key) — what the audio pipeline asks before the first request of a render, so nothing is half-synthesized (an#305).

Return type:

None

client_factory

api_key -> client; tests inject a fake so nothing reaches the API.

list_voices(*, search=None)

The account’s voices (its own plus the ones it saved), newest API.

search filters by name, description and labels on the server. An absent key or SDK yields []; a key that is present and a call that fails RAISES — an empty listing must mean “no voices”, not “it broke”.

Return type:

Iterable[VoiceMeta]

synthesis_options(voice, *, emotion=None, direction=None)

The synthesize keyword arguments a voice document and a line imply.

Only what is declared appears ({} for a plain voice), so the audio cache key — which includes this dict when it is non-empty — is unchanged for every voice that declares nothing new. audio_tags are the line’s [emotion] (unless neutral) then its direction cues, and only on a model that reads tags; elsewhere a direction is dropped with a warning and the emotion stays a face-only cue, as before.

Return type:

dict[str, Any]

synthesize(text, voice_id=None, *, model_id=None, voice_settings=None, seed=None, audio_tags=None, **kw)

Speak text. The clip’s transcript is text WITHOUT the tags, so alignment and captions never read a cue.

Return type:

AudioClip

take_options(options, take)

The request for candidate take of a best-of-N line (an.audio.takes).

A declared seed is offset by the take, so a model that does honour it (every model but eleven_v3, measured) still returns different takes; without a seed the request is unchanged and the model’s own sampling varies the take.

Return type:

dict[str, Any]

>>> ElevenLabsTTS(api_key="unused").take_options({"seed": 11}, 2)
{'seed': 13}

class an.audio.LipSyncProvider(*args, **kwargs)

Bases: Protocol

Audio + transcript → aligned viseme track.

A provider that fills VisemeTrack.words also declares emits_word_timings = True (a plain class attribute, absent means False). The audio pipeline reads it to decide whether a line whose word_timings are missing needs re-alignment — without the flag, a provider that cannot supply words would re-align every line forever.

align(audio, transcript)

Produce a viseme track for audio given its transcript.

Return type:

VisemeTrack

class an.audio.MacSayTTS(*, default_voice_id='Samantha', sample_rate=22050, rate_wpm=None)

Bases: object

macOS say-backed TTSProvider.

Implements the TTSProvider protocol. Audible, deterministic, and fully offline — uses Apple’s voice synthesis bundled with the OS.

billed: bool = False

an#311).

Type:

Costs nothing per request (stated, never inferred

repeatable: bool = True

The same request gives the same audio, so best-of-N takes never apply (an.audio.takes.voice_takes()) and nothing is billed.

class an.audio.NullLipSync

Bases: object

The lip-sync provider of a core without a genre: no mouth shapes at all.

A genre that animates mouths (cutan) registers real providers (offline, rhubarb, whisper); without one, a line’s audio is still synthesized and the track it gets is empty – nothing draws a mouth anyway.

>>> NullLipSync().align(AudioClip(bytes_=b"", duration=0.5), "hi").duration
0.5

repeatable: bool = True

a reader of the cache keys may run it in memory (an#311).

Type:

The same request gives the same track, and nothing is billed

class an.audio.OfflineTTS(*, sample_rate=22050, channels=1, seconds_per_char=0.06)

Bases: object

Default TTS provider: silent WAV of length proportional to text.

Implements the TTSProvider protocol.

billed: bool = False

an#311).

Type:

Costs nothing per request (stated, never inferred

repeatable: bool = True

The same request gives the same audio, so best-of-N takes never apply (an.audio.takes.voice_takes()) and nothing is billed.

silent: bool = True

a line it speaks for a voice written for a real provider is a stand-in the render says so loudly (an#305).

Type:

Its audio is silence

class an.audio.TTSProvider(*args, **kwargs)

Bases: Protocol

Text-to-speech provider.

list_voices()

Return all voices the provider exposes.

Return type:

Iterable[VoiceMeta]

synthesize(text, voice_id, **kw)

Render text in voice_id’s voice. Returns an AudioClip.

Return type:

AudioClip

class an.audio.Viseme(time, code, intensity=1.0)

Bases: object

A single mouth-shape keyframe.

class an.audio.VisemeTrack(visemes=<factory>, convention='rhubarb', duration=0.0, words=None)

Bases: object

Aligned viseme sequence produced by a LipSyncProvider.

words carries the word timings the provider aligned from, when it had any (whisper, WordTimingsLipSync); None for providers that never see words (offline, Rhubarb — whose JSON is mouth cues only). Retained since an#96 rather than discarded after the viseme conversion: captions (Wave 8) and any consumer that wants to know which word a mouth shape belongs to read them from the IR.

class an.audio.VoiceMeta(voice_id, name, provider, language='en', gender=None, extra=<factory>)

Bases: object

Metadata for a TTS voice as exposed by a provider.

class an.audio.WordTimingProvider(*args, **kwargs)

Bases: Protocol

Audio → [(word, start_s, end_s), ...].

Implementations may run a transcriber (Whisper / Scribe) or look up pre-computed timings (muvid’s lacing store). Returned tuples are expected to be in ascending start-time order; gaps between words are fine and represent silence the lipsync provider should rest through.

words_for(audio, *, transcript='')

Return the word timings for audio.

Return type:

Sequence[tuple[str, float, float]]

an.audio.default_lipsync()

The default lip-sync provider: the loaded genre’s offline one (the deterministic char-to-viseme provider of cutan), else NullLipSync.

Return type:

LipSyncProvider

>>> default_lipsync().name in {"offline", "none"}
True

an.audio.default_tts()

The default TTS provider: OfflineTTS.

Return type:

TTSProvider

an.audio.known_lipsync_names()

Return the registered LipSync provider names.

Return type:

list[str]

an.audio.known_tts_names()

Return the registered TTS provider names.

Return type:

list[str]

an.audio.make_lipsync(name, *, language='en')

Instantiate a LipSync provider by name.

language (BCP-47) reaches providers that select behaviour by language — Rhubarb’s recognizer today; a future aligner’s weight allowlist.

The default name ("offline") resolves to NullLipSync when no loaded genre provides it, so voicing a line needs no genre; any other name that nothing provides is an error that says how to get it.

Return type:

LipSyncProvider

>>> make_lipsync("offline").name in {"offline", "none"}
True

an.audio.make_tts(name)

Instantiate a TTS provider by name.

Raises ValueError for unknown names with a list of known options.

Return type:

TTSProvider

an.audio.produce_audio_for_dialogue(dialogue, mall=None, *, tts=None, lipsync=None, effects=None, voice_id=None, takes=<object object>, take_scorer=<function make_take_scorer>)

Synthesize audio + visemes for one dialogue line.

Side effects: when mall is provided, persists the WAV to mall["audio"] keyed by the content-hash of the dialogue, and persists the viseme JSON to mall["visemes"] similarly. Cache-friendly: a second call with identical inputs returns the cached versions.

effects (default: what the line’s voice declares in mall["voices"]) is applied to the synthesized audio BEFORE alignment, so the visemes are computed on the audio the viewer hears. The raw synthesis stays cached under its own key, so changing an effect never re-pays the TTS.

tts (default: the provider the voice document declares, else offline — tts_chooser()) is an override when given.

voice_id (default: the line’s voice_ref, else "default") is the voices-store key; produce_audio_for_scene() passes the one an.audio.voices.line_voice_id() resolves, so a character’s bound voice reaches here (an#194). The provider is handed the voice document’s own voice id when it names one.

What the provider’s optional synthesis_options hook derives from the voice document and the line (ElevenLabs: model_id, voice_settings, seed, and the [emotion]/direction audio tags — an#209) is passed to synthesize and keyed; the text handed to alignment is always the bare dialogue.text, never the tagged one.

takes (default: what the voice declares for this line — see an.audio.takes; None forces one take) synthesizes several takes, scores each with take_scorer(spec) on the audio the viewer hears, keeps the best and records the choice in mall["takes"]. A recorded choice is restored from the record and never re-rolled; a recorded take whose audio is gone raises TakeLostError before any request.

Return type:

tuple[AudioClip, VisemeTrack]

an.audio.produce_audio_for_scene(scene, mall=None, *, tts=None, lipsync=None, take_scorer=<function make_take_scorer>, announce=<function _announce_to_stderr>, overruns=True, strict=False, tts_factory=None)

Walk every dialogue line, synthesize, and stamp viseme tracks back.

Mutates the scene in place AND returns it (for chaining). Stamps Dialogue.duration, Dialogue.viseme_track, and Dialogue.audio_ref (mall[“audio”] key) so the renderer can find the audio later. Lines with an existing viseme_track AND audio_ref are not re-synthesized (idempotent).

Dialogue.start is DERIVED on every pass, synthesized or not, by Dialogue.planned_start(): the line’s at if set, else the previous line’s end plus its pause (an#187). So editing a pause re-times the shot without touching the audio, and every consumer of start — the mux, the visemes, captions, ducking — follows. A start on a line that was never synthesized is an authored start from before at existed, and is kept as the line’s at.

Every line’s request is resolved BEFORE anything is synthesized, so a malformed voice (an effect, a takes), a corrupt takes record or a recorded take whose audio is gone fails before a credit is spent; and announce (default: a line on stderr; None for silence) is told, before the first request, what best-of-N takes will bill (requests and the provider’s characters) and which recorded takes were chosen by an older scorer version than the current one (they are kept). After synthesis, a line that ends past its shot’s end (dialogue_overruns()) is announced too — or, with announce=None, a DialogueOverrunWarning — unless overruns=False: an render passes that, because it reports every post-synthesis finding together in its summary (an#254).

Who speaks each line (an#305, tts_chooser()): with tts not given (None, "" or VOICE_TTS), the line’s voice document’s provider, built by tts_factory (default an.audio.providers.make_tts()), else offline; a tts given speaks every line. A line spoken by another provider than its voice declares is a VoiceStandInWarning — the silent offline voice says so — and, with strict, a VoiceStandInError before any request. A provider with requests to send is asked check_available() first (ElevenLabs: is there a key?), and what a provider that bills per character will bill is announced before the first request — whatever chose the provider; a cached line is never billed and never counted.

Return type:

SceneIR

an.audio.word_timings_to_visemes(words, *, total_duration, char_to_viseme, rest_viseme='X', min_gap_for_rest=0.2)

Distribute viseme keyframes across word boundaries.

Used by WhisperLipSync and WordTimingsLipSync. The algorithm: per word, walk its characters in order, look up each character’s viseme code, dedupe consecutive identical codes, and space the resulting keyframes evenly across the word’s [start, end] interval. In gaps wider than min_gap_for_rest — between two words, or after the last one before total_duration — insert a single rest keyframe just after the previous word ended.

Times are clamped to [0, total_duration] since some transcribers occasionally round the last word’s end past the audio’s actual length.

Return type:

list[Viseme]

Modules

cli	an voices ... — browse a TTS provider's voices from the shell (an#209).
effects	Voice effects: a deterministic transform applied to a synthesized line (an#163).
elevenlabs_tts	ElevenLabsTTS — real speech via the ElevenLabs API.
lipsync	Lip-sync provider protocol + viseme dataclasses.
loudness	Voice loudness: one integrated loudness for every voice of a film (an#315).
mac_say_tts	MacSayTTS — audible offline speech via macOS's built-in say command.
offline_tts	OfflineTTS — produces silent audio of plausible duration.
pipeline	Audio pipeline orchestration: dialogue → audio → visemes → IR mutation.
providers	Provider factory: name → concrete TTS/LipSync provider instance.
takes	Best-of-N takes: re-roll a line, score every take, keep the best, record the choice.
tts	TTS provider protocol + supporting dataclasses.
voices	Which voice speaks a dialogue line: the character → voice binding (an#194).