<!-- generated by epythet -->

# foley

Find (or generate) the **right sound effect for a moment of narration** — and weave it in.

foley is a **retrieval-first façade for sound effects**: one simple surface over many
sound *sources* (your own library, service APIs, and generative-AI models), a searchable
*index* of every sound (by keyword **and** meaning), an *agent* that picks the right
sound for a narrative context, and a *compositor* that places it under the voice.

It’s the SFX sibling of [`arioso`](https://github.com/thorwhalen/arioso) (a unified
façade over AI music-generation backends): same discipline — one entry function,
config-driven plugin adapters, a unified vocabulary translated per-backend, zero
required core deps with lazy optional-deps — but centered on **search** rather than
generation, with generation as just one of several sources.

> **Status: v1.** All four stages — source, index, select, weave — plus the MCP server
> and the licensing/provenance, evaluation, and observability layers are implemented
> (Epic #13 complete). The API below is live; see the [roadmap]()
> for what’s next. Follow along in [`misc/docs/`]().

## The idea

```python
import foley

# The headline — right sounds for a narrative context:
# decompose the passage into salient sound events → search → verify → decide
candidates = foley.find("She pushed open the heavy oak door; rain hammered outside.")

# Direct hybrid search of your library (text query or a reference clip)
hits = foley.search("distant thunder rumble", k=10)  # commercial use is the default

# Generate a sound when nothing fits (arioso-style; pluggable backends)
# needs foley[stable-audio] and the gated stabilityai/stable-audio-open-1.0 weights
# (accept its license on Hugging Face, set HF_TOKEN); backend="elevenlabs" needs
# ELEVENLABS_API_KEY and is paid
clip = foley.generate("a single wooden door creak", backend="stable_audio", duration=3)

# Grow the library — ingest auto-tags and embeds every file (captioning is pluggable)
foley.ingest("~/my_sounds/", license="user-owned")  # omit it and rights stay unknown
foley.add_from(
    "freesound", query="ocean waves", license="cc0"
)  # needs FREESOUND_API_KEY

# Compose: place the sounds under the narration (find → plan → weave)
timeline = foley.plan(candidates)  # the editable sound-design plan
result = foley.weave("narration.wav", timeline)  # mastered mix + SDH captions + credits
result.notes  # how placements were timed, e.g. estimates when foley[align] is absent
```

## How it works — four stages

| Stage      | What it does                                                                                    | Built on                                                                                                                           |
|------------|-------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------|
| **Source** | your own files · Freesound (CC0) · generate (Stable Audio Open / ElevenLabs)                    | config-driven adapters, per-sound license tracking                                                                                 |
| **Index**  | probe → tag → caption → embed every sound; hybrid keyword+semantic search                       | PANNs · CLAP · LanceDB (local→cloud via `dol`); captions come from metadata until a learned captioner is plugged in                |
| **Select** | decompose a narrative context → search → verify → generate-or-retrieve                          | CLAP retrieval + LLM decomposition + a verification ladder                                                                         |
| **Weave**  | align to the voice, duck, place, master → mastered mix + editable timeline + captions + credits | word timing (WhisperX transcription with `foley[align]`; otherwise evenly spaced script words; `notes` says which) · LUFS/EBU-R128 |

The selection tools publish as an **MCP server** (via `py2mcp`) so the same capabilities
drive the agent, a CLI, and external hosts.

## For AI agents

foley is **AI-first**: most callers are agents (directly, and via downstream packages like
`braidio` and `nw`). The headline is one call — hand it narration text and it chooses tasteful,
license-clean sounds and (given the audio) weaves them in:

```python
import foley

# Plan only — choose sounds, get an editable timeline + a rationale:
result = foley.score("She pushed open the heavy oak door; rain hammered outside.")
print(result.rationale)

# Plan + render — weave under the actual narration audio:
result = foley.score(segments, audio="narration.wav", commercial_ok=True)
result.weave.audio  # mastered mix   result.weave.captions_vtt   # SDH captions
result.weave.credits  # attribution    result.timeline             # still editable
```

`foley.score(...)` is the stable contract `braidio`/`nw` call. The same surface is available as
**MCP tools** (`foley_score`, `foley_guide`, `foley_search`, `foley_preview`, `foley_weave`, …) —
serve them over stdio (`foley-mcp`) or authenticated HTTP (`foley-mcp --http --token …`).

Ship the **agent kit** into your agent host (a `foley-sound-design` skill, a `/foley-score`
slash command, and a `sound-designer` subagent):

```default
foley agent-install            # → ./.claude/{skills,commands,agents}/…
```

The **one rule** is restraint — most sentences need no sound. The design is grounded in the
research reports below; taste heuristics (layering, ducking, onset, licensing) live in the skill.

## Design & research

foley’s design is grounded in five research reports (unified, cited):

- [Design]() · [Roadmap]()
- [Research reports]() — sound sources, SFX generation, recognition/
  tagging, embeddings/indexing, and the context-retrieval agent
- [Deep-research prompt library]() — 12
  ready-to-run prompts covering every dimension of the project

## Install

```bash
pip install foley                         # core — dol-only, light; `import foley` stays minimal
pip install "foley[audio,clap,index]"     # the usual retrieval stack (CLAP + hybrid index)
foley demo                                # no-download smoke test on the bundled fixture
foley agent-install                       # drop the agent kit into ./.claude (skill + command + subagent)
```

foley is **zero-dep at its core** and grows by capability via optional extras — each imported
lazily, so a bare install stays light. What each adds:

| Extra                                       | Adds                                                               |
|---------------------------------------------|--------------------------------------------------------------------|
| `audio`                                     | numpy/soundfile DSP + LUFS mastering                               |
| `clap`                                      | the CLAP text↔audio retrieval engine                               |
| `index` / `index-sqlite`                    | the hybrid vector+keyword index (LanceDB / sqlite-vec)             |
| `freesound` · `stable-audio` · `elevenlabs` | live retrieve / generate sources                                   |
| `agent` · `local-llm`                       | the SELECT LLM rungs (Claude / a local OpenAI-compatible endpoint) |
| `align` · `weave` · `provenance` · `c2pa`   | word timing (WhisperX), OTIO/rubberband, watermarking, signed C2PA |
| `mcp` · `obs`                               | the MCP server · OpenTelemetry export                              |

**Nothing paid runs unless you ask for it.** Having `ANTHROPIC_API_KEY` set does not switch the SELECT rungs to Claude: pass `llm="anthropic"` to `find()` / `score()` or set `FOLEY_LLM=anthropic` (`llm="local"` uses `FOLEY_LLM_BASE_URL`). By default they run the free deterministic fakes, or your local endpoint if one is configured. `with foley.offline():` (or `FOLEY_OFFLINE=1`) blocks every external source and LLM call foley makes, on every surface, raising `foley.runtime.EgressBlocked` (model weights are never downloaded offline: a model not already cached raises `foley.runtime.ModelNotCached`, naming the command to fetch it while online).

**Every paid call is priced and capped.** `foley.estimate("generate", backend="elevenlabs", duration=3)` says what a call costs before it is made (`None` means unknown, never free). A `find` / `score` / `generate` run stops before the first paid call that would take it past its `Budget(max_usd=...)` (default $1, or $3 when you chose Claude with `llm="anthropic"` / `FOLEY_LLM=anthropic`, since a default `find` with Claude is bounded at about $2.52), and a call of unknown cost needs `Budget(approve_unknown_cost=True)`. Every paid response is kept in `~/.local/share/foley/generations/` before QC, so an identical request is served from there instead of paying twice. What a backend could not honour is never silent: a parameter that changes the result (`seed` on ElevenLabs) raises, and every other drop or clamp is listed in the returned candidate’s `notes`.

`foley.check_requirements()` (and the `foley_capabilities` MCP tool) report what’s installed and
what’s degraded. Full docs: **[thorwhalen.github.io/foley](https://thorwhalen.github.io/foley)**.

## Changelog

Per-release changes are in [CHANGELOG.md]().

## License

MIT

<p class="epythet-aggregates">This documentation as a single file: <a href="foley.md">foley.md</a> (Markdown, for agents).</p>