foley.index.ingest#

The ingestion pipeline — turn any audio file into a searchable SoundRecord.

probe -> QC -> supervised tag -> zero-shot tag -> caption -> resolve taxonomy -> embed -> assemble -> store (report 03 Stages 0-5, report 08 §3 QC gate, report 09 §5 decode-once). It is almost entirely composition over primitives that already exist:

decode / archive: foley.audio.load() / encode() (decode once, then fan the one array out to QC + taggers + embedder),

QC gate: foley.qc.run_qc() (a fail clip is quarantined, not added),

embed: the library’s ClapEmbedder (the vector is reused for zero-shot tagging — no second CLAP pass),

taxonomy: foley.index.taxonomy.resolve_catid() (tags+caption -> UCS),

store + index: foley.index.library.SoundLibrary.add() (the by-value/by-reference license gate + vector upsert + BM25 index).

Enrichment stages degrade gracefully: a missing foley[tag] (PANNs) or captioner is skipped with a note, never a crash — only the CLAP embedding is required (retrieval-first). Everything heavy is lazy-imported; importing this module costs only the stdlib.

Module Attributes

`AUDIO_EXTS`	Audio file extensions the folder walker ingests.
`UNASSERTABLE_LICENSE_IDS`	`unknown` by definition, `PDM-1.0` (a claim about the work, not a grant — verify it by building the `LicenseRecord` yourself), and the legacy `ElevenLabs-SFX` row (say the plan).
`ASSERTABLE_LICENSE_IDS`	The ids a caller may assert on ingest / restamp (`foley ingest --license`).
`LEGACY_SELECTORS`	Named selections for `restamp_rights()` (`foley restamp-rights --select NAME`).
`FROM_LICENSE_URL`	re-derive each record's id from the licence URL / label its source served (kept in `license_url`).

Functions

`content_id`(src)	Return the reproducible content-hash id foley assigns to `src`.
`has_license_url`(lic)	Whether `lic` kept the licence string its source served (re-derivable).
`ingest_folder`(path, *[, library, recursive, ...])	Ingest every audio file under `path` and return an `IngestReport`.
`ingest_one`(src, *[, library, sound_id, ...])	Ingest one clip into `library` and return an `IngestResult`.
`is_legacy_default_user_owned`(lic)	Whether `lic` is the `user-owned` stamp foley used to put on any unlicensed ingest.
`is_legacy_elevenlabs`(lic)	Whether `lic` is an ElevenLabs generation stamped before the plan was explicit (#56).
`iter_audio_files`(path, *[, recursive, exts])	Yield the audio files under `path` (or `path` itself if it is a file).
`resolve_ingest_license`(license, *[, source_url])	The rights record for a local ingest: fail-closed unless rights are asserted (#55).
`restamp_rights`([library, license, ids, ...])	Re-stamp the licence of stored sounds; a dry run unless `apply=True`.

Classes

`IngestReport`(root[, results, notes])	The rolled-up outcome of a folder ingest (JSON-serializable).
`IngestResult`(id, status[, record, qc, ...])	The outcome of ingesting one clip.

foley.index.ingest.ASSERTABLE_LICENSE_IDS = frozenset({'CC-BY-2.0', 'CC-BY-2.5', 'CC-BY-3.0', 'CC-BY-4.0', 'CC-BY-NC-2.0', 'CC-BY-NC-2.5', 'CC-BY-NC-3.0', 'CC-BY-NC-4.0', 'CC-Sampling+-1.0', 'CC0-1.0', 'MIT', 'Pixabay-Content', 'RemArc', 'Sonniss-GDC', 'Stability-Community', 'elevenlabs-free-plan', 'elevenlabs-paid-plan', 'user-owned'})#: The ids a caller may assert on ingest / restamp (foley ingest --license).

foley.index.ingest.AUDIO_EXTS: tuple[str, ...] = ('.wav', '.flac', '.aiff', '.aif', '.ogg', '.mp3', '.opus', '.m4a')#: Audio file extensions the folder walker ingests.

foley.index.ingest.FROM_LICENSE_URL = 'from-url'#

re-derive each record’s id from the licence URL / label its source served (kept in license_url).

Type:: restamp_rights(license=FROM_LICENSE_URL)

class foley.index.ingest.IngestReport(root, results=<factory>, notes=<factory>)[source]#

Bases: SerializableMixin

The rolled-up outcome of a folder ingest (JSON-serializable).

error(path, exc)[source]#

Record a per-file error without aborting the run.

Return type:: None

property errored: list[IngestResult]#: Results that raised during ingest.

exception = None#: The exception behind the last error result, kept so a raising façade can chain it (raise ... from); a plain attribute, never serialized.

property ingested: list[IngestResult]#: Results that were added to the library (pass or warn).

property quarantined: list[IngestResult]#: Results rejected by the QC gate.

record(result)[source]#

Append one IngestResult.

Return type:: None

property rights_blocked: list[IngestResult]#: Results refused by the fail-closed AI-training/license rights gate.

property skipped: list[IngestResult]#: Results skipped as content-addressed duplicates.

summary()[source]#

A counts dict for a console/CLI summary.

Return type:: dict

class foley.index.ingest.IngestResult(id, status, record=None, qc=None, notes=<factory>, error=None, cost_estimate_usd=None, cost_actual_usd=None)[source]#

Bases: SerializableMixin

The outcome of ingesting one clip.

status: 'pass'/'warn' (ingested), 'quarantined' (QC-rejected, not added), 'skipped_dup' (content already in the library), 'rights_blocked' (license forbids AI training / embedding, refused before embed — see ingest_one()), 'skipped_license' (dropped by a bootstrap commercial-use / fail-closed license filter), or 'error'. record is present only when the clip was ingested.

foley.index.ingest.LEGACY_SELECTORS = {'has-license-url': <function has_license_url>, 'legacy-elevenlabs': <function is_legacy_elevenlabs>, 'legacy-user-owned': <function is_legacy_default_user_owned>}#: Named selections for restamp_rights() (foley restamp-rights --select NAME).

foley.index.ingest.UNASSERTABLE_LICENSE_IDS = frozenset({'ElevenLabs-SFX', 'PDM-1.0', 'unknown'})#

unknown by definition, PDM-1.0 (a claim about the work, not a grant — verify it by building the LicenseRecord yourself), and the legacy ElevenLabs-SFX row (say the plan).

Type:: Licence ids that can never be asserted as verified

foley.index.ingest.content_id(src)[source]#

Return the reproducible content-hash id foley assigns to src.

Decodes src (path / bytes / file-like) to canonical PCM and hashes it — the SAME value ingest_one() mints as the record id when sound_id is None. Exposed so a caller can learn a clip’s stored id before ingesting it (e.g. the generate façade keys a content-credential sidecar by the id and sets license.c2pa_manifest_ref in the same pass — see foley.sources.generate.generate()). Cheap-ish: it fully decodes src.

Return type:: str

foley.index.ingest.has_license_url(lic)[source]#

Whether lic kept the licence string its source served (re-derivable).

Return type:: bool

foley.index.ingest.ingest_folder(path, *, library=None, recursive=True, exts=('.wav', '.flac', '.aiff', '.aif', '.ogg', '.mp3', '.opus', '.m4a'), on_error='collect', **ingest_one_kw)[source]#

Ingest every audio file under path and return an IngestReport.

Parameters:

path – A folder (walked) or a single audio file.
library – Target library (default: the process-wide default).
recursive (bool) – Recurse into sub-folders.
exts (tuple[str, ...]) – Audio extensions to ingest.
on_error (str) – 'collect' records per-file errors and continues; 'raise' re-raises the first error. Either way an EgressBlocked (offline, the embedder’s weights not cached) raises at once: every file would fail the same way.
**ingest_one_kw – Forwarded to ingest_one() (license, taggers, QC flags, …).

Return type:

IngestReport

Returns:

An IngestReport (with .summary() counts and per-file results).

foley.index.ingest.ingest_one(src, *, library=None, sound_id=None, source_uri=None, license=None, tagger=None, zeroshot_tagger=None, captioner=None, do_qc=True, min_status=QCStatus.warn, do_supervised=True, do_zeroshot=True, do_caption=True, thresholds=QCThresholds(clip_full_scale=0.999, clip_min_run=3, clip_reject_ratio=0.0001, clip_reject_run=10, true_peak_max_dbtp=-1.0, true_peak_oversample=4, dc_offset_fail=0.01, dc_offset_warn=0.001, silence_rms_dbfs=-60.0, snr_clean_db=20.0, snr_quiet_percentile=10.0, snr_frame_s=0.025, snr_hop_s=0.01, edge_rel_peak_dbfs=-40.0, edge_fade_s=0.01, lufs_gate_floor=-70.0, lufs_outlier_lu=6.0, duration_min_s=0.1, deliver_min_sample_rate=44100), store=True, allow_ai_training_forbidden=False, seed_tags=None, commercial=None)[source]#

Ingest one clip into library and return an IngestResult.

Pipeline: probe + decode-once -> content-address dedup -> QC gate -> embed (once) -> supervised + zero-shot tags -> caption -> resolve UCS -> assemble SoundRecord -> SoundLibrary.add().

Parameters:

src (Union[str, PathLike, bytes, BinaryIO]) – A path, bytes, or file-like audio source.
library – Target SoundLibrary (default: the process-wide default library).
sound_id (Optional[str]) – Optional canonical id override. Defaults to None → the content-hash of the decoded PCM (the local-ingest identity, used as the record id and dedup key). A live source adapter passes a short, case-stable, source-native id (e.g. 'freesound:12345') so dedup keys on the stable id rather than on re-fetched (lossy, byte-varying) preview bytes; when it does, the PCM hash is computed only for the (skipped) default and is not persisted. Separately, content_sha256 records the hash of the stored FLAC archive bytes (set by store_sound), which is a different byte source from this PCM hash.
source_uri (Optional[str]) – Optional by-reference fetchable URI override. Defaults to None → the resolved local path when src is path-like. A live adapter passes the stable source page URL (e.g. 'https://freesound.org/s/12345/') that foley.stores.store_sound() requires for a by-reference sound.
license (Optional[LicenseRecord]) – Rights record, or a license_id string the caller asserts ('user-owned'), or None (default): rights unknown, so the clip is indexed but foley.keep() refuses it (#55). See resolve_ingest_license().
tagger – Supervised Tagger (default: PANNs via default_tagger()).
zeroshot_tagger – Zero-shot tagger (default: CLAP via default_zeroshot_tagger()).
captioner – Optional Captioner (default: none — the caption stage is off unless one is injected).
do_qc (bool) – Run the Tier-0 QC gate.
min_status (QCStatus) – Admission floor — a QC status worse than this is quarantined (default warn: only fail clips are rejected).
do_caption (bool) – Toggle each enrichment stage.
thresholds (QCThresholds) – QC thresholds.
store (bool) – If False, assemble the record but do not add it to the library (probe/QC/enrich only).
seed_tags (Optional[list]) – Optional caller-supplied tags (e.g. a corpus’s folder-path taxonomy) unioned into the record’s tags alongside the supervised/zero-shot tags — so they feed the BM25 keyword index.
allow_ai_training_forbidden (bool) – The universal fail-closed rights gate. A sound whose license has ai_training_ok=False (e.g. Sonniss, BBC RemArc) is refused with status 'rights_blocked' before it is embedded or stored — CLAP-embedding-and-persisting is itself a form of AI training on the corpus. Pass True to record explicit operator consent and admit it anyway (see foley.bootstrap.bootstrap()’s accept_ai_restricted). Protects every ingest path, not just bootstrap.
commercial (Optional[bool]) – Whether the AI use is for a commercial purpose (decides a nc_open_source_only scope, #69). None (default) means DEFAULT_INTENDED_USE’s (commercial).

Return type:

IngestResult

Returns:

An IngestResult; its record is None when quarantined, a duplicate, or rights-blocked.

foley.index.ingest.is_legacy_default_user_owned(lic)[source]#

Whether lic is the user-owned stamp foley used to put on any unlicensed ingest.

Before #55, foley.ingest(path) with no licence stamped every file user-owned + verified. An explicit assertion now also records verified_at; the old default never did, which is what tells them apart.

Return type:: bool

foley.index.ingest.is_legacy_elevenlabs(lic)[source]#

Whether lic is an ElevenLabs generation stamped before the plan was explicit (#56).

Return type:: bool

foley.index.ingest.iter_audio_files(path, *, recursive=True, exts=('.wav', '.flac', '.aiff', '.aif', '.ogg', '.mp3', '.opus', '.m4a'))[source]#

Yield the audio files under path (or path itself if it is a file).

The shared corpus/folder walk — reused by ingest_folder() and the bulk-corpus adapters in foley.sources so the traversal is not forked.

Parameters:

path – A folder (walked) or a single audio file.
recursive (bool) – Recurse into sub-folders.
exts (tuple[str, ...]) – Audio extensions to include (lowercased suffix match).

Yields:

Each matching file as a pathlib.Path, in sorted order.

Return type:

Iterator[Path]

foley.index.ingest.resolve_ingest_license(license, *, source_url=None)[source]#

The rights record for a local ingest: fail-closed unless rights are asserted (#55).

None (no licence given) → license_id='unknown', rights_verified=False: foley.keep() refuses it for every use. The bytes are still kept locally

(cache_bytes_ok=True: it is the user’s own disk, no terms of service apply), so the clip is searchable and can be re-stamped later without re-ingesting.
a str → that license_id, which must be a LICENSE_FLAGS row, with rights_verified=True (the caller is asserting it) and verified_at set — e.g. 'user-owned' for the user’s own recordings.
a LicenseRecord → used as given.

Raises:: ValueError – If a str licence id has no LICENSE_FLAGS row.
Return type:: LicenseRecord

foley.index.ingest.restamp_rights(library=None, *, license='unknown', ids=None, where=None, select='legacy-user-owned', apply=False)[source]#

Re-stamp the licence of stored sounds; a dry run unless apply=True.

The migration for libraries built before #55 / #56. A re-ingest does not do it (content-addressed dedup skips files already stored), so this rewrites the stored records in place:

foley restamp-rights --apply — every sound an older foley stamped user-owned without being told becomes unknown (fail closed);
foley restamp-rights --license user-owned --apply — the same sounds, but you assert that you own them (now recorded with a verified_at timestamp);
foley restamp-rights --select legacy-elevenlabs --license elevenlabs-paid-plan --apply — ElevenLabs generations made before the plan was explicit;
foley restamp-rights --select has-license-url --license from-url --apply — every record that kept its source’s licence string (Freesound pulls) is re-derived by today’s mapper (CC versions, PDM, NC spellings);
corpus clips (Clotho, FSD50K) are repaired by re-running foley bootstrap, which re-stamps already-stored clips from the corpus metadata.

Parameters:

library – The SoundLibrary (default: the default one).
license (str) – The license_id to stamp (a LICENSE_FLAGS row; default 'unknown'), or FROM_LICENSE_URL to re-derive each record from its own license_url.
ids (Optional[list[str]]) – Re-stamp exactly these sound ids (overrides where / select).
where – A predicate LicenseRecord -> bool choosing the records (overrides select).
select (str) – A LEGACY_SELECTORS name (default 'legacy-user-owned').
apply (bool) – Write the change. False (default) only reports what would change.

Return type:

list[str]

Returns:

The ids that were (or, in a dry run, would be) re-stamped.

Raises:

ValueError – If license or select is unknown.