foley.index.protocols

Structural contracts for the foley index (the retrieval boundary).

These three Protocol``s are the small, swappable seams report 04 §6.2 calls for: keep the boundary as *protocols*, give each a zero-config sensible default, and let every piece be replaced by keyword injection. They are the SSOT for the Index-stage contracts (report 10 §4.2) and are deliberately kept here, separate from any concrete backend, so the façade (:mod:`foley.index.library`) and the search logic (:mod:`foley.index.search`) depend only on the interface — never on ``torch / lancedb / sqlite_vec.

• Embedder — the joint text<->audio space (CLAP default).

• VectorIndex — approximate-nearest-neighbour over embeddings.

• KeywordIndex — BM25 / full-text over tags + caption.

A single backend object may satisfy both index protocols at once (LanceDB holds the vector column and the FTS index in one table); the façade simply passes it as both vindex and kindex.

This module is stdlib-only: ndarray appears solely in annotations, imported under TYPE_CHECKING so import foley.index.protocols never pulls numpy.

Classes

Captioner(*args, **kwargs)	Produce one natural-language sentence describing a clip (report 03).
Embedder(*args, **kwargs)	A joint text<->audio embedding space (CLAP by default).
KeywordIndex(*args, **kwargs)	BM25 / full-text index over each sound's tags + caption.
Tagger(*args, **kwargs)	Map a clip to (label, score) pairs against a label vocabulary.
VectorIndex(*args, **kwargs)	Approximate-nearest-neighbour store over embedding vectors.

class foley.index.protocols.Captioner(*args, **kwargs)

Bases: Protocol

Produce one natural-language sentence describing a clip (report 03).

The caption feeds the BM25 keyword index and human display. Default is a dedicated AAC model (EnCLAP); Qwen2-Audio is a richer promptable upgrade. Both are foley[caption] adapters plugged in behind this protocol.

caption(wav, sr)

Return a one-sentence caption for the clip.

Return type:

str

class foley.index.protocols.Embedder(*args, **kwargs)

Bases: Protocol

A joint text<->audio embedding space (CLAP by default).

One space serves both text->audio search (embed_text a query) and audio<->audio similarity (embed_audio a clip). Implementations MUST return L2-normalized float32 arrays so a plain inner product is cosine similarity, and MUST stamp model_id/dim so mixed-model libraries stay coherent (each SoundRecord records the embedding_model/embedding_dim it was indexed under).

model_id

The checkpoint id (e.g. 'laion/larger_clap_general').

dim

The embedding dimensionality (e.g. 512).

embed_audio(wav, sr)

Embed one audio clip.

Parameters:

• wav (ndarray) – A working-array clip (float32, mono preferred). CLAP expects 48 kHz; implementations resample as needed.

• sr (int) – The clip’s sample rate in Hz.

Return type:

ndarray

Returns:

A 1-D (dim,) L2-normalized float32 array.

embed_text(text)

Embed one or more query strings.

Parameters:

text (Union[str, list[str]]) – A single string or a list of strings.

Return type:

ndarray

Returns:

A 2-D (n_texts, dim) L2-normalized float32 array (n_texts is 1 for a single string) — always 2-D so callers can index [0] for the single-query case.

class foley.index.protocols.KeywordIndex(*args, **kwargs)

Bases: Protocol

BM25 / full-text index over each sound’s tags + caption.

The default is LanceDB’s Tantivy FTS (report 04 §3.4); SQLite FTS5 is the single-file fallback. Same where push-down contract as VectorIndex.

bm25(query, k, *, where=None)

Return the top-k BM25 matches for query, best first.

Parameters:

• query (str) – A natural-language / keyword query.

• k (int) – Number of matches to return.

• where (Optional[dict]) – Optional metadata predicates for push-down filtering.

Return type:

list[tuple[str, float]]

Returns:

[(id, bm25_score), ...] in descending-score order.

index(id, text, meta)

Insert or replace the searchable text (and light metadata) for id.

Return type:

None

class foley.index.protocols.Tagger(*args, **kwargs)

Bases: Protocol

Map a clip to (label, score) pairs against a label vocabulary.

Used on ingest to auto-fill SoundRecord.audioset_labels (supervised) and tags (zero-shot). The default supervised tagger is PANNs CNN14 over AudioSet; the default zero-shot tagger scores a clip against a custom/UCS label set via CLAP (report 03). BEATs/AST are drop-in upgrades. Consumes the working array (float32, any sr — the impl resamples to its model’s rate).

tag(wav, sr, *, taxonomy='audioset', top_k=10)

Return the top-k (label, score) tags for the clip, best first.

Return type:

list[tuple[str, float]]

class foley.index.protocols.VectorIndex(*args, **kwargs)

Bases: Protocol

Approximate-nearest-neighbour store over embedding vectors.

The default is LanceDB (report 04 §2); Qdrant/pgvector/sqlite-vec bind the same protocol behind the scenes. where is an optional metadata push-down the façade may pass; a backend that cannot push filters down MAY ignore it (the façade over-fetches and post-filters to stay correct either way).

get_vector(id)

Return the stored vector for id (or None if absent).

Needed by SoundLibrary.similar (fetch a sound’s own vector, then run knn()) and by the optional CLAP rerank (score keyword-only hits).

Return type:

Optional[ndarray]

knn(vector, k, *, where=None)

Return the k nearest ids to vector, most-similar first.

Parameters:

• vector (ndarray) – A (dim,) query vector (already L2-normalized).

• k (int) – Number of neighbours to return.

• where (Optional[dict]) – Optional metadata predicates for push-down filtering.

Return type:

list[tuple[str, float]]

Returns:

[(id, cosine_similarity), ...] in descending-similarity order.

upsert(id, vector, meta)

Insert or replace the vector (and light metadata) for id.

Return type:

None