foley.eval.golden#

The frozen golden set + the deterministic Ring-0 retrieval harness.

A GoldenItem is one (narrative context → expected sounds) judgment; its answer_clip_ids + grade become the TREC qrels the metrics score against. build_eval_library() stands up an ephemeral, in-memory library over the bundled Ring-0 synthetic fixture with injected caption vectors (via HashingBowEmbedder), so retrieval is deterministic and CLAP-free; run_ring0_retrieval_eval() runs every golden query through the real SoundLibrary.search path and scores it.

The eval fixtures (seed.json, baseline.json) are package data under foley/data/golden/ — so they ship in the wheel and foley.evaluate() / foley eval work on a bare pip install (not only from a source checkout). build_eval_library builds records directly with stable ring0:<stem> ids (not via ingest_one’s content-hash ids), so the run doc ids join 1:1 with the qrels — no alias map, no content-hash keyspace trap. The gate therefore does NOT exercise ingest_one (that path is covered by test_ingest/test_bootstrap).

Module Attributes

`RING0_MANIFEST_PATH`	The tiny wav-backed demo fixture (6 clips) — decoded by `foley.bootstrap` / `demo`.
`RING0_CORPUS_PATH`	The larger caption-only eval corpus (the 6 demo clips + ~150 synthetic clips).

Functions

`build_eval_library`(*[, embedder, manifest_path])	Build the ephemeral Ring-0 eval library (stem ids + injected caption vectors).
`load_golden`([path])	Load and validate the frozen golden set from `path` (JSON list).
`run_ring0_retrieval_eval`(*[, k, ...])	Score the golden set's queries against the Ring-0 library — the gate input.
`to_qrels`(golden)	Flatten the golden set into TREC qrels: `{query_id: {clip_id: grade}}`.

Classes

GoldenItem(id, context, expected_events, ...)

One frozen (context → expected sounds) judgment (report 08 §1.3).

class foley.eval.golden.GoldenItem(id, context, expected_events, answer_clip_ids, grade, negatives=<factory>, labeler='llm+human', schema_version=1)[source]#

Bases: object

One frozen (context → expected sounds) judgment (report 08 §1.3).

id#: Unique item id (e.g. gld_0001).

context#: The narrative paragraph (the future SELECT input).

expected_events#: One or more {query, ucs_catid, layer, diegetic, salience, ...} dicts; query is the string fed to search.

answer_clip_ids#: {ucs_catid: [clip_id, ...]} — the acceptable clips.

grade#: {clip_id: grade} (2 ideal / 1 acceptable / 0 wrong).

negatives#: Free-text distractors (unused by scoring; documentation).

labeler#: Provenance of the labels (llm+human etc.).

schema_version#: The GoldenItem schema version.

foley.eval.golden.RING0_CORPUS_PATH = PosixPath('/home/runner/work/foley/foley/foley/data/golden/corpus.json')#: The larger caption-only eval corpus (the 6 demo clips + ~150 synthetic clips). The retrieval eval needs only captions/tags/vectors — never decoded PCM — so this corpus ships without wav files, decoupling the nDCG gate’s scale from the demo fixture’s size.

foley.eval.golden.RING0_MANIFEST_PATH = PosixPath('/home/runner/work/foley/foley/foley/data/ring0/manifest.json')#: The tiny wav-backed demo fixture (6 clips) — decoded by foley.bootstrap / demo.

foley.eval.golden.build_eval_library(*, embedder=None, manifest_path=PosixPath('/home/runner/work/foley/foley/foley/data/golden/corpus.json'))[source]#

Build the ephemeral Ring-0 eval library (stem ids + injected caption vectors).

Each manifest clip becomes a SoundRecord with id ring0:<stem> and a vector = embedder.embed_text(caption + tags) — so both the vector and BM25 legs carry the caption bag-of-words and the answer clip lands at integer rank 1 (deterministic, cross-platform).

Parameters:

embedder – A text embedder (default: HashingBowEmbedder).
manifest_path – The Ring-0 manifest.json (default: the bundled fixture).

Returns:

A populated in-memory SoundLibrary.

foley.eval.golden.load_golden(path=PosixPath('/home/runner/work/foley/foley/foley/data/golden/seed.json'))[source]#

Load and validate the frozen golden set from path (JSON list).

Return type:: list[GoldenItem]

foley.eval.golden.run_ring0_retrieval_eval(*, k=10, golden_path=PosixPath('/home/runner/work/foley/foley/foley/data/golden/seed.json'), embedder=None)[source]#

Score the golden set’s queries against the Ring-0 library — the gate input.

Runs every golden expected_events[].query through the real SoundLibrary.search() path (vector ⊕ BM25 ⊕ RRF) and evaluates the resulting runs against the golden qrels.

Parameters:

k (int) – Retrieval cutoff (and the metric @k).
golden_path – The golden set JSON.
embedder – The eval embedder (default: HashingBowEmbedder).

Return type:

RetrievalReport

Returns:

A RetrievalReport (per-query + mean nDCG@10 / recall / mAP / MRR).

foley.eval.golden.to_qrels(golden)[source]#

Flatten the golden set into TREC qrels: {query_id: {clip_id: grade}}.

One qrels row per (item, event) — query_id = f"{item.id}::{event_idx}" — future-proofing multi-event items (seed items are single-event today). Each clip in answer_clip_ids carries its grade.

Return type:: dict[str, dict[str, int]]