foley.eval.fit#

Tier-2 fit evaluation — “does the accepted clip actually match the intent?” (report 08 §2/§5).

The judge-based sibling of foley.eval.retrieval. Where Tier-1 asks *did we rank the right sound?* (a per-PR nDCG gate), Tier-2 asks *does the clip the pipeline accepted genuinely depict the event?* — audited over a seeded stratified sample of the golden set by the authoritative fit-judge (report 08 §2). It is **nightly / pre-release and cost-gated, never a silent per-PR gate**: FitReport.gate() is a no-op unless the caller passes an explicit min_fit_precision floor, and foley eval-fit is deliberately off the CI command path.

Discipline: it reuses the Tier-1 golden builders (load_golden / build_eval_library) and the #7 SELECT tools (search_sounds / gate_candidates / verify_match), but it never imports build_run / evaluate_run / to_qrels — the three symbols that construct or score the retrieval run — so it structurally cannot compute or perturb the nDCG@10 gate. Metrics are pure numpy/stdlib; the fit-judge is injected (a deterministic fake in CI, an audio-LM / LLM behind foley[fit] / foley[agent] in prod). Fit judging runs strictly AFTER the fail-closed license gate (verify_match asserts candidate.license_ok is True).

Functions

`fit_f1`(matches, relevants)	The harmonic mean of `fit_precision()` and `fit_recall()`.
`fit_precision`(matches, relevants)	Fraction of judge-accepted candidates that are actually gold-relevant.
`fit_recall`(matches, relevants)	`TP / (TP + FN)` — of the gold-relevant candidates, how many the judge confirmed.
`run_fit_eval`(*[, golden, golden_path, ...])	Run the Tier-2 fit eval over a stratified golden sample → a `FitReport`.
`stratified_sample`(units, *, strata_keys, ...)	Deterministic seeded stratified draw over `units` — the cost gate.

Classes

FitReport([per_item, fit_precision, ...])

Tier-2 fit metrics over a stratified golden sample (JSON-friendly, mirrors RetrievalReport).

class foley.eval.fit.FitReport(per_item=<factory>, fit_precision=0.0, fit_recall=0.0, fit_f1=0.0, fit_score=0.0, auto_accept_rate=0.0, n_accepted=0, n_confirmed=0, strata=<factory>, calibration=None, fidelity=<factory>, judge_model='', embedder_model_id='', seed=0, k=10, schema_version=1)[source]#

Bases: object

Tier-2 fit metrics over a stratified golden sample (JSON-friendly, mirrors RetrievalReport).

The fit numbers are co-emitted with the judge-vs-human calibration (an AlphaResult dict) so the §5 human-calibration guardrail is legible: a fit_precision is only trusted where α certifies the judge. fidelity holds stamped FAD/KAD results for release-level backend comparison. Both default empty on the pure-fake path (no human labels, no generated-set wavs).

format_regression_diff(baseline)[source]#

A human diff of fit_precision vs a committed baseline (trend reporting).

The committed fit-baseline stamp is deferred until real judges produce stable numbers; this ships the trend path only.

Return type:: str

gate(min_fit_precision)[source]#

The Tier-2 gate — a NO-OP returning True unless a floor is supplied.

Structural guarantee that Tier-2 is never a silent per-PR gate: CI passes no floor, so this is always True; only the nightly/pre-release runner sets one.

Return type:: bool

foley.eval.fit.fit_f1(matches, relevants)[source]#

The harmonic mean of fit_precision() and fit_recall().

Return type:: float

foley.eval.fit.fit_precision(matches, relevants)[source]#

Fraction of judge-accepted candidates that are actually gold-relevant.

TP / (TP + FP) where a candidate is a positive iff the fit-judge said match and gold-relevant iff its golden grade ≥ 1. This measures the JUDGE’s auto-accept purity (a low value ⇒ the judge waves through wrong clips) — NOT the ranker. Pure, closed-form, and the decisive correctness oracle (the Ring-0 + fake-judge plumbing makes the end-to-end value tautological, so the metric math is tested in isolation).

Parameters:

matches (Sequence[bool]) – Per-candidate fit-judge match booleans.
relevants (Sequence[bool]) – Per-candidate gold-relevance booleans (grade ≥ 1).

Return type:

float

Returns:

The fit precision in [0, 1] (0.0 when nothing was accepted).

foley.eval.fit.fit_recall(matches, relevants)[source]#

TP / (TP + FN) — of the gold-relevant candidates, how many the judge confirmed.

Return type:: float

foley.eval.fit.run_fit_eval(*, golden=None, golden_path=PosixPath('/home/runner/work/foley/foley/foley/data/golden/seed.json'), sample=None, strata_keys=('family', 'diegetic'), fit_judge=None, embedder=None, level=VerifyLevel.judge, seed=0, k=10, llm=None)[source]#

Run the Tier-2 fit eval over a stratified golden sample → a FitReport.

Composes the Tier-1 golden builders + the #7 SELECT loop, applies the authoritative fit-judge as an independent audit, and aggregates fit-precision/recall/F1 + fit-score

auto-accept-rate, with a per-stratum breakdown. Report-only unless min_fit_precision is supplied.

Parameters:

golden – A pre-loaded golden list (default: load golden_path).
golden_path – The golden JSON (default: the bundled Ring-0 seed set).
sample (Optional[int]) – The stratified sample cap (default: the whole set — the cost gate).
strata_keys – The stratification axes (default ('family', 'diegetic')).
fit_judge – The injected authoritative Judge (default: foley.agent.verify._default_fit_judge() for llm — the hermetic fake unless llm / $FOLEY_LLM opts in).
embedder – The Ring-0 text/audio embedder (default: HashingBowEmbedder).
level (str | VerifyLevel) – The verify rung the fit-judge audits at — 'listen' or 'judge' (default VerifyLevel.judge). 'clap' is rejected: it never invokes the fit-judge (it only re-runs the retrieval clap gate), so it cannot measure fit.
seed (int) – The sampling RNG seed.
k (int) – The retrieval shortlist depth per event.

Return type:

FitReport

Returns:

A FitReport. Gating is the caller’s job via FitReport.gate().

Raises:

ValueError – If level is 'clap' (fit judging requires a listen/judge rung).

foley.eval.fit.stratified_sample(units, *, strata_keys, sample, seed)[source]#

Deterministic seeded stratified draw over units — the cost gate.

Round-robins across strata (so an easy family cannot dominate a capped sample); sample=None keeps the full set. Each unit is (GoldenItem, event_index, event_dict).

Parameters:

units (list) – The full (item, ev_idx, ev_dict) unit list.
strata_keys – The stratification axes (e.g. ('family', 'diegetic')).
sample (Optional[int]) – The total draw size, or None for all.
seed (int) – The RNG seed (reproducibility).

Return type:

list