foley.eval.retrieval

Tier-1 retrieval metrics + run assembly — pure stdlib, CI-safe, ranx-exact.

The metrics (nDCG@10, Recall@k, Precision@k, mAP@10, MRR@10) are hand-rolled in stdlib (no ranx/numba) so the nDCG PR gate runs in foley’s numpy-only CI with zero heavy deps — yet they are **numerically identical** to ranx/pytrec_eval (guarded by the opt-in tests/eval/test_metrics_vs_ranx.py oracle). nDCG uses the linear Järvelin gain Σ grade_i / log2(rank+1) — the definition ranx’s "ndcg@10" and pytrec_eval’s ndcg_cut_10 actually implement (report 08 §1.1’s 2^grade−1 is a doc bug).

Vocabulary (TREC-style):

• qrels — {query_id: {clip_id: grade}} — the ground-truth relevance grades (0 = wrong, 1 = acceptable, 2 = ideal); the raw grade is the gain, so the metrics are grade-scale-agnostic.

• run — {query_id: {clip_id: score}} — the system’s scored results.

build_run() assigns strictly-descending integer scores in the search order, so a run has no score ties and its ranking reproduces search()’s (already id-tiebroken) order exactly — the source of cross-platform determinism.

Module Attributes

DEFAULT_METRICS

The metrics reported by evaluate_run() (nDCG@10 is the sole PR gate).

Functions

average_precision_at_k(qrels_q, run_q[, k, ...])	Average precision (trec_eval map: divide by TOTAL relevant, not min(R,k)).
build_run(candidates)	Turn an ordered search() result into a run row with distinct scores.
evaluate_run(run, qrels, *[, k, metrics, ...])	Score a run against qrels, returning a RetrievalReport.
mean_over_queries(values)	Macro-average of per-query metric values (0.0 for an empty list).
mrr_at_k(qrels_q, run_q[, k, rel_lvl])	Reciprocal rank of the first relevant doc in the top-k (0.0 if none).
ndcg_at_k(qrels_q, run_q[, k])	Normalized DCG at k with linear gains (0.0 when no graded answer).
precision_at_k(qrels_q, run_q[, k, rel_lvl])	Fraction of the top-k that is relevant (denominator is literal k).
recall_at_k(qrels_q, run_q[, k, rel_lvl])	Fraction of ALL relevant docs (grade ≥ rel_lvl) retrieved in the top-k.

Classes

RetrievalReport([per_query, mean, ranks, k])

Per-query + mean Tier-1 metrics over a golden set (JSON-friendly).

foley.eval.retrieval.DEFAULT_METRICS = ('ndcg@10', 'recall@10', 'map@10', 'mrr@10')

The metrics reported by evaluate_run() (nDCG@10 is the sole PR gate).

class foley.eval.retrieval.RetrievalReport(per_query=<factory>, mean=<factory>, ranks=<factory>, k=10)

Bases: object

Per-query + mean Tier-1 metrics over a golden set (JSON-friendly).

format_regression_diff(baseline)

A human diff for a failing gate: mean vs baseline + the worst queries.

Return type:

str

foley.eval.retrieval.average_precision_at_k(qrels_q, run_q, k=10, *, rel_lvl=1)

Average precision (trec_eval map: divide by TOTAL relevant, not min(R,k)).

Return type:

float

foley.eval.retrieval.build_run(candidates)

Turn an ordered search() result into a run row with distinct scores.

Assigns each candidate a strictly-descending integer score (len - rank) in the order returned, so the run carries **no score ties** and _ranked_ids() reproduces the search ranking exactly — the determinism the gate relies on. The doc id is candidate.sound.id.

Parameters:

candidates – An ordered list[Candidate] from foley.index.library.SoundLibrary.search() (best first).

Return type:

dict[str, float]

Returns:

{clip_id: score} with distinct descending scores.

foley.eval.retrieval.evaluate_run(run, qrels, *, k=10, metrics=('ndcg@10', 'recall@10', 'map@10', 'mrr@10'), rel_lvl=1)

Score a run against qrels, returning a RetrievalReport.

Only query ids present in qrels are scored (a run row without a ground-truth judgment is ignored). ranks records, per query, the 1-based rank of the highest-graded answer (for the failure diff).

Return type:

RetrievalReport

foley.eval.retrieval.mean_over_queries(values)

Macro-average of per-query metric values (0.0 for an empty list).

Return type:

float

foley.eval.retrieval.mrr_at_k(qrels_q, run_q, k=10, *, rel_lvl=1)

Reciprocal rank of the first relevant doc in the top-k (0.0 if none).

Return type:

float

foley.eval.retrieval.ndcg_at_k(qrels_q, run_q, k=10)

Normalized DCG at k with linear gains (0.0 when no graded answer).

Return type:

float

foley.eval.retrieval.precision_at_k(qrels_q, run_q, k=10, *, rel_lvl=1)

Fraction of the top-k that is relevant (denominator is literal k).

Return type:

float

foley.eval.retrieval.recall_at_k(qrels_q, run_q, k=10, *, rel_lvl=1)

Fraction of ALL relevant docs (grade ≥ rel_lvl) retrieved in the top-k.

Return type:

float