foley.eval.fidelity#

Set-level generation-fidelity — Fréchet Audio Distance (FAD) + Kernel Audio Distance (KAD).

The distribution-level question (report 08 §4.2): *is generator A’s output distribution closer to real SFX than generator B’s?* — a release-level backend comparison over hundreds of samples, never a per-clip gate. Both metrics take (n, d) **embedding arrays**, so the hermetic-CI path and the real path differ only by which embedder produced them (a fake in CI, PANNs Wavegram-Logmel / CLAP behind foley[fit] in prod — “FAD-P” is just frechet_distance() fed PANNs embeddings). Every score carries a mandatory FidelityStamp, because FAD/KAD numbers are **not comparable across embeddings, toolkits, or versions**.

Pure numpy / stdlib — numpy is imported function-locally (import foley stays dol-only), and the PSD matrix-square-root trace is computed via numpy.linalg.eigvals (there is no scipy.linalg.sqrtm in the numpy-only CI env). Reference toolkits (fadtk / kadtk) are optional externals for reproducing exact published numbers; the functions here cover the math.

Functions

frechet_distance(x_ref, x_gen)

Fréchet distance between two embedding sets (the FAD formula).

generation_fidelity(ref_wavs, gen_wavs, *, ...)

Embed two wav sets through an injected embedder and compute FAD or KAD.

kernel_audio_distance(x_ref, x_gen, *[, ...])

Kernel Audio Distance — the unbiased RBF-MMD² between two embedding sets.

Classes

FidelityResult(metric, value, stamp)

A stamped set-level fidelity score (attached to FitReport.fidelity).

FidelityStamp(embedding, toolkit, version, ...)

Mandatory provenance for a FAD/KAD score — it is meaningless without it.
class foley.eval.fidelity.FidelityResult(metric, value, stamp)[source]#

Bases: object

A stamped set-level fidelity score (attached to FitReport.fidelity).

class foley.eval.fidelity.FidelityStamp(embedding, toolkit, version, n_ref, n_gen)[source]#

Bases: object

Mandatory provenance for a FAD/KAD score — it is meaningless without it.

FAD/KAD are not comparable across embeddings/toolkits/versions, so a bare number is uninterpretable; this stamp makes the basis of comparison legible.

foley.eval.fidelity.frechet_distance(x_ref, x_gen)[source]#

Fréchet distance between two embedding sets (the FAD formula).

‖μ_r − μ_g‖² + Tr(Σ_r + Σ_g − 2(Σ_r Σ_g)^½) — lower = the generated set’s embedding distribution is closer to the reference. Feed PANNs Wavegram-Logmel embeddings for “FAD-P”, or CLAP for a domain-matched FAD.

Parameters:

  • x_ref – Reference embeddings (n_ref, d).

  • x_gen – Generated embeddings (n_gen, d).

Return type:

float

Returns:

The FAD as a float (≥ 0 up to numerical round-off; ~0 for identical distributions).

foley.eval.fidelity.generation_fidelity(ref_wavs, gen_wavs, *, embedder, sr=48000, metric='fad', toolkit='foley-numpy', version='1')[source]#

Embed two wav sets through an injected embedder and compute FAD or KAD.

The real fidelity path is a zero-call-site-change embedder swap: a fake / hashing embedder in CI, PANNs / CLAP (foley[fit]) in prod. The embedder’s model_id is stamped into the result.

Parameters:

  • ref_wavs – An iterable of reference waveforms (1-D arrays).

  • gen_wavs – An iterable of generated waveforms (1-D arrays).

  • embedder – An object with embed_audio(wav, sr) -> vector and model_id.

  • sr (int) – The sample rate passed to embed_audio.

  • metric (str) – 'fad' (default) or 'kad'.

  • toolkit (str) – Provenance label for the stamp (default 'foley-numpy').

  • version (str) – Provenance version for the stamp.

Return type:

FidelityResult

Returns:

A FidelityResult carrying the score and its FidelityStamp.

foley.eval.fidelity.kernel_audio_distance(x_ref, x_gen, *, bandwidth=None)[source]#

Kernel Audio Distance — the unbiased RBF-MMD² between two embedding sets.

Distribution-free and small-sample-convergent; preferred over FAD for the small reference/generated sets of early-stage eval. The diagonal (same-sample) kernel terms are dropped (the unbiased estimator), so KAD(X, X) ≈ 0 and can be slightly negative.

Parameters:

  • x_ref – Reference embeddings (n_ref, d).

  • x_gen – Generated embeddings (n_gen, d).

  • bandwidth (Optional[float]) – RBF bandwidth σ (default: the median heuristic over the pooled set).

Return type:

float

Returns:

The unbiased MMD² as a float.