foley.eval.reliability#
Inter-rater reliability — Krippendorff’s α + the judge-vs-human calibration guardrail.
Whatever judges foley’s fit — humans or an LLM/audio-LM — you must **quantify agreement before trusting the scores** (report 08 §2.3). This module is the numbers side of that guardrail: one pure-numpy Krippendorff’s α (the most general reliability coefficient — any number of raters, ordinal/graded labels, missing-data tolerant), reused for BOTH human inter-rater reliability on the gold grades AND judge-vs-human calibration (the model judge is just one more rater row). A model judge is only promoted to unattended use once it reaches human-level agreement on a calibration slice.
Pure numpy / stdlib, deterministic — numpy is imported function-locally so importing
this module keeps import foley dol-only (the same discipline as
foley.eval.retrieval). No scipy, no external reliability toolkit (a PyPI
krippendorff oracle cross-check lives behind an opt-in test only).
Module Attributes
Krippendorff's benchmark bands for a reliability coefficient (report 08 §2.3). |
Functions
|
Compute judge-vs-human agreement and the promotion verdict (report 08 §2.3). |
|
Krippendorff's α over |
|
Raw pairwise percent agreement, reported ALONGSIDE α (report 08 §2.3). |
|
Map an α (or κ) to Krippendorff's benchmark band. |
Classes
|
A judge-vs-human calibration record (surfaced as |
- foley.eval.reliability.ALPHA_RELIABLE: float = 0.8#
Krippendorff’s benchmark bands for a reliability coefficient (report 08 §2.3).
- class foley.eval.reliability.AlphaResult(alpha, level, n_units, n_raters, percent_agreement, band, promoted)[source]#
Bases:
objectA judge-vs-human calibration record (surfaced as
FitReport.calibration).Codifies the promotion rule in one legible object: a model judge is
promotedto unattended use only once its agreement with the human raters reaches theALPHA_RELIABLEband.
- foley.eval.reliability.calibrate_judge_vs_human(human_grades, judge_grades, *, level='ordinal')[source]#
Compute judge-vs-human agreement and the promotion verdict (report 08 §2.3).
Stacks the human rating row(s) and the model-judge row into one reliability matrix and returns an
AlphaResult;promotedisTrueiff α reaches the reliable band — the model judge may then be trusted unattended on that slice.- Parameters:
human_grades – A 1-D per-unit human grade sequence, or a 2-D
(humans × units)matrix (nan= missing).judge_grades – The model judge’s 1-D per-unit grades (
nan= missing).level (
str) – The measurement level (default'ordinal').
- Return type:
- Returns:
An
AlphaResult.
- foley.eval.reliability.krippendorff_alpha(reliability_data, *, level='ordinal', value_domain=None)[source]#
Krippendorff’s α over
(raters × units)ratings (nan= missing).α = 1 − D_o/D_e— 1 is perfect agreement, ~0 is chance, negative is systematic disagreement. Ordinal is the default (foley’s 0–3 grades); missing ratings are tolerated (units with < 2 ratings are dropped, no imputation). Pure numpy.- Parameters:
reliability_data – A 2-D array-like
(raters × units);nan= missing.level (
str) –'ordinal'(default) |'nominal'|'interval'.value_domain – Optional explicit value set (for a label unobserved by some rater).
- Return type:
- Returns:
α as a bare
float(1.0when there is no expected disagreement,D_e = 0).
- foley.eval.reliability.percent_agreement(reliability_data)[source]#
Raw pairwise percent agreement, reported ALONGSIDE α (report 08 §2.3).
On foley’s skewed ‘most candidates irrelevant’ label distribution a chance-corrected α can look pessimistic, so the uncorrected agreement is co-reported for context.
- Parameters:
reliability_data – A 2-D array-like
(raters × units);nan= missing.- Return type:
- Returns:
The fraction of same-unit rater pairs that agree (
0.0if no pairs).