an.bench.compare#

an bench --compare: read two ledger rows, and refuse when they are not comparable.

Refusing is the feature. Two rows measured on different scenes, at different resolutions, or on different x264 builds are not “one better and one worse” — every number in them is uninterpretable relative to the other, and a number reported across incomparable rows is worse than no number at all.

Three things here that are easy to get wrong in a way that still produces a plausible verdict:

A single per-metric sign mis-reports one of the two mutations. The sharpness family moves in opposite directions for AA-off and high-CRF, so the direction is read per metric and per mutation, from the under_mutation block the row already carries. Where the optimum is interior (edge_transition_width — under 1 is a staircase, 3+ is soft) there is no “better” direction at all, and the comparison says changed, never regression. Inventing a band there would be the silently-widened threshold this whole wave exists to prevent.

The two sides have opposite comparison rules, and both are measured. Render-side rows compare across any machine: zero differing pixels across arm64 macOS, x86-64 Linux and arm64 Linux, across two SwiftShader JIT backends — *at a pinned Chromium build*, which is why the browser build is itself a render-side comparability key. Encode-side rows are machine-scoped: same ISA and x264 build is byte-identical, a different ISA moves the decoded stream a little, and a different x264 build moves it by two orders of magnitude. A band wide enough to absorb that would swallow flat_field_deviation’s entire crf18->23 signal (0.0003 -> 0.0005), so the answer is to refuse, not to widen.

There is no tolerance, and none is needed. Every comparison is exact. Measured: two consecutive an bench runs on the same machine produce bit-identical numbers for every metric on all six scenes, and identical source_pixels_sha256 — the render is pinned and the encode is pinned, so a delta of exactly zero is the normal case and any nonzero delta is a real change. An epsilon here would only hide small real movements.

A direction must survive the metric’s own knobs (an#140). The two hard-threshold counters (flat_field_deviation, encode_flicker_on_held_pixels) carry their count at every cell of a small grid of their own free parameters, and a movement whose direction some cells contradict is reported unstable — neither counted toward a family nor contrary. Measured: under 4:2:0 -> 4:4:4, graded_field’s flat-field count read +84.2% at tol 6 and -81.6% at tol 8, and the panel had no way to say so. That is not a tolerance band: every cell is still compared exactly; what changes is whether the answer is a direction at all.

A fourth, quieter one: a metric’s own declaration is a comparability key. If family or optimum changed between the two rows, the metric means something different in each and the comparison is refused for that metric alone — which is why every row carries its full metric_declarations block rather than referencing the registry that happened to be installed.

Module Attributes

`SUPPORTED_SCHEMA_VERSIONS`	Row schema versions this comparer understands.
`SCENE_KEYS`	Scene-provenance fields that must match for ANY metric to be comparable.
`MASK_PARAM_PATHS`	Mask parameters, addressed by path into the scene's `masks` block.
`RENDER_ENV_PATHS`	Row-provenance paths that must match for a render-side metric.
`ENCODE_ENV_PATHS`	Row-provenance paths that must match for an encode-side metric.
`COMMON_ENV_PATHS`	Row-provenance paths that must match for either side.
`DECLARATION_KEYS`	Per-metric declaration fields that must agree, or the metric means something different in each row.
`REQUIRED_FAMILIES`	How many distinct causal families must move as declared for a mutation to count as caught.
`CROSS_CHECKED_FIELDS`	Every field a row stores TWICE — inline on the metric and in `metric_declarations` — except `under_mutation`, whose nested shape gets `_prediction_disagreements` instead.
`SCORING_FIELDS`	The fields of a per-mutation prediction that the verdict actually reads.

Functions

`compare`(before, after, *[, mutation])	Compare two ledger rows.
`direction_of`(before, after)	`increase` / `decrease` / `no_change` — exactly, with no tolerance.
`format_comparison`(report)	The human-readable digest.
`latest_rows`(*[, root, count])
`load_row`(path)	Read one ledger row from disk.
`readable_sweep`(sweep)	`sweep` if every cell is a possible `[counted, of]` pair, else `None`.
`sweep_verdict`(before, after, *[, shipped])	Does a threshold counter's direction survive its own parameter grid?

Exceptions

ComparisonError

The comparer was handed something it cannot read at all.

an.bench.compare.COMMON_ENV_PATHS: tuple[tuple[str, ...], ...] = (('render_kwargs',),)#: Row-provenance paths that must match for either side.

an.bench.compare.CROSS_CHECKED_FIELDS: tuple[str, ...] = ('family', 'side', 'comparison_scope', 'reference')#

Every field a row stores TWICE — inline on the metric and in metric_declarations — except under_mutation, whose nested shape gets _prediction_disagreements instead. Both copies are written from one registry at one moment, so a disagreement means the row was edited, and compare reads the INLINE one.

Three defects of this exact class were found in one review pass, each on a field that had been left out: family (moved a witness between families and took criterion_met_on from three scenes to five), under_mutation (flipped contrary to as_declared), and comparison_scope (compared an encode-side metric across a different ISA). The list is therefore checked for COMPLETENESS by a test rather than maintained by hand — see tests/test_bench_compare.py::test_every_doubly_stored_field_is_cross_checked.

exception an.bench.compare.ComparisonError[source]#

Bases: ValueError

The comparer was handed something it cannot read at all.

an.bench.compare.DECLARATION_KEYS: tuple[str, ...] = ('family', 'side', 'optimum', 'unit')#: Per-metric declaration fields that must agree, or the metric means something different in each row. optimum decides which way “better” points and family decides what an#41’s criterion counts.

an.bench.compare.ENCODE_ENV_PATHS: tuple[tuple[str, ...], ...] = (('environment', 'encode_side', 'isa'), ('environment', 'encode_side', 'x264_sei'), ('environment', 'encode_side', 'x264_argv'), ('environment', 'encode_side', 'pix_fmt'), ('environment', 'encode_side', 'scale_filter'), ('encode_command_source',), ('decode_commands',))#: Row-provenance paths that must match for an encode-side metric.

an.bench.compare.MASK_PARAM_PATHS: tuple[tuple[str, ...], ...] = (('masks', 'edge', 'operator'), ('masks', 'edge', 'threshold'), ('masks', 'flat', 'operator'), ('masks', 'flat', 'dilate_k'), ('masks', 'held', 'operator'), ('masks', 'ring', 'operator'), ('masks', 'render_edge', 'operator'), ('masks', 'render_edge', 'threshold'))#: Mask parameters, addressed by path into the scene’s masks block. The counts and fractions beside them are measurements and are excluded.

an.bench.compare.RENDER_ENV_PATHS: tuple[tuple[str, ...], ...] = (('environment', 'render_side', 'chromium_build'), ('environment', 'render_side', 'playwright'), ('environment', 'render_side', 'launch_argv'))#: Row-provenance paths that must match for a render-side metric. The Chromium build is here and not merely informational: the cross-arch verdict measured ISA- and OS-invariance at a pinned build, and an#38’s golden path keys on the build for the same reason. One bump has been measured to move zero pixels (1187 -> 1223), so this refusal is precautionary rather than a known break — and a deliberate re-bless is the intended response.

an.bench.compare.REQUIRED_FAMILIES: int = 3#: How many distinct causal families must move as declared for a mutation to count as caught. an#41’s criterion, restated from the research: “>=3 metrics from >=3 distinct causal families, evaluated per mutation, with a per-metric per-mutation sign declared in advance”.

an.bench.compare.SCENE_KEYS: tuple[str, ...] = ('scene_contract_sha256', 'resolution', 'fps', 'n_frames', 'shot_order', 'palette_hex', 'tolerances')#: Scene-provenance fields that must match for ANY metric to be comparable. Deliberately not the whole provenance block: that also carries per-run measurements (mask pixel counts, wall_seconds, the golden diagnostics), which change exactly when the render changes — i.e. when the two rows are most worth comparing.

an.bench.compare.SCORING_FIELDS: tuple[str, ...] = ('expect', 'counts', 'gate', 'state')#: The fields of a per-mutation prediction that the verdict actually reads. reason is prose and is deliberately NOT here — the declarations block carries it and the inline block drops it, so requiring it would refuse every real row.

an.bench.compare.SUPPORTED_SCHEMA_VERSIONS: tuple[int, ...] = (1,)#: Row schema versions this comparer understands. A row it cannot read is refused rather than guessed at — the whole point of the version field.

an.bench.compare.compare(before, after, *, mutation=None)[source]#

Compare two ledger rows. Returns a report; raises only on an unreadable row.

mutation selects the question. With one, the report answers an#41’s: did the declared witnesses move in the declared direction, and did at least three distinct causal families do so. Without one, it answers “is the second row worse”, which only the one-sided metrics can answer at all.

Return type:: dict

an.bench.compare.direction_of(before, after)[source]#

increase / decrease / no_change — exactly, with no tolerance.

Two consecutive runs on one machine are bit-identical, so zero is the normal delta and any nonzero one is real. Booleans compare as booleans: a tripwire that went True -> False has not “decreased”.

Return type:: str

>>> direction_of(1.0, 1.0)
'no_change'
>>> direction_of(True, False)
'decrease'

an.bench.compare.format_comparison(report)[source]#

The human-readable digest. Refusals first, because they are the verdict.

Return type:: str

an.bench.compare.latest_rows(*, root=None, count=2)[source]#

Return type:: list[Path]

The most recent committed ledger rows, newest last — **by `generated_at`**.

Not by filename. Filenames are <date>-<sha7>[-dirty].json, so a filename sort orders same-day rows by sha hex. On a Wave 3 PR the re-baseline and the after-run are plausibly the same day, and when the after-commit’s sha sorts lower a bare an bench-compare silently swaps before and after and reports every improvement as a regression.

A row whose generated_at cannot be read sorts before every dated one, so it is dropped as soon as two dated rows exist: a corrupt or pre-schema file must never become the after row a verdict is drawn from. The filename stays the tiebreak, which is what keeps a directory whose rows all lack the key ordered by date.

-dirty rows are excluded: a row measured against uncommitted edits describes no commit, and comparing one is comparing against nothing nameable. an bench --out still lets a caller point at one explicitly.

an.bench.compare.load_row(path)[source]#

Read one ledger row from disk.

Return type:: dict

an.bench.compare.readable_sweep(sweep)[source]#

sweep if every cell is a possible [counted, of] pair, else None.

Possible means integers (a bool is not a count), of > 0 and 0 <= counted <= of — a cell no measurement could have produced is an edited row, and reading it would let the edit choose the verdict.

Return type:: dict | None

>>> readable_sweep({"t=1": [3, 10]})
{'t=1': [3, 10]}
>>> readable_sweep({"t=1": [11, 10]}) is None
True
>>> readable_sweep({"t=1": [True, 10]}) is None
True

an.bench.compare.sweep_verdict(before, after, *, shipped=None)[source]#

Does a threshold counter’s direction survive its own parameter grid?

an#140. flat_field_deviation counts pixels past a hard threshold over a mask eroded by a second free parameter, and a row carries its count at every cell of a declared grid of both (sweep: {cell: [counted, of]}). This reads the direction at every cell the two rows SHARE. When some cells rise and others fall, the shipped value’s direction is a property of where the threshold happened to be set, not of the change under test — measured, a 4:2:0 -> 4:4:4 change moved graded_field +84.2% at tol 6 and -81.6% at tol 8 — and state is unstable. no_change cells contradict nothing.

Exact, like everything else here: fractions are compared by cross-multiplying the integer counts, never as rounded floats.

Shared cells rather than identical grids: a dilation whose mask is empty in one row is omitted from that row, and discarding the other fourteen cells for it would let the verdict count unchecked. unknown only when a side has no readable sweep, or the shipped cell is not among the shared ones — the check could not run, which is a caveat and not a refusal, because a row written before an#140 has no sweep and is otherwise perfectly readable. (A row that DECLARES a sweep and carries none is refused upstream, in _compare_scene.)

Return type:: dict | None

>>> sweep_verdict({"t=1": [1, 10], "t=2": [5, 10]}, {"t=1": [2, 10], "t=2": [4, 10]})["state"]
'unstable'
>>> sweep_verdict({"t=1": [1, 10]}, {"t=1": [2, 20]})["state"]
'stable'
>>> sweep_verdict({"t=1": [1, 10]}, None)["state"]
'unknown'
>>> sweep_verdict(None, None) is None
True