an.bench

an bench — render a fixed corpus, compute a metrics panel, write one ledger row.

The instrument Wave 2 exists to build. Its job is not to say whether the animation is good; it is to make a **deliberate degradation move a number in a direction declared in advance**, so that a future regression is caught by something other than someone noticing.

Three things about the panel that are easy to assume wrongly:

• It is not the epic’s metric list. All twelve originally-proposed metrics were refuted; every one here is a corrected form. mean adjacent-frame SSIM in particular is out — it moves the wrong way (0.958 at crf18 -> 0.977 at crf51, because a crushed video is smoother), so shipping it would put a number in the ledger that rewards the degradation the gate exists to catch.

• **Render-side and encode-side metrics are blind to each other’s mutations by construction**, and are labelled so nothing mixes them. Render-side rows compare across any machine; encode-side rows are machine-scoped.

• The vision judge is deliberately not here. Not because its input is nondeterministic — over frozen frames it is perfectly reproducible — but because a cassetted judge is a constant, invariant to the code under test, so it can never move under a deliberate degradation.

Entry points: an.bench.run.run_bench(), and an bench on the CLI.

Functions

bless_scene(capture, *, times, ...[, ...])	Write one scene's golden frames and its bless record.
compare_scene(capture, *, times, chromium_build)	Compare today's render against the committed goldens for one scene.
decode_png(data)	Decode an 8-bit truecolour PNG to (H, W, C) uint8, C in {3, 4}.
encode_png(rgb, *[, level])	Encode (H, W, 3) uint8 as an 8-bit truecolour PNG, every row filter 0.
frame_key(index)	The filename stem for a frame, zero-padded so a directory listing sorts.
pixels_sha256(rgb)	sha256 over the decoded pixels — shape and dtype included.
png_dimensions(data)	(width, height) from a PNG's IHDR, without decoding a single pixel.
read_png(path)	(H, W, 3) uint8 for a PNG on disk, alpha dropped only if opaque.
read_png_dimensions(path)	(width, height) for a PNG on disk, reading only its header.
write_png(path, rgb, *[, level])	Write rgb as a filter-0 PNG and verify the round trip before returning.
build_ledger(*, provenance, scenes)	The whole row.
build_scene_block(*, provenance, metrics, ...)	Assemble one scene's three blocks, refusing anything unreadable.
witnesses(ledger_scene, mutation)	Which metrics would count for mutation, grouped by family.
run_bench(*[, scenes, out, keep_render, ...])	Render the corpus, compute the panel, and (by default) write the row.
format_panel(ledger)	A human-readable digest of a row — the thing an bench prints.

Classes

MetricSpec(key, family, unit, optimum, ...)	One row of the panel.
Prediction(expect[, counts, gate, reason, ...])	What one metric is expected to do under one mutation, declared in advance.
Value(value[, state, gate, detail, extra])	One measured (or deliberately absent) number.

Exceptions

GoldenError	A bless was refused, or a committed golden is unusable.
PngFormatError	A PNG this module deliberately does not decode, or a malformed one.
RegistryError	A metric declaration violates one of the table's invariants.
LedgerSchemaError	A ledger row violates an invariant that would make it misreadable.
BenchError	The bench could not produce a row it would be honest to file.

exception an.bench.GoldenError

Bases: RuntimeError

A bless was refused, or a committed golden is unusable.

exception an.bench.LedgerSchemaError

Bases: ValueError

A ledger row violates an invariant that would make it misreadable.

class an.bench.MetricSpec(key, family, unit, optimum, predictions, sentence, role=None, reference='none', provisional=False, unreviewed=False, tripwire=False, requires='', notes=<factory>, sweep=None)

Bases: object

One row of the panel.

requires: str

What a scene must HAVE for this row to exist at all, or "" when the row applies to every scene (an#111).

The render-side panel rule is “nothing may be null on a real capture, because a null render-side row is a blind panel”. That rule assumes every metric could have been measured. stage_min_plane_ratio_gap could not: a displacement ratio needs two planes moving at different depths, and single_character has no planes at all — so its null is structural, not a gap in the instrument.

Declared rather than hardcoded in the test, so the panel rule keeps naming its own exceptions instead of a test file carrying a list the registry does not know about.

sweep: Sweep | None

Set only on a hard-threshold counter (an#140); see Sweep.

exception an.bench.PngFormatError

Bases: ValueError

A PNG this module deliberately does not decode, or a malformed one.

Typed and specific on purpose: the alternative to refusing is returning a plausible array, and a golden gate that compares a plausible array is worse than one that does not run.

class an.bench.Prediction(expect, counts=False, gate=None, reason='', reference=None)

Bases: object

What one metric is expected to do under one mutation, declared in advance.

expect=None means gated: the number is uninterpretable rather than good or bad, and gate says why. It is not “no change”.

exception an.bench.RegistryError

Bases: ValueError

A metric declaration violates one of the table’s invariants.

class an.bench.Value(value, state='measured', gate=None, detail='', extra=<factory>)

Bases: object

One measured (or deliberately absent) number.

an.bench.bless_scene(capture, *, times, chromium_build, reason, git, scene_contract_sha256, golden_note='', root=None)

Write one scene’s golden frames and its bless record. Refuses, loudly.

Every refusal here is a case where writing the file would produce a gate that cannot fail:

• no reason — a re-bless with no recorded reason is the same failure as a silently widened threshold, which is the named failure mode this whole wave exists to prevent;

• fewer than two frames — one frame cannot notice a scene that renders its first instant correctly and then stops;

• a pixel-identical pair — measured on promote_demo: frame 0 and the duration/2 frame differ by exactly zero pixels, so the obvious choice blesses one picture twice and the second golden tests nothing;

• an unknown Chromium build — the path keys on it, so without it the frames would be written under a name no future run could look up.

Return type:

dict

an.bench.build_ledger(*, provenance, scenes)

The whole row.

Return type:

dict

an.bench.build_scene_block(*, provenance, metrics, tripwires)

Assemble one scene’s three blocks, refusing anything unreadable.

Completeness is enforced in both directions. A metric the registry declares but the row omits is a silently narrower panel; a metric the row carries but the registry does not declare has no family, no side and no predicted direction, so nothing downstream can count it.

Return type:

dict

an.bench.compare_scene(capture, *, times, chromium_build, root=None)

Compare today’s render against the committed goldens for one scene.

Returns a dict with state in {"measured", "gated", "unavailable"}, the reduced numbers the ledger carries, and a per-frame record for provenance. The reduction is worst frame wins: identical is the conjunction and min_ssim_win8 is the minimum, because a maximum or a mean lets one clean frame hide a broken one, and this metric’s own name is “the worst small window”.

Return type:

dict

an.bench.decode_png(data)

Decode an 8-bit truecolour PNG to (H, W, C) uint8, C in {3, 4}.

Refuses — rather than approximates — 16-bit, palette, greyscale and interlaced images, naming what it found.

Return type:

Any

>>> import numpy as np
>>> a = np.array([[[1, 2, 3], [4, 5, 6]]], np.uint8)
>>> np.array_equal(decode_png(encode_png(a)), a)
True

an.bench.encode_png(rgb, *, level=9)

Encode (H, W, 3) uint8 as an 8-bit truecolour PNG, every row filter 0.

Return type:

bytes

>>> import numpy as np
>>> data = encode_png(np.zeros((2, 3, 3), np.uint8))
>>> data[:8] == PNG_SIGNATURE
True
>>> decode_png(data).shape
(2, 3, 3)

an.bench.frame_key(index)

The filename stem for a frame, zero-padded so a directory listing sorts.

Keyed on the frame index, not the pinned time: the index is what names the picture. A change to fps moves the index, which changes the path, which makes the golden absent and therefore gated — loud, and pointing at the right cause.

Return type:

str

>>> frame_key(7)
'f0007'

an.bench.pixels_sha256(rgb)

sha256 over the decoded pixels — shape and dtype included.

Never over file bytes, for the reason in the module docstring. And never over the raw buffer alone either: tobytes() carries no shape, so a 320x240 frame and a 240x320 one holding the same bytes hash identically, and issue #38’s literal “the criterion is sha256(decoded RGB array)” would report PASS on a transposed frame. Both orientations are live in this corpus. an.bench.metrics.golden_comparison() catches it separately via its shape_mismatch branch; this makes the digest agree with the gate rather than quietly disagreeing with it.

Return type:

str

>>> import numpy as np
>>> a = np.arange(24, dtype=np.uint8)
>>> pixels_sha256(a.reshape(2, 4, 3)) == pixels_sha256(a.reshape(4, 2, 3))
False

an.bench.png_dimensions(data)

(width, height) from a PNG’s IHDR, without decoding a single pixel.

Width first, matching the PNG header itself — and deliberately the opposite order from read_png(), which returns (H, W, C) like every other array here. Getting it backwards produces a square-looking check that passes on square frames only, so the order is asserted in the tests.

Cheap on purpose. The bench reads this for every frame on disk rather than sampling one, because the failure it exists to catch — a render whose frame size changed partway through — is exactly the one sampling misses.

Return type:

tuple[int, int]

>>> import numpy as np
>>> png_dimensions(encode_png(np.zeros((240, 320, 3), np.uint8)))
(320, 240)

an.bench.read_png(path)

(H, W, 3) uint8 for a PNG on disk, alpha dropped only if opaque.

Return type:

Any

an.bench.read_png_dimensions(path)

(width, height) for a PNG on disk, reading only its header.

A 1080p frame is megabytes and its declared size is in the first 24 of them. Reading only those is what makes checking every frame of every shot free rather than a second full decode of the corpus.

Return type:

tuple[int, int]

an.bench.witnesses(ledger_scene, mutation)

Which metrics would count for mutation, grouped by family.

Reads the row rather than the registry, so an#41’s criterion is evaluated against what was actually written down.

Return type:

dict[str, list[str]]

>>> witnesses({"metrics": {"m": {"family": "A",
...     "under_mutation": {"high_crf": {"counts": True}}}}}, "high_crf")
{'A': ['m']}

an.bench.write_png(path, rgb, *, level=9)

Write rgb as a filter-0 PNG and verify the round trip before returning.

The verification is not defensive noise: it is the only thing standing between a bug in this module’s own encoder and a committed golden that silently disagrees with the frame it was blessed from. Research §3 asks for exactly this — “assert the round trip at bless time against the in-memory screenshot pixels, so a bug in an’s own encoder cannot hide”.

Return type:

Any

Modules

capture	Render one corpus fixture into a throwaway copy, and hand back its artifacts.
compare	an bench --compare: read two ledger rows, and refuse when they are not comparable.
contract	scene_contract_sha256: the fact that decides whether two rows are comparable.
core_corpus	The core corpus: what an renders with no character and no genre (ADR 0001 decision 7).
corpus	The bench corpus: which projects are measured, and what each must actually render.
environment	The environment tuple — the fields that decide whether two rows may be compared.
golden	The golden gate: committed frames, compared on decoded pixels.
imageio	The pinned ffmpeg decodes, and the lossless leg every encode-side metric is measured against.
ledger	The ledger: three blocks that must never be mixed, and the guards that keep them apart.
masks	Every mask the panel is computed over, and the recorded operator for each.
metrics	The metric panel: pure numpy, no I/O, no subprocess.
mutants	The guard mutants: "I mutation-tested it" as a runnable artifact, not a claim.
mutations	The levers: deliberate, declared changes through seams the shipped code already has.
palette	Derive the set of colours a compiled shot declared — never hand-pin it.
paths	Where the bench reads its corpus from and writes its ledger to.
png	A filter-0 PNG writer and a full-filter reader — numpy and stdlib only.
registry	The metric declaration table — what each number is, and which way it should move.
run	run_bench: render the corpus, compute the panel, write one ledger row.
stage	Measuring a pan: does the stage move as planes, or as one rigid image?