Dario LipicarandClaude Opus 5 318286de7a conformance: the consumer axis, two relocks, and the two races they exposed (#22) ...

* build: relock onto test-modules and logoscore-cli master

test-modules ff5dbbd brings the ext qtproxy fixture this branch consumes, and
the conformance registries with the arity entries retired.

logoscore-cli 1ec3683 carries ed19258 (#99) and b3f1a403 (#101), which is what
retires the pre-99 cells: the daemon reads the error channel instead of
inferring failure from a null RESULT, and asks the module for
getPluginMethods on a null return so an unknown name becomes METHOD_NOT_FOUND.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* conformance(ext): run the ext table through the Qt consumer, twice

The ext gate ran one consumer point and computed no consumer differential. It
now runs the same three-consumer shape the full_api gate has — py,
extqtproxy-sync, extqtproxy-async — against test_fullapi_ext_qtproxy, the
Qt-typed consumer of full_api_ext (a SEPARATE module from test_fullapi_qtproxy,
because a Qt consumer wrapper is generated per contract).

THE PROBE IS THE ONLY DRIVER CHANGE, and it is a real one. A consumer point with
a call mode proves the selected wrapper table actually ran by making one
known-good call and reading `lastCallStatus()` back — `useCallMode("async")`
returning true only says the flag is set. That probe was hard-coded to
`echoInt(1)`, which is a full_api method. full_api_ext has none like it: no
method takes a bare scalar, and probing its one zero-parameter method with a
spare `1` would work only because of a REGISTERED defect (known-ext.json
B-arity-overflow drops extra arguments) and would break the day that is fixed.

So `--proxy-consumer` grew a fifth field: `METHOD[:JSON_ARGS]`, defaulting to
`echoInt:[1]` so no existing invocation moves. Arguments go through the table's
own `materialize`, so bytes are spelled `{"_bytes": ...}` in the probe exactly
as they are in a case. A malformed spec is REFUSED rather than defaulted —
falling back to `echoInt` would silently stop proving the call mode and the run
would stay green. tests/unit/test_matrix_proxy_consumer.py pins that, the
default, and the four-field spelling.

Measured on framework.lan with logos-test-modules at its matching branch:
264 cells, 217 pass / 47 xfail / 0 fail / 0 xpass / 0 uncovered.

* conformance: an error expectation may name a set of acceptable codes

`matches` tested one code as a substring of the reported error. A cell whose
answer is decided by which of two deadlines fires first cannot be spelled that
way: asserting either token asserts the timing rather than the claim.

A LIST means any-of. Deliberately not a general escape hatch -- widening a
single-code expectation weakens it, so a case that does this owes an
explanation in `why` of what the codes have in common. The one case that uses
it (failure/A/module-not-loaded) gives it: both are reported on the error
channel and name a transport reason, and a VALUE or a provider-side code still
fails.

The single-string form is untouched, and the new test pins both -- including
that a successful call is still a failure for an error expectation, which is
the property the widening must not cost.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* conformance: retry the SUBSCRIBE half of the event race, not just the fire half

`capture_event` already knew the watcher and the fire race each other, and
re-fired on a cadence to close it. Subscribing races too, and that half was
attempted exactly once: `on_event` asked too early is REFUSED, the daemon
answers WATCH_FAILED, and nothing tried again -- so an event cell reported a
transport error instead of a value.

MEASURED, across three otherwise identical runs of the same table at the same
revision: event/tstr on test_fullapi_rust failed, passed, then failed. In both
failing runs BOTH proxy consumers received "hello" from the same provider,
which is what rules out the module: it was emitting, and the `py`
subscription was losing the race. It is not a regression from the SDK bump
either -- the bump is what made module startup slow enough to lose more often.

Retried against the same deadline as the fire loop rather than a fixed count,
so a slow start costs time instead of a false negative. A subscription that
never succeeds still raises and still fails the cell, so this closes a race
without hiding a refusal.

Not registered as an xfail: the cell PASSES most of the time, and a registered
cell that passes is an xpass, which reddens the gate. A flake has to be fixed
or left visible; it cannot be parked.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* conformance: an any-of expectation declares the DIFFERENTIAL too, not just the cell

Widening failure/A/module-not-loaded to accept either transport code fixed the
CELL and left three differential rows failing: qtproxy-sync answered
RPC_FAILED while qtproxy-async answered object_unavailable, and the
differential compares actual values independently of `expect`. I had assumed
the delta rows follow the cell's status; they do not, and the run said so.

That independence is deliberate and worth keeping -- it is what caught the
`void` divergence nobody predicted -- so this does not become "both sides
matched the expectation, therefore they agree".

The narrow rule instead: an `expect` naming a LIST of codes has already said
more than one answer satisfies the claim, which is the same kind of statement
`expect_by_provider` makes for the provider axis. It gets the same treatment --
status `declared`, printed so the divergence stays visible, not counted as a
failure. It applies only when the expectation names several codes AND BOTH
sides satisfy it; a coordinate answering something outside the set, or a value
instead of an error, still fails.

One implementation, at module scope, called by the differential and exercised
directly by the test -- the first draft had the rule written twice, once in a
closure and once for the test to reach, which is a pair that drifts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test: move the driver tests to the file that loads the driver

CI red on both arches, 2 failed / 140 passed:

  AttributeError: module 'matrix_report' has no attribute 'matches'
  AttributeError: module 'matrix_report' has no attribute 'Result'

I put two tests for run_matrix.py's `matches` and `declared_multi_divergence`
into tests/unit/test_matrix_report.py, whose `R` is the matrix_report module.
The driver is a different file and is loaded by importlib, in
tests/unit/test_matrix_proxy_consumer.py. The tests move there, unchanged in
substance.

Why this reached CI at all: I could not run pytest locally and shipped the
tests unexecuted. That was wrong twice over — `python3 -m pytest` failed only
because the homebrew interpreter has no pytest, and pyenv had one the whole
time. "The tool is missing" was a conclusion I drew from one command instead of
looking.

Now run: 138 passed, 4 skipped locally (the 4 skips are environmental — the
shared conformance table is not resolvable outside the nix build; CI resolves
it and runs them). Total matches CI's 142.

Both conformance matrices were already green in CI on this branch; this touches
nothing they exercise.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

2026-08-26 15:42:34 -03:00

21 MiB

Raw Permalink Blame History

The file is too large to be shown. View Raw