mirror of
https://github.com/logos-co/logos-logoscore-py.git
synced 2026-08-27 11:11:09 +00:00
Dario LipicarandClaude Opus 5
d2475cfc1e
* feat(conformance): the py driver for the LIDL type matrix
The driver half of the conformance matrix; the case table and the xfail
registry live in logos-test-modules/conformance/, with the providers they
describe. The driver lives here because it uses this package's client — and
logoscore-py already depends on logos-test-modules, so the reverse would be a
cycle.
`conformance/run_matrix.py` replays every case against BOTH providers and
reports one JSONL line per cell, keyed by coordinate. `checks.conformance-matrix`
runs it; it fails on a red cell, on an `xpass` (a registered known-broken cell
that started passing — the registry has to be updated), or on a (type, position)
the contract declares that no case covers.
Three things it took a run to get right, all recorded in the code:
* `same()` is type-strict — a matrix that compares with `==` cannot see an
integer degrading to a float, which is most of what this exists to catch,
and `1 == True` would pass too.
* a case may set `"raw": true` to opt out of tagged-bytes materialization.
Without it the adversarial `_bytes`-collision cases are inexpressible: the
driver would convert BOTH the argument and the expectation to bytes and the
cell would compare bytes to bytes and pass no matter what the system did —
the exact fake-green the matrix is meant to remove.
* a dispatch rejection arrives as the RESULT ({"code": "dispatch_failed"}),
not as a raised error, so a driver that only watches for exceptions records
a rejection as a successful call returning a dict.
Each provider gets one daemon PER PHASE. Sharing a daemon between the ~80-call
method phase and the event phase wedged it partway through the events under the
nix sandbox — the last five failed contiguously with RPC_FAILED while every one
of them passes on a fresh daemon. A phase that can poison the next one makes a
red cell mean "something earlier used up a resource", which is the kind of
unreliable signal this exercise exists to remove.
Verified end to end: green (139 pass / 13 registered xfail / 68 differential),
and RED on 19 cells when pointed at the pre-fix logoscore CLI — so it demonstrably
catches the class of bug it is for, rather than merely claiming to.
Also adds a unit test asserting the inline `_fullapi_module_cases.py` table and
the shared `cases.json` agree. It does not merge them — rewriting a passing
integration suite to prove a point is a bad trade — but it makes the drift
between two hand-maintained copies a red test instead of a silent surprise.
(Confirmed it fails on an injected divergence, not just that it passes.)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(conformance): N-ary events and positional cells in the driver
Follows the arity surface added to the contract. Three changes:
* an event case may carry `values` (a list) instead of `value`; the driver
fires with *values and compares the ORDERED argument list, so slot order is
part of the assertion rather than something the payload shape hides;
* a multi-parameter event arrives as {arg0, arg1, ...} — the driver rebuilds
the ordered list, and a single-argument event still reduces to its one
value, so no existing case changes;
* coverage now distinguishes `method_arg` from `method_arg@k` (and the event
equivalent). A sole argument cannot catch a generator that mixes up
positional slots, so they are genuinely different cells, and a case that
covers several declares the (type, position) PAIRS explicitly instead of
the cross-product of its type and position lists — the cross-product would
claim cells the case never exercises.
79 cases x 2 providers, green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(conformance): data-driven provider set + the ext matrix check
`--modules NAME=DIR` (repeatable) generalises the two hardcoded provider flags,
so a table with a different provider set runs through the SAME driver instead of
a second one. The full_api_ext table has one provider — the C++ cdylib backend
cannot express records or [bstr] yet — so the differential simply has nothing to
compare there; it is not silently skipped, there is just no pair.
The table names the providers it describes and the driver runs the intersection,
failing loudly if a declared provider has no module dir rather than quietly
reporting a smaller matrix.
Adds `checks.conformance-matrix-ext`: 20 cases, full contract coverage, green
with 9 registered xfails (E1 bytes-at-depth mangling, E2 empty-bytes-at-depth
dropped to null, E3 M1 through a record field).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(conformance): per-case provider precision + the C++ ext provider
The ext check now runs both ext providers, so that table has a differential like
full_api does.
`known.json` entries may carry `per_case_providers`, narrowing an individual
case to the providers that actually fail it. `providers` alone is an entry-wide
union, and a defect that only one provider surfaces then gets registered against
both — so the provider that PASSES is reported as `xpass` and the registry
manufactures a failure it then demands you fix. 4 of E1's 9 cases are
single-provider, which is how this surfaced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(conformance): state what this driver does not cover
The module docstring claimed "Other consumers (the C++, Rust and QML proxies)
replay the same cases.json". No such driver exists — `--consumer` is a label
written into the report, not a driver selector. Grepping the workspace finds
only this one.
That overclaim mattered. The event bridge failed to decode canonical
{"_bytes": ...} into a QByteArray, and this driver could not see it: the
undecoded map round-trips to JSON and the python client decodes the tag itself.
The cells stayed green while a Qt/C++ or QML subscriber got a map. A matrix with
one consumer cannot see a defect that its own consumer happens to undo, and a
docstring promising four consumers hides that.
Also states the transport limit: the daemon is built with no `transports=`, so
every cell is measured over LocalSocket/QtRO and the plain wire's separate uint64
defect is out of reach.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(fullapi): 64-bit boundaries, and re-pin onto the merged chain
The integration matrix — replayed by the local, tcp and tcp_ssl checks and by
the docker codec matrix — topped out at 2^53-1 for `int` and 2^32-1 for `uint`.
So the 64-bit band that the whole LIDL type contract is about was untested on
every transport, and it was broken on two of them.
Against the pinned protocol over tcp the new cases fail exactly as the code
predicts:
echoUint(2^63) -> -9223372036854775808 (RpcValue has no unsigned alt)
echoUint(2^64-1) -> -1 (same)
uintEvent(2^64-1) -> 1.8446744073709552e+19 (the event bridge, M6)
Two different defects, on the same transport, that no existing case could see.
Both are fixed in logos-protocol; all 68 pass on local, tcp and tcp_ssl.
2^53+1 is in the table on purpose: it is the smallest integer a double cannot
represent, so it separates "degraded through a float" from "wrapped as an
integer" — the two failure modes look alike in a report and have different
causes.
Also re-pins onto the merged chain now that logos-test-modules#28 landed:
logos-test-modules -> d4c0d04 (the conformance matrix)
logos-logoscore-cli -> a143727 (64-bit call args)
That re-pin matters beyond housekeeping: the lock previously resolved
logos-protocol to a pre-#29 revision, so the matrix's own registry claimed fixes
that the code under measurement did not contain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* chore: re-pin onto the merged uint64 chain
logos-logoscore-cli -> 94f54b3 (#75 — carries logos-protocol 8b8a358)
logos-test-modules -> 1157a46 (#29 — M6 retired from the registry)
Both in one commit because each is the other's precondition. 1157a46 removes M6
from the xfail registry, so pinning it against a daemon without the fix turns
that cell into a hard failure; pinning the fixed daemon without it turns the same
cell into an xpass, which also fails the run. Either alone is red — which is the
registry's forcing function working in both directions.
This is also the commit that makes the matrix describe the code it measures. The
lock previously resolved logos-protocol to a pre-#29 revision, so known.json
claimed fixes that the daemon under measurement did not contain.
verified with NO overrides, on merged revisions only:
conformance-matrix 158 pass / 6 xfail, differential 73
conformance-matrix-ext 40 pass, differential 19
integration local/tcp/tcp_ssl 68 passed each
unit 51 passed, 2 skipped
The 64-bit boundary cases added earlier in this branch are the ones that were
red before the protocol fix landed — echoUint(2^64-1) as -1 on the plain wire,
uintEvent(2^64-1) as 1.8446744073709552e+19 on every transport.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>