Files
Dario LipicarandClaude Opus 5 d2475cfc1e feat(conformance): the py driver for the LIDL conformance matrix (#11)
* feat(conformance): the py driver for the LIDL type matrix

The driver half of the conformance matrix; the case table and the xfail
registry live in logos-test-modules/conformance/, with the providers they
describe. The driver lives here because it uses this package's client — and
logoscore-py already depends on logos-test-modules, so the reverse would be a
cycle.

`conformance/run_matrix.py` replays every case against BOTH providers and
reports one JSONL line per cell, keyed by coordinate. `checks.conformance-matrix`
runs it; it fails on a red cell, on an `xpass` (a registered known-broken cell
that started passing — the registry has to be updated), or on a (type, position)
the contract declares that no case covers.

Three things it took a run to get right, all recorded in the code:

  * `same()` is type-strict — a matrix that compares with `==` cannot see an
    integer degrading to a float, which is most of what this exists to catch,
    and `1 == True` would pass too.
  * a case may set `"raw": true` to opt out of tagged-bytes materialization.
    Without it the adversarial `_bytes`-collision cases are inexpressible: the
    driver would convert BOTH the argument and the expectation to bytes and the
    cell would compare bytes to bytes and pass no matter what the system did —
    the exact fake-green the matrix is meant to remove.
  * a dispatch rejection arrives as the RESULT ({"code": "dispatch_failed"}),
    not as a raised error, so a driver that only watches for exceptions records
    a rejection as a successful call returning a dict.

Each provider gets one daemon PER PHASE. Sharing a daemon between the ~80-call
method phase and the event phase wedged it partway through the events under the
nix sandbox — the last five failed contiguously with RPC_FAILED while every one
of them passes on a fresh daemon. A phase that can poison the next one makes a
red cell mean "something earlier used up a resource", which is the kind of
unreliable signal this exercise exists to remove.

Verified end to end: green (139 pass / 13 registered xfail / 68 differential),
and RED on 19 cells when pointed at the pre-fix logoscore CLI — so it demonstrably
catches the class of bug it is for, rather than merely claiming to.

Also adds a unit test asserting the inline `_fullapi_module_cases.py` table and
the shared `cases.json` agree. It does not merge them — rewriting a passing
integration suite to prove a point is a bad trade — but it makes the drift
between two hand-maintained copies a red test instead of a silent surprise.
(Confirmed it fails on an injected divergence, not just that it passes.)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* feat(conformance): N-ary events and positional cells in the driver

Follows the arity surface added to the contract. Three changes:

  * an event case may carry `values` (a list) instead of `value`; the driver
    fires with *values and compares the ORDERED argument list, so slot order is
    part of the assertion rather than something the payload shape hides;
  * a multi-parameter event arrives as {arg0, arg1, ...} — the driver rebuilds
    the ordered list, and a single-argument event still reduces to its one
    value, so no existing case changes;
  * coverage now distinguishes `method_arg` from `method_arg@k` (and the event
    equivalent). A sole argument cannot catch a generator that mixes up
    positional slots, so they are genuinely different cells, and a case that
    covers several declares the (type, position) PAIRS explicitly instead of
    the cross-product of its type and position lists — the cross-product would
    claim cells the case never exercises.

79 cases x 2 providers, green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* feat(conformance): data-driven provider set + the ext matrix check

`--modules NAME=DIR` (repeatable) generalises the two hardcoded provider flags,
so a table with a different provider set runs through the SAME driver instead of
a second one. The full_api_ext table has one provider — the C++ cdylib backend
cannot express records or [bstr] yet — so the differential simply has nothing to
compare there; it is not silently skipped, there is just no pair.

The table names the providers it describes and the driver runs the intersection,
failing loudly if a declared provider has no module dir rather than quietly
reporting a smaller matrix.

Adds `checks.conformance-matrix-ext`: 20 cases, full contract coverage, green
with 9 registered xfails (E1 bytes-at-depth mangling, E2 empty-bytes-at-depth
dropped to null, E3 M1 through a record field).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* feat(conformance): per-case provider precision + the C++ ext provider

The ext check now runs both ext providers, so that table has a differential like
full_api does.

`known.json` entries may carry `per_case_providers`, narrowing an individual
case to the providers that actually fail it. `providers` alone is an entry-wide
union, and a defect that only one provider surfaces then gets registered against
both — so the provider that PASSES is reported as `xpass` and the registry
manufactures a failure it then demands you fix. 4 of E1's 9 cases are
single-provider, which is how this surfaced.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(conformance): state what this driver does not cover

The module docstring claimed "Other consumers (the C++, Rust and QML proxies)
replay the same cases.json". No such driver exists — `--consumer` is a label
written into the report, not a driver selector. Grepping the workspace finds
only this one.

That overclaim mattered. The event bridge failed to decode canonical
{"_bytes": ...} into a QByteArray, and this driver could not see it: the
undecoded map round-trips to JSON and the python client decodes the tag itself.
The cells stayed green while a Qt/C++ or QML subscriber got a map. A matrix with
one consumer cannot see a defect that its own consumer happens to undo, and a
docstring promising four consumers hides that.

Also states the transport limit: the daemon is built with no `transports=`, so
every cell is measured over LocalSocket/QtRO and the plain wire's separate uint64
defect is out of reach.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(fullapi): 64-bit boundaries, and re-pin onto the merged chain

The integration matrix — replayed by the local, tcp and tcp_ssl checks and by
the docker codec matrix — topped out at 2^53-1 for `int` and 2^32-1 for `uint`.
So the 64-bit band that the whole LIDL type contract is about was untested on
every transport, and it was broken on two of them.

Against the pinned protocol over tcp the new cases fail exactly as the code
predicts:

    echoUint(2^63)        -> -9223372036854775808   (RpcValue has no unsigned alt)
    echoUint(2^64-1)      -> -1                     (same)
    uintEvent(2^64-1)     -> 1.8446744073709552e+19 (the event bridge, M6)

Two different defects, on the same transport, that no existing case could see.
Both are fixed in logos-protocol; all 68 pass on local, tcp and tcp_ssl.

2^53+1 is in the table on purpose: it is the smallest integer a double cannot
represent, so it separates "degraded through a float" from "wrapped as an
integer" — the two failure modes look alike in a report and have different
causes.

Also re-pins onto the merged chain now that logos-test-modules#28 landed:

    logos-test-modules  -> d4c0d04  (the conformance matrix)
    logos-logoscore-cli -> a143727  (64-bit call args)

That re-pin matters beyond housekeeping: the lock previously resolved
logos-protocol to a pre-#29 revision, so the matrix's own registry claimed fixes
that the code under measurement did not contain.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* chore: re-pin onto the merged uint64 chain

  logos-logoscore-cli -> 94f54b3  (#75 — carries logos-protocol 8b8a358)
  logos-test-modules  -> 1157a46  (#29 — M6 retired from the registry)

Both in one commit because each is the other's precondition. 1157a46 removes M6
from the xfail registry, so pinning it against a daemon without the fix turns
that cell into a hard failure; pinning the fixed daemon without it turns the same
cell into an xpass, which also fails the run. Either alone is red — which is the
registry's forcing function working in both directions.

This is also the commit that makes the matrix describe the code it measures. The
lock previously resolved logos-protocol to a pre-#29 revision, so known.json
claimed fixes that the daemon under measurement did not contain.

verified with NO overrides, on merged revisions only:

  conformance-matrix       158 pass / 6 xfail, differential 73
  conformance-matrix-ext   40 pass, differential 19
  integration local/tcp/tcp_ssl   68 passed each
  unit                     51 passed, 2 skipped

The 64-bit boundary cases added earlier in this branch are the ones that were
red before the protocol fix landed — echoUint(2^64-1) as -1 on the plain wire,
uintEvent(2^64-1) as 1.8446744073709552e+19 on every transport.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 09:25:16 -03:00

279 lines
10 KiB
Python

"""Integration tests exercising the **full supported type surface** of a
universal Logos module — parameters, return values, and events — against
`test_fullapi_cpp`.
`test_fullapi_cpp` is the C++ provider of the shared `full_api` contract
(`logos-test-modules/test-fullapi-module-cpp`). It declares one echo method
**and one typed event per event-legal type**, so this file is the end-to-end
proof that every type the generator + protocol support round-trips through
the logoscore client — both as a call argument/return and as an event
payload.
Type surface covered (each as param, return, and — where legal — event):
LIDL Python arg Python return / event payload
------------ ------------------ ------------------------------
tstr str str
bstr bytes bytes (canonical {"_bytes"} tag)
int int int
uint int int
float64 float float
bool bool bool
any str/int/float/ same Python value
bool/list/dict
[tstr] list[str] list[str]
[int]/[uint] list[int] list[int]
[float64] list[float] list[float]
[bool] list[bool] list[bool]
[any] list list (heterogeneous)
{tstr:any} dict dict
result — {"success","value","error"}
void — True (CLI success sentinel)
Container args + non-scalar `any` reach the daemon via the CLI's `json:`
prefix, and `bytes` via the canonical `{"_bytes": "<b64url>"}` tag — both
handled transparently by `LogoscoreClient` (see `_arg_to_str`). Byte-array
**event** payloads decode back to `bytes` in the event pump, symmetric
with `call`.
See:
repos/logos-test-modules/test-fullapi-module-cpp/src/test_fullapi_cpp_impl.h
Skipped unless LOGOSCORE_BIN and LOGOSCORE_TEST_MODULES_DIR are set — the
Nix `integration` check wires both up (and bundles this module into the
test modules dir).
"""
from __future__ import annotations
import threading
import time
import pytest
from logoscore import LogoscoreDaemon
from .._fullapi_module_cases import FULLAPI_EVENT_CASES
MODULE = "test_fullapi_cpp"
@pytest.fixture(scope="module")
def client(logoscore_bin, test_modules_dir, transport, request):
"""Build a daemon + client wired to whatever transport the suite is
parametrised on. Kept inline (rather than moved to a conftest helper)
so each test file can be read end-to-end without jumping between
files — mirrors the fixture in `test_end_to_end.py`."""
import socket
def _pick_free_port() -> int:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.bind(("127.0.0.1", 0))
return s.getsockname()[1]
kwargs = {}
client_kwargs: dict = {"transport": transport}
if transport != "local":
kwargs["transports"] = [transport]
if transport == "tcp":
kwargs["tcp_port"] = _pick_free_port()
elif transport == "tcp_ssl":
cert, key = request.getfixturevalue("self_signed_cert")
kwargs["tcp_ssl_port"] = _pick_free_port()
kwargs["ssl_cert"] = cert
kwargs["ssl_key"] = key
client_kwargs["no_verify_peer"] = True
with LogoscoreDaemon(
modules_dir=test_modules_dir, binary=logoscore_bin, **kwargs,
) as daemon:
c = daemon.client(**client_kwargs)
c.load_module(MODULE)
yield c
# ── Scalar params / returns ──────────────────────────────────────────────────
def test_who_am_i(client):
assert client.call(MODULE, "whoAmI") == "test_fullapi_cpp"
def test_echo_string(client):
assert client.call(MODULE, "echoString", "round-trip") == "round-trip"
# 64-bit boundaries belong here specifically. This module is replayed by the
# local, tcp and tcp_ssl checks, and the highest value it used to carry was
# 2^53-1 (int) / 2^32-1 (uint) — so the plain wire's 64-bit handling was never
# exercised at all. It was broken: RpcValue had no unsigned alternative, and a
# uint above int64max arrived as -1.
# 2^53+1 is the smallest integer a double cannot hold, which separates
# "degraded through a float" from "wrapped as an integer".
@pytest.mark.parametrize(
"n", [0, 42, -7, 9007199254740991, 2**53 + 1, 2**63 - 1, -(2**63)]
)
def test_echo_int(client, n):
assert client.call(MODULE, "echoInt", n) == n
@pytest.mark.parametrize("n", [0, 7, 4294967295, 2**53 + 1, 2**63, 2**64 - 1])
def test_echo_uint(client, n):
assert client.call(MODULE, "echoUint", n) == n
@pytest.mark.parametrize("x", [0.0, 2.5, -3.25, 1e-6])
def test_echo_double(client, x):
assert client.call(MODULE, "echoDouble", x) == pytest.approx(x)
@pytest.mark.parametrize("b", [True, False])
def test_echo_bool(client, b):
assert client.call(MODULE, "echoBool", b) is b
@pytest.mark.parametrize(
"payload",
[
b"", # empty
b"hello", # ascii
b"\x01\x02\x03", # low bytes
b"\x00\x10\xff\x80", # NUL + high bytes — canonical tag is lossless
bytes(range(256)), # full byte range
],
ids=["empty", "ascii", "low", "nul-high", "full-range"],
)
def test_echo_bytes(client, payload):
# bytes cross the wire as the canonical {"_bytes": "<b64url>"} tag, so
# every byte value (incl. NUL and >= 0x80) round-trips losslessly — a
# raw latin-1 arg would UTF-8-mangle the high bytes.
assert client.call(MODULE, "echoBytes", payload) == payload
@pytest.mark.parametrize(
"value",
["hello", 42, 3.5, True, [1, 2, 3], {"k": "v", "n": 1}],
ids=["str", "int", "float", "bool", "list", "map"],
)
def test_echo_any(client, value):
got = client.call(MODULE, "echoAny", value)
if isinstance(value, float):
assert got == pytest.approx(value)
else:
assert got == value
# ── Container params / returns ───────────────────────────────────────────────
def test_echo_string_list(client):
assert client.call(MODULE, "echoStringList", ["a", "b", "c"]) == ["a", "b", "c"]
@pytest.mark.parametrize("xs", [[], [1, 2, 3], [-1, 0, 5]])
def test_echo_int_list(client, xs):
assert client.call(MODULE, "echoIntList", xs) == xs
def test_echo_uint_list(client):
assert client.call(MODULE, "echoUintList", [4, 5, 6]) == [4, 5, 6]
def test_echo_double_list(client):
assert client.call(MODULE, "echoDoubleList", [1.5, 2.5, -3.0]) == pytest.approx(
[1.5, 2.5, -3.0]
)
def test_echo_bool_list(client):
assert client.call(MODULE, "echoBoolList", [True, False, True]) == [True, False, True]
def test_echo_list_heterogeneous(client):
# [any] — a LogosList carrying mixed element types (incl. a nested map).
value = [1, "two", 3.5, True, {"k": 1}, [9, 8]]
got = client.call(MODULE, "echoList", value)
assert got == value
def test_echo_map(client):
value = {"s": "v", "n": 42, "f": 1.5, "b": True, "nested": {"x": [1, 2]}}
assert client.call(MODULE, "echoMap", value) == value
# ── result / void returns ────────────────────────────────────────────────────
def test_make_result_success(client):
assert client.call(MODULE, "makeResult", True) == {
"success": True,
"value": {"ok": True, "provider": "test_fullapi_cpp"},
"error": None,
}
def test_make_result_error(client):
assert client.call(MODULE, "makeResult", False) == {
"success": False,
"value": None,
"error": "deliberate error for testing",
}
def test_do_void(client):
# void methods have no value; the CLI reports `true` as the success
# sentinel.
assert client.call(MODULE, "doVoid") is True
# ── Events: one typed event per event-legal type ─────────────────────────────
# Each event is fired by its bool-returning `fire<Name>Event(v)` shim (a
# bare void trigger would make the CLI exit non-zero). The provider emits
# the corresponding event with the argument as its single `arg0` payload.
def _capture_event(
client, event: str, fire_method: str, value, overall_timeout: float = 20.0
) -> dict:
"""Subscribe, then fire the trigger and wait — re-firing until the event
arrives or `overall_timeout` elapses.
The watcher subscribes on a background subprocess, so there is an
unavoidable race between "watch is live" and "we fire". A single fixed
sleep can't cover a slow-to-subscribe watcher on CI: if the first (and
only) fire lands before the subscription is live, the event is missed
and no later wait can recover it. The `fire<X>Event` triggers are
idempotent emits, so re-firing on a short cadence closes the race
without hard-coding a settle duration."""
received: list[dict] = []
got = threading.Event()
def on_event(e: dict) -> None:
received.append(e)
got.set()
with client.on_event(MODULE, event, on_event):
deadline = time.monotonic() + overall_timeout
while True:
assert client.call(MODULE, fire_method, value) is True
if got.wait(timeout=1.0):
break
assert time.monotonic() < deadline, (
f"{event} not received within {overall_timeout}s"
)
return received[0]
# The typed-event matrix is shared with the docker-smoke suite so the two
# stay in lockstep — see tests/_fullapi_module_cases.py::FULLAPI_EVENT_CASES.
@pytest.mark.parametrize(
"event,fire,value", FULLAPI_EVENT_CASES,
ids=[c[0] for c in FULLAPI_EVENT_CASES],
)
def test_typed_event(client, event, fire, value):
evt = _capture_event(client, event, fire, value)
assert evt["event"] == event
assert evt["module"] == MODULE
payload = evt["data"]["arg0"]
# bytes decode back to `bytes` in the pump; floats compare approximately.
if isinstance(value, float) or (
isinstance(value, list) and value and isinstance(value[0], float)
):
assert payload == pytest.approx(value)
else:
assert payload == value