Files
Egor RachkovskiiandEgor Rachkovskii 3f8a0ed99a test: add Python e2e suite driving delivery_module over RPC (#51)
* test: add Python e2e suite driving delivery_module over RPC

Adds tests/e2e/: a pytest suite that boots logoscore daemons in docker, loads delivery_module, and drives it over RPC — exercising the plugin-load + codegen + RPC + two-node network-delivery paths the in-process C++ tests can't. Reuses logos-integration-test-framework / logoscore, pinned to chat-module's known-good SHAs.

Single-daemon: load/createNode/start/stop lifecycle, createNode-twice rejection, and query + subscribe/unsubscribe round-trips. Two-daemon (direct staticnodes peering, relay-only single shard): sender-side messagePropagated is the hard gate; messageReceived is xfail until the receive path and event-payload shape are confirmed on a green run.

CI: new e2e-tests job (needs build-and-test, ubuntu-latest) building .#install-portable plus the logoscore smoke image and CLI, with a cachix pre-warm step added to build-and-test.

* test(e2e): add RPC-reply-wedge diagnostic probes

Opt-in (E2E_PROBES=1) probe suite that isolates which factor wedges a
delivery node's RPC reply channel after a watched send: event emission
alone, the watcher subscription alone, messagePropagated transmission, or
the loopback messageReceived (vector<uint8_t>). Each probe runs on a fresh
two-node pair so a wedge can't pollute later probes. Enabled in the e2e CI
job; set E2E_QT_DEBUG=1 to also capture Qt RemoteObjects transport logs.

* update(README): clarify module packaging and add build reproducibility note

* Revert "update(README): clarify module packaging and add build reproducibility note"

This reverts commit 5aed6447a9.

* Revert "test(e2e): add RPC-reply-wedge diagnostic probes"

This reverts commit ef1f541358.

* test(e2e): mark propagation and messageReceived tests xfail due to known issues (#59)

* ci: make e2e-tests non-gating while build bug #58 is open

The module build is reproducible per source tree but some trees produce a
plugin whose start() never returns (#58), which fails the solo lifecycle
tests at start. xfail can't cover that (it hits setup), so allow the e2e
job to fail without blocking the PR until #58 is fixed.

* test(e2e),ci: drop narrative/decorative comments per repo style

Remove explanatory comment blocks, section-banner comments, and redundant
inline rationale across the e2e suite and the CI workflow; keep only linter
directives and a single footgun one-liner. No behaviour change.

* test(e2e): refactor node interaction to use LogosDelivery API

Refactor message delivery tests and helpers for clarity using the new `LogosDelivery` class. Replaced low-level RPC calls with higher-level methods (`subscribe`, `send`, `watch`, etc.), streamlining node interaction and removing redundant logic. Updated lifecycle tests for consistency. No behavior change.

* ci(e2e): capture gdb thread backtraces when start wedges (#58)

Non-gating diagnostic in the e2e-tests job: boot a solo logoscore daemon,
call start on a background thread, and if it wedges attach gdb to every
container process for thread backtraces, /proc maps and kernel stacks.
Uploaded with the e2e-test-logs artifact.

* ci(e2e): fix #58 wedge detection so gdb actually fires

The client RPC errors (exit 4) ~20s before the daemon-side start() releases
its 30s semaphore, so a populated result no longer means success. Classify any
non-success outcome as wedged, poll to detect it early, and attach gdb to every
container process in parallel to freeze them before the semaphore releases.

* debug(#58): log start callback path and capture daemon log past 30s

Add fprintf in callApiRetVoid at callback entry, after invoke, and on the
semaphore timeout to show whether liblogosdelivery ever invokes our completion
callback (upstream) vs a userData/map miss (adapter). Keep the wedge probe
daemon alive past the 30s CALLBACK_TIMEOUT so the start outcome is captured.

* debug(#58): add pre-invoke positive control and inspect loaded plugin

Log unconditionally right before invoke() so a bad build distinguishes
'start_node did not return' from 'the loaded plugin lacks the instrumentation'.
On a wedge the probe now strings/sha256 the loaded delivery_module_plugin.so
for the instrumentation literals and ldd's the bundled libs.

* debug(#58): force stderr unbuffered so start-path logs are reliable

Prior run showed the 'about to invoke' line (which provably executed — the node
started right after) missing from the captured log, making log-line absence
useless as evidence. setvbuf(_IONBF) at module init + fflush after each
instrumented fprintf so the next bad build's log is trustworthy.

* debug(#58): probe RPC channel after a start wedge

start() returns and no thread is stuck (kernel stacks), so the wedge is a lost
RPC reply, not a hang. On a wedge, issue follow-up version/getAvailableConfigs
calls: if they also time out the whole QtRO reply channel is dead (matches the
sync-reply-wedge bug); if they succeed only start's reply was lost.

* debug(#58): revert useless stderr instrumentation

The fprintf/setvbuf tracing added to localize #58 is dead weight — logoscore's
stderr capture drops adapter log lines during the node-startup flood, so the
log-based method never worked. Revert it (src/ is now net-zero vs master) and
drop the diagnostic's now-moot instrumentation-strings check. The gdb + kernel
stacks + RPC-channel probe in wedge_gdb.py are what actually localized the bug
and stay.

* ci(e2e): bump logoscore daemon+CLI past the QtRO reply-wedge fix (#58)

Root-caused #58/#59 to the pre-fix ModuleProxy in the e2e daemon's SDK: it
emits module events directly on the caller's (module FFI) thread, racing QtRO
source serialization against an in-flight reply and permanently wedging the
sync-reply channel. Fixed upstream in logos-cpp-sdk d77c3dd (marshal events
onto the source thread) + logos-protocol #7 (defer async completion off the
QtRO read stack). Our daemon was pinned 6 commits before it.

Bump the smoke-image daemon (logoscore-py aa45db52 -> 71e00381) and the CLI
(logoscore-cli 5a1cf746 -> a9e18455) to a post-split closure carrying both
fixes. The Python client already floats logoscore@master, so no requirements
change is needed.

* test(e2e): un-xfail two-node delivery tests; drop the #58 wedge diagnostic

The daemon-pin bump (901c525) fixed the QtRO reply-wedge behind #58/#59, so the
three two-node delivery tests now pass (propagation both directions + receive).
Un-xfail them and delete the temporary gdb wedge diagnostic + its CI step; revert
the artifact upload to failure-only. Keep continue-on-error on e2e-tests for now
to gather a flake baseline on the real cross-node delivery before it gates.

* test(e2e): rename lifecycle test for clarity

---------

Co-authored-by: Egor Rachkovskii <egorrachkovskii@status.im>
2026-07-28 10:21:21 +01:00

151 lines
5.4 KiB
Python

"""Fixtures for the delivery_module e2e suite.
Two daemon flavours:
* `solo_daemon` (function scope) — one fresh logoscore container per test, used
by the single-node lifecycle/query tests. Fresh because createNode is
once-per-context: a started node can't be re-created, so tests can't share one.
* `node_a` / `node_b` (module scope) — two daemons in a shared docker network,
peered directly via staticnodes (node A's container-routable multiaddr), used
by the two-node delivery tests.
Every prerequisite (docker, the logoscore image, the built module layout) is
gated by `_e2e_env_or_skip`, so the suite skips cleanly when run without setup.
"""
from __future__ import annotations
import os
import subprocess
import sys
import time
import uuid
from collections.abc import Callable, Iterator
from contextlib import ExitStack
from pathlib import Path
import pytest
from libs.constants import NETWORK_SUBNET
from libs.helpers import DeliveryNode, make_delivery_config, node_multiaddr, setup_delivery_node
MESH_STABILIZATION_S = 12.0
def _docker_logs(container_name: str) -> str:
r = subprocess.run(["docker", "logs", container_name], capture_output=True, text=True)
return (r.stdout or "") + (r.stderr or "")
def _save_logs(container_name: str) -> None:
log_dir = Path(os.environ.get("E2E_LOG_DIR", "/tmp"))
log_dir.mkdir(parents=True, exist_ok=True)
try:
(log_dir / f"{container_name}.log").write_text(_docker_logs(container_name))
except OSError as e:
sys.stderr.write(f"warning: failed to save log for {container_name}: {e}\n")
@pytest.fixture(scope="session")
def _e2e_env_or_skip() -> tuple[str, Path]:
"""Single gate: skip unless docker, the logoscore image, and the built
delivery_module layout are all present."""
from logoscore import docker_available, image_present # noqa: PLC0415
if not docker_available():
pytest.skip("docker not available")
image = os.environ.get("LOGOSCORE_IMAGE")
if not image:
pytest.skip("LOGOSCORE_IMAGE not set")
if not image_present(image):
pytest.skip(f"LOGOSCORE_IMAGE={image!r} not present locally")
modules_dir_env = os.environ.get("LOGOS_MODULES_DIR")
if not modules_dir_env:
pytest.skip("LOGOS_MODULES_DIR not set")
modules_dir = Path(modules_dir_env)
if not (modules_dir / "delivery_module" / "manifest.json").is_file():
pytest.skip(
f"LOGOS_MODULES_DIR={modules_dir} doesn't contain "
"delivery_module/manifest.json — did you run `nix build .#install-portable`?"
)
return image, modules_dir
@pytest.fixture(scope="session")
def shared_docker_network(_e2e_env_or_skip: tuple[str, Path]) -> Iterator[str]:
name = f"logosdelivery-e2e-{uuid.uuid4().hex[:8]}"
r = subprocess.run(
["docker", "network", "create", "--subnet", NETWORK_SUBNET, name],
capture_output=True, text=True,
)
if r.returncode != 0:
pytest.fail(f"failed to create docker network {name!r}: {r.stderr.strip()}")
try:
yield name
finally:
subprocess.run(["docker", "network", "rm", name], capture_output=True)
def _new_daemon(image: str, modules_dir: Path, network: str, label: str):
from logoscore import LogoscoreDockerDaemon # noqa: PLC0415
container_name = f"logoscore-{label.lower()}-{uuid.uuid4().hex[:8]}"
daemon = LogoscoreDockerDaemon(
image=image,
modules_dir=modules_dir,
container_name=container_name,
network=network,
startup_timeout=60.0,
extra_args=["--verbose"],
)
return daemon, container_name
@pytest.fixture
def solo_daemon(_e2e_env_or_skip: tuple[str, Path], shared_docker_network: str) -> Iterator:
"""A fresh, started logoscore container — module mounted, nothing loaded yet.
The test drives load_module/createNode/start/stop itself."""
image, modules_dir = _e2e_env_or_skip
daemon, container_name = _new_daemon(image, modules_dir, shared_docker_network, "solo")
try:
daemon.start()
yield daemon
finally:
_save_logs(container_name)
daemon.stop()
@pytest.fixture(scope="module")
def delivery_node_factory(
_e2e_env_or_skip: tuple[str, Path],
shared_docker_network: str,
) -> Iterator[Callable[[str, str], DeliveryNode]]:
"""Factory spawning a fully set-up (load+createNode+start) DeliveryNode per
call. All teardown + log capture is registered on one ExitStack closed when
the module's tests finish."""
image, modules_dir = _e2e_env_or_skip
with ExitStack() as stack:
def _create(label: str, config_json: str) -> DeliveryNode:
daemon, container_name = _new_daemon(image, modules_dir, shared_docker_network, label)
stack.enter_context(daemon)
stack.callback(_save_logs, container_name)
return setup_delivery_node(daemon, config_json, label)
yield _create
@pytest.fixture(scope="module")
def node_a(delivery_node_factory: Callable[[str, str], DeliveryNode]) -> DeliveryNode:
return delivery_node_factory("A", make_delivery_config())
@pytest.fixture(scope="module")
def node_b(
delivery_node_factory: Callable[[str, str], DeliveryNode],
node_a: DeliveryNode,
) -> DeliveryNode:
peer = node_multiaddr(node_a)
node = delivery_node_factory("B", make_delivery_config(static_peers=[peer]))
time.sleep(MESH_STABILIZATION_S)
return node