logos-co/logos-protocol
1
0
Fork 0
mirror of https://github.com/logos-co/logos-protocol.git synced 2026-08-31 05:51:08 +00:00
Code Issues Packages Projects Releases Wiki Activity
40 Commits 72 Branches 0 Tags
378d889a1da3581cae8732e4bc661796bdb3a09c
Commit Graph
7 Commits
Author SHA1 Message Date
Dario Gabriel Lipicar 378d889a1d fix(protocol): a call that FINISHED must not park its thread for the object's life 
The per-call waiters are joinable rather than detached, which is what closed the
use-after-free where release() deleted the object under a still-running waiter
(4f9d824), and they are interruptible, so teardown no longer waits out the call's
timeout (731e579). Both stay. What neither did was retire a waiter that had
FINISHED: m_waiters was only ever swap()ped, in stopAndJoinWaiters(), so an
exited-but-unjoined std::thread — whose stack and pthread struct are not
reclaimed until somebody joins it — stayed parked for the lifetime of the handle.

Measured against a live PlainTransportHost over TCP, every call completing
normally, one handle held throughout, before:

    10000 calls   m_waiters   300 -> 10300   rss +156.56 MiB   16417 B/call
    30000 calls   m_waiters   300 -> 30300   rss +469.28 MiB   16403 B/call

and the same through the production C ABI — one lp_client, N lp_invoke_async —
at +156.53 MiB. That path is why this matters: LogosAPIConsumer caches ONE
handle per module and reuses it for every async call, releasing it only on
eviction or teardown (cpp/logos_api_consumer.cpp:129 and :207), so a
long-lived module leaks per lp_invoke_async. The ~16KB constant is one page on
this 16KiB-page arm64 and will be smaller elsewhere; the UNBOUNDEDNESS is the
platform-independent part, and follows from m_waiters.size() rising 1:1 with
completed calls and only ever falling in teardown. Attribution: the retention
arrived with the join in 4f9d824, not with 731e579 — but 731e579 is what makes
the join permanent.

The registry is now KEYED, because a thread cannot join itself and so a waiter
can never retire its own entry. Each waiter publishes its id as its FINAL act (a
scope guard declared first, so it destructs last, covering all four exit paths),
and the next spawn — plus teardown — joins those ids and erases them. Joining a
thread that has already returned is a couple of syscalls. Same probe, same
workload, after:

    10000 calls   m_waiters    15 -> 16      rss +0.08 MiB         8 B/call
    30000 calls   m_waiters    16 -> 16      rss +0.06 MiB         2 B/call
    10000 calls via lp_invoke_async          rss +0.09 MiB        10 B/call

Retention is now bounded by the waiters that finish after the LAST spawn, i.e.
by peak in-flight concurrency — 16 at the in-flight window above, and exactly 1
when calls are issued sequentially — instead of by call count.

THE DEADLOCK THIS SHAPE INVITES is a reaper that joins while holding m_waiterMu,
against a waiter blocked on m_waiterMu trying to publish. It is avoided by
construction rather than by argument: nothing is joined with a lock held, in the
reaper or in teardown, whatever a waiter does on its way out. Proven by building
the naive variant that does join under the lock — the new hammer wedges it, with
the main thread in reapFinishedWaiters -> pthread_join and a waiter in
publishFinishedWaiter -> mutex wait, and the test's watchdog names the cause
instead of letting CI hang.

Teardown's guarantee is restated rather than weakened. It is not "every waiter
has been joined by the time stopAndJoinWaiters() returns" — a waiter a
concurrent reaper is mid-join on is no longer in the map — but the thing that
guarantee was ever for: NO WAITER TOUCHES THE OBJECT AFTER IT RETURNS. An entry
leaves m_waiters only once its thread has published, and publishing is that
thread's last access.

The TODO above the waiter still stands: the real fix is to fold the wait into
the shared Asio io_context and have no thread per pending RPC at all. This makes
the interim honest; it does not replace that.

Two more things review turned up, folded in here:

  * The two wait sites resolved stop-vs-result in OPPOSITE directions.
    waitForResult tested the stop flag BEFORE polling, so an already-ready
    future was still reported as transport_error, while awaitCompletion
    deliberately preferred a completion that had landed — and both were
    commented as intentional. One rule now, applied to both: AN ANSWER ALREADY
    IN HAND BEATS A CONCURRENT STOP, and the stop only decides what happens when
    there is nothing to hand over. The callback fires either way
    (postToQtEventLoop copies everything it delivers), so the only thing a stop
    can change is what the callback SAYS — and manufacturing transport_error
    while the true answer sits in the future reports a failure that did not
    happen, to callers that re-acquire, retry and log on that code. Preferring
    the answer costs nothing, since it is already there: the flag is still
    checked before every sleep, so the teardown-latency bound is unchanged.

  * CORRECTION to 731e579's message, which claimed it "closes the registration
    window" where a call arriving after the stop would never be joined. That
    branch is unreachable in defined behaviour: m_stopping is raised only by
    teardown, so any thread that can read it inside callMethodAsyncWithError is
    already calling a method on an object whose destructor is running — the load
    is itself the use-after-free, reproduced as a SIGSEGV on that commit and on
    its parent alike, and nothing inside that function can repair it. The guard
    is harmless and stays (one predictable branch, and it fails safe with one
    callback), but its comment now says what it is instead of claiming a fix it
    does not make.

Verified by running, with every check first shown to FAIL on unfixed code:

  * Retention: the probe above, plus a committed regression test that reads
    m_waiters out of the live object through the explicit-instantiation access
    hole ([temp.spec] does not check access on an explicit instantiation's
    template arguments) — so the code under test keeps its private state, with
    no friend, no test-only accessor and no `#define private public`. 200
    sequential completed calls keep 1 waiter; without pruning they keep 200.
  * Exactly-once on all four paths — normal completion, timeout, cancellation
    and the deferred-completion (pending-sentinel) arm — counted PER CALL so a
    dropped one and a doubled one cannot cancel out, plus the 60-round
    release-during-call race. Shown to catch a cancelled path that returns
    silently (3 failures) rather than delivering.
  * Teardown latency unchanged from 731e579: 10-17ms with an in-flight 8000ms
    call and 0-1ms mid-defer, against 15ms / 1ms on that commit.
  * The UAF stays closed: 11 teardown + reaping tests clean under macOS Guard
    Malloc (ASan/TSan remain unusable on this toolchain).
  * Full suite 281/281 twice, `nix build .#tests` green (281/281 in the
    sandbox), CallErrorAfterAcquireTest hammered 40x clean.
 2026-08-05 20:24:11 -03:00
Dario Gabriel LipicarandClaude Opus 5 731e579064 fix(protocol): teardown must not wait out the call it is abandoning 
Joining the per-call waiters (rather than detaching them) closed a real
use-after-free: the waiter captures `this`, and release() used to `delete this`
underneath it. But joinWaiters() could only join. It had no way to ASK a waiter
to stop, so destroying a PlainLogosObject with a call in flight blocked for the
remainder of that call's timeout — up to 20s on the protocol default. A module
unloading mid-call stalled the unload for that long, on the releasing thread.

Measured, 8s call timeout, provider parked:

    release()                     before        after
    future wait  (site 1)         7804 ms       11 ms
    deferred completion (site 2)  7703 ms        0 ms

Both blocking sites are now interruptible, and they need different treatment:

  * the std::future wait cannot be interrupted at all, so it is SLICED: one
    deadline computed up front, waited in 25ms increments, stop flag checked
    between them. Teardown latency is one slice; the timeout the caller asked
    for is unchanged, because the last slice ends exactly on the deadline. 25ms
    is under two frames (so a module unload stays imperceptible) and costs 40
    wakeups/second per in-flight call, which is nothing beside the Qt event loop
    these threads already sit next to.

  * awaitCompletion's condition_variable is interruptible by construction:
    widen the predicate, notify_all. No latency floor at all — hence 0 ms. The
    flag is published under m_completionMu so a waiter cannot evaluate the
    predicate, decide to sleep, and then miss the notify.

A CANCELLED CALL STILL DELIVERS, EXACTLY ONCE. This is the part a naive fix
breaks: callMethodAsyncWithError and lp_invoke_async promise the callback fires
exactly once, so a waiter that simply returns on stop trades a bounded stall for
an unbounded hang in every caller awaiting it. Proven by building that naive
variant: it passes the latency test and fails three exactly-once tests with the
callback never arriving.

The code is "transport_error", from the existing vocabulary rather than a new
one, since these codes are the wire contract. logos_call_error.h defines it as
"the connection failed or was torn down mid-call", which is precisely what
happened — the consumer tore its own end down. The alternatives all misattribute
it: "object_unavailable" says the module is absent (it is not, and callers
re-acquire on that code), "call_failed" blames the peer for a dispatch it
performed fine, and "timeout" — what this used to report, after waiting the
deadline out — claims a deadline elapsed that did not. It is also already what
the wire produces for the same event seen from the other side (callErrorFromWire
maps TRANSPORT_CLOSED to transport_error).

Delivering during teardown is safe because postToQtEventLoop touches nothing
owned by the object: it is a free function taking the callback, value and error
BY VALUE, and the waiter copies objectName/method up front. That was already
true and is now load-bearing, so it is documented at the function. The queued
lambda runs after the object may be gone; everything the waiter reaches through
`this` runs before the join returns, which is why the join must stay.

Also closes the registration window it opens: a call arriving after the stop
would push a thread onto an m_waiters that teardown has already swapped out, so
it would never be joined. It is answered as cancelled instead.

The UAF is verified still closed under macOS Guard Malloc rather than ASan —
libclang_rt livelocks in its own initializer before main on this toolchain, for
both ASan and TSan, on a hello-world. Under Guard Malloc the race test is clean
across 5 runs and SIGSEGVs immediately when the join is turned back into a
detach, so the check is a real detector and not a vacuous pass.

Tests: 277/277 (was 270; 7 new). CallErrorAfterAcquireTest hammered 40x, 0
failures — it was ~2/50 flaky before this branch's earlier fixes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-05 15:12:58 -03:00
Dario Gabriel LipicarandCursor 1e9c93434b fix(protocol): stop the macOS flake that was sinking #41 
Three real races the new CallErrorAfterAcquire suite exposed (and that
Copilot flagged on the QtRO half):

1. ~PlainTransportHost stopped the acceptor but did not quiesce the shared
   Asio I/O thread. Server-side RpcConnections hold a raw IncomingCallHandler*
   back to the host; a fail()/onConnectionClosed racing teardown freed the
   handler mid-call. That is the macOS CI SIGSEGV in
   AsyncSuccessStillReportsTheValue — it fires with no output of its own
   because the previous live-host test's destructor left the heap corrupted.
   Restore the I/O barrier that landed on the qtfree branches but never on
   master (proven: 80/80 clean on the CI crash sequence that was ~2/50 before).

2. PlainLogosObject::callMethodAsync detached its per-call waiter while
   capturing `this`. release()/delete this could then race the waiter.
   Join waiters in the destructor/release, and register the thread under the
   lock before it can outrun teardown.

3. QtRO async could deliver the user callback twice when the timeout timer
   and the pending-call watcher finished around the same moment, violating
   the exactly-once contract. Gate both paths (and the deferred-completion
   arm) on one atomic.

Also drain queued onCall invokes after host.reset() in the #40 live-target
control, matching LiveHost's teardown discipline.

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-08-04 18:19:29 -03:00
Dario Gabriel LipicarandClaude Opus 5 ef7539efbe fix(protocol): report the failures that happen AFTER acquire, on both twins 
#40 made lp_invoke_async able to report a failure, but only for the two
conditions produced ABOVE the transport: acquire failure and the unauthorized
sentinel. Everything the transport learns while the call is in flight was still
discarded — PlainLogosObject answered a bare QVariant() for a timeout and for
`ResultMessage.ok == false` alike, and LogosAPIConsumer hard-coded an empty
CallError next to it.

Two ordinary failures therefore still reported success on both entry points:
a TIMEOUT, and MODULE NOT LOADED against a host that is up (which is not an
acquire failure on the plain wire — requestObject hands back a handle for any
name over an open connection).

The information already exists: ResultMessage carries err/errCode, the futures
know they expired, QtRO knows its pending call never finished. It had nowhere to
go because LogosObject's callMethod returns a lone QVariant and its
callMethodAsync callback takes a lone QVariant.

Widening those virtuals would append a vtable slot to an installed, subclassed
interface, so instead this adds LogosObjectErrorChannel — a SIBLING interface
reached by dynamic_cast. LogosObject's size, layout and vtable are unchanged
(verified: a subclass compiled against the old and new headers emits the same
14-entry vtable with identical slot indices), and a transport that does not
implement it keeps today's behaviour.

logos_protocol.cpp needs no change: lp_invoke and lp_invoke_async already render
this CallError, so both twins gain the coverage together.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-03 16:29:37 -03:00
Dario LipicarandClaude Opus 5 c0df466172 fix: integer signedness in the codec, and a shape check on the pending-call sentinel (#31) 
* fix(codec): signedness and range are part of the integer type

Codec<T>::from accepted any integral JSON number and handed it to .get<T>().
That is silent in both directions:

  .get<uint64_t>() on -1   -> 18446744073709551615   (a sign flip)
  .get<int32_t>()  on 2^40 -> truncated

Both now reject with the usual path-carrying CodecError instead. Rejecting is the
codec's existing contract — a value the declared type cannot represent must not
reach business logic wearing a different one — this just extends it to the half
of the integer domain it was skipping.

Note the check is on the JSON category, not the value: a negative literal parses
as number_integer and never as number_unsigned, so `is_number_unsigned()` is the
reliable discriminator rather than a comparison after conversion.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(async): the pending-call sentinel is matched by shape, not by key presence

All four detection sites tested `m.contains(pendingCallKey())` and nothing else,
so ANY user map carrying that key was taken for a deferred call: the consumer
extracted a call id, found no completion, and waited out a nested event loop.
The measured outcome is a ~20s HANG, not a fast failure. An `any` slot is enough
to reach it — anything a user can put in a map.

logos::isPendingCallSentinel now requires the canonical shape: exactly one entry,
under the sentinel key, holding a non-empty string. Shape and signature are
mirrored from isUnauthorizedSentinel (logos_rpc_status.h), QJsonObject arm
included — the two are the same kind of in-band marker and there was no reason
for them to be guarded differently. That guard, and isTaggedBytes's, both already
existed in this repo; the difference was chronology, not principle.

Behaviour-preserving: the generated glue builds this map with exactly one entry
whose value is a QString call id, so no real sender changes. The concurrent
dispatch tests pass unchanged.

NARROWS, DOES NOT CLOSE — and the tests say so out loud. A one-key, string-valued
forgery IS the sentinel; no predicate can separate them. It still hangs, and
because call ids are a per-object counter from 0, a forged "lc-0" can collide
with a genuine in-flight completion and steal its result. Closing that needs an
out-of-band channel for "deferred", which the single-QVariant dispatch slot
cannot express without an ABI break — the constraint is stated at
logos_rpc_status.h:24-27 and is real.

tests: 10 new, including one asserting the forgery still matches, so a future
reader cannot mistake the green cells for "the sentinel is safe". 236/236.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
 2026-07-29 11:50:18 -03:00
Dario LipicarandClaude Opus 4.8 4ea32a314a Per-module concurrent dispatch: async provider seam + transports (#5) 
* feat: per-module concurrent dispatch (concurrency:"multi") — zero ABI change

A "multi" module serves calls concurrently behind the ORDINARY callMethod — no
new provider/host vtable method, so LogosProviderObject's ABI is byte-identical
to before and an old host/daemon loads and forwards a multi module unmodified.

Mechanism: a multi module's generated glue returns a pending sentinel
({"__logos_pending_call__": callId}) from callMethod and pushes the real result
back later as a __logos_call_complete__ event keyed by callId, over the existing
event channel. The consumer transport detects the sentinel and awaits the
completion transparently, so generated clients are unchanged.

- logos_async_dispatch.h: shared wire constants + the contract.
- remote_transport.cpp (QtRO) / plain_logos_object.{h,cpp} (plain): consumer
  sentinel detection + await keyed by callId. The host is a pure forwarder.
- logos_protocol.h + nix/default.nix: protocol 0.2.0 (additive minor; same MAJOR
  stays compatible, so an old host accepts a 0.2 "multi" module).
- rpc_server.cpp: fix a teardown self-deadlock (stop() held m_mu while invoking a
  per-connection error handler that re-locks m_mu) that the new in-process
  subscription path exposed.
- tests/protocol/test_concurrent_dispatch.cpp: proves a multi provider overlaps
  two concurrent calls (peak 2) while single serializes (peak 1), over the plain
  transport, with the host unchanged from master.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: coalesce concurrent async requestModule handshakes (+ async fan-out test)

A driver that fans out N async calls to an un-tokened target before any
completes used to fire N separate requestModule handshakes. Each mints a
distinct capability token and informs the target, and the later inform
OVERWRITES the earlier token there (the target stores one token per caller),
so the already-dispatched calls carried a superseded token and the target
rejected them as unauthorized ("auth token not recognized"). The sync path
never hit this — it blocks per call, so handshakes never overlap.

Coalesce in LogosAPIClient::invokeRemoteMethodAsync: the first async call to
an un-tokened target starts ONE handshake; concurrent calls to the same
target queue behind it and all drain with the single minted token when it
resolves. m_pendingHandshakes is touched only on the owner thread, so no lock
(appended last per the class's ABI note). This is what lets a concurrency:
"multi" worker actually run a single-threaded driver's fan-out concurrently —
otherwise the fanned-out calls are rejected before reaching dispatch.

Also add MultiProviderOverlapsAsync / SingleProviderSerializesAsync to the
concurrent-dispatch gtest: they fire N concurrent callMethodAsync() calls (the
fan-out pattern over the async consumer path, which the sync tests don't
exercise) and assert peak overlap 4 for "multi", 1 for "single".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-19 15:54:42 -03:00
Dario Lipicar 29afbac532 Extract the Logos protocol layer from logos-cpp-sdk (lp_* C ABI + protocol semver) (#2) 
* Extract the Logos protocol layer from logos-cpp-sdk

Transports (plain TCP/TLS, qt_local, qt_remote/QRO, mock), token manager,
consumer core (LogosAPIClient/LogosAPIConsumer incl. the capability
auto-requestModule flow), ModuleProxy, the abstract LogosProviderObject
interface, and the canonical QVariant<->JSON conversion — now behind the
language-neutral lp_* C ABI (logos_protocol.h) carrying the protocol
semver (LOGOS_PROTOCOL_VERSION_*, lp_protocol_version()).

Bytes crossing the ABI use the lossless {"_bytes": base64url} tagging
(NUL-safe), matching the plain wire encoding.

Provider lp_* surface is compiled groundwork; serving lands with module
authoring.

* consumer: typed requestModule for the capability flow

Port of logos-cpp-sdk master f5a127dd ('use updated capability module',
cpp-sdk#85, Iuri Matias) — the touched files (logos_api_client.cpp,
logos_api_consumer.{h,cpp}) moved into this repo in the P1 extraction.
The capability auto-requestModule path now calls a typed std::string
helper on the consumer (which acquires the capability object directly)
instead of a stringly invokeRemoteMethod round-trip. 111/111 tests.
 2026-06-12 18:59:01 -03:00