mirror of
https://github.com/logos-co/logos-protocol.git
synced 2026-08-27 20:11:07 +00:00
Dario LipicarandClaude Opus 5
0fef299362
* fix(plain): deliver async callbacks in a Qt-free host, and make release()-racing-a-call diagnosable
Two pre-existing defects in the plain transport's async surface. Both are
older than #45/#46 and neither is caused by the io_context fold; the fold is
just what this is stacked on.
DEFECT 5 — the async surface promised exactly-once and delivered ZERO in a
Qt-free host. Every completion went through one hop, and the hop was:
QCoreApplication* app = QCoreApplication::instance();
if (!app) return; // <- the callback, dropped
In a Qt host that branch only fires at shutdown, which is why it read as a
reasonable guard. In a process that never had a QCoreApplication — the
deployment the plain transport exists for — it fires for EVERY call, forever,
on all four resolvers (reply, deferred completion, deadline, cancellation).
Not an error, not a timeout: silence, which turns a bounded call into an
unbounded wait in every caller that awaits it, including lp_invoke_async and
every generated async wrapper.
Fixed with a dedicated DELIVERY THREAD, used only when the process has no Qt
loop. NOT inline on the completing stack: inline delivery on an Asio read
handler is the re-entrancy class that already cost this codebase a SIGSEGV
(deferred-multi completion on the QtRO read stack), so a fix that delivers by
removing the hop is not a fix. NOT the deadline thread either — user callbacks
there would make every deadline in the process hostage to user code, which is
exactly the coupling DeadlineService was extracted to prevent.
The Qt-loop check LATCHES, so Qt hosts see no behavioural difference at all:
instance() also goes null inside ~QCoreApplication, and module teardown after
the application is gone is what static-destruction ordering produces — with
stopAndCancelCalls() handing every in-flight call a cancellation callback at
exactly that moment. Running user code on a side thread into half-destroyed
module state would be a NEW failure mode introduced by a bug-fix change, so a
process that has ever been seen with an event loop keeps the old shutdown
behaviour. logos_object.h now states that residue instead of glossing it.
DEFECT 3 — release() racing a call on another thread. NOT FIXED, because it
cannot be, and the honest answer is a contract plus a detector.
release() ends in `delete this`, so a synchronous call parked in its future
wait dereferences freed memory when it comes back. Reproduced deterministically
on master (exit 139 under Guard Malloc, 3/3) and on cf1b9b0 (exit 139 with AND
without Guard Malloc, 3/3), faulting in callMethodWithError one line after the
wait.
It is not fixable from inside the object: every mechanism that could make the
racing call safe — a refcount, a flag, a lock, an epoch — is a MEMBER, so the
racing thread's first act would be to read it out of storage that has just been
freed. There is no synchronising with a destruction you can only learn about by
reading the destroyed object. Three alternatives were considered and rejected,
each for a stated reason (an atomic alive-flag is check-then-use on freed
memory; a blocking release() breaks the fast-teardown guarantee and deadlocks
in the shipped reentrant shape; an immortal forwarding handle works but trades
the crash for permanent retention proportional to requestObject count, in a
transport whose two preceding changes were spent proving retention does not
grow with call count — and would fix one of four transports). The reasoning is
in the note over PlainLogosObject::release().
So: the contract is stated (logos_object.h, plain_logos_object.h), and the
object counts entries into its public methods and REPORTS when release() or
the destructor finds the count non-zero — aborting in debug builds. The misuse
becomes a named diagnostic at the line that committed it instead of a SIGSEGV
somewhere else. It is a diagnostic, not a rescue, and it is deliberately biased
to under-report rather than ever accuse a correct program.
EVIDENCE, all by running:
* Defect 5: six detectors in a NEW binary (protocol_noqt_tests) that never
constructs a QCoreApplication — the state protocol_tests can never reach,
since its main() constructs one first. All six red on cf1b9b0 (0/300
replies, 0/40 deferred, 0/20 deadlines, 0/20 cancellations delivered),
all six green after, including under Guard Malloc.
* Defect 3: a death test red on BOTH pre-fix trees, 3/3 each, with and
without Guard Malloc ("died but not with expected error"), green after.
Its three companion tests prove the detector never fires on a correct
program, and were themselves validated by deleting the decrement from
EntryGuard's destructor in a throwaway build: all three then abort.
* Exactly-once still holds via the release-race shape — the only one that
detects a broken gate — on both delivery vehicles: 20 rounds x 500 calls
released mid-burst, 0 double deliveries, 0 dropped, with both resolvers
live, under Guard Malloc too.
* nix build '.#tests': 312/312 ctest cases pass. Both installed binaries run
clean through the exact CI commands.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(plain): bound the concurrent-callers wait, and fail the harness loudly
Two ways this file could have reported something other than what it measures.
An unbounded `while (ok < N) processEvents()` does not fail when it goes
wrong — it hangs the CI job until the job timeout, and a hang says nothing
about what broke. Bounded at 60s; the assertion below it then reports the
actual count.
And the death test's harness setup checked the host and the connection but
not the handle, so a failed acquire would have crashed on a null pointer and
been reported as "died but not with expected error" — indistinguishable from
the defect the test is looking for. It now exits 9 with a message, like the
other two harness paths.
Re-validated after the change: still red on cf1b9b0 (3/3), green here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(plain): the detector must not touch the object after dropping its count
CI caught this, on Linux, in the shape this whole change is about — the
detector inventing the use-after-free it exists to report.
EntryGuard's destructor restored m_lastEntryPoint AFTER decrementing
m_callsInFlight. That opens a window exactly one store wide: the count reaches
zero, a release() racing on another thread reads zero, concludes nothing is in
flight and runs `delete this`, and the store lands in freed memory. Exit 139
in IoFoldTest.ReleaseFromInsideAnIoThreadEventCallbackDoesNotWedge on
ubuntu-latest; macOS was green in the same run, and the retry was green too,
which is exactly how a one-store window behaves.
The count is now the FIRST and LAST thing either the constructor or the
destructor touches. Between them the object is covered — a concurrent release()
sees a non-zero count and reports. Outside them the guard touches nothing. The
cost is a vaguer message across threads (the restore now happens before the
decrement, so a reader can see the outer frame's name); a diagnostic string is
worth less than not storing into freed memory.
AND THE REASON IT WAS REACHABLE AT ALL: that test really does violate the
contract this PR documents. It issued its triggering `fire` call on the same
handle its io-thread event callback releases, so release() ran while the main
thread was still inside that handle's callMethodAsyncWithError. The violation
was always UB and always silent — the pre-existing code touches no member after
sendCallAsync() returns, so losing the race cost nothing observable — which is
why it stayed green for seven runs on #46. Adding bookkeeping to the epilogue
made it visible.
Both tests with that shape now fire the event through a SECOND handle, which
changes nothing about what they pin: the event still arrives on the io thread,
the handler still releases the handle it was delivered through, and that handle
still has an outstanding call for teardown to cancel.
Verified by running:
* With a 300ms sleep injected into callMethodAsyncWithError's epilogue — a
window the old code lost every time — both tests reported
"LOGOS FATAL: ... callMethodAsyncWithError()" before the fix and are clean
after it. That is the violation demonstrated and then removed, not narrowed.
* The same injection at 5ms across the WHOLE suite produces zero LOGOS FATAL
reports: no other test has this shape. (The one failure it causes,
IoFoldTest.ReleaseRacingRepliesInFlightDeliversEachCallOnce, is that test's
own "the race did not run" guard firing because a 5ms-per-call sleep lets
every reply land before the release — 10000 answered-by-reply, 0
by-teardown, 0 doubles, 0 drops. Correct behaviour from the test.)
* Full suite green again: 306/306 Qt, 6/6 no-Qt, and the UAF-sensitive subset
green under Guard Malloc.
* Detectors re-validated on cf1b9b0 after the edits: death test still red 3/3.
Also fixes a fragility this found in the new no-Qt race test. In that binary
the provider shares the process's single io thread with the consumer, so under
the nix sandbox the issuing thread enqueued all 500 calls and released before
one reply came back: answered-by-reply=0, cancelled-by-teardown=10000. Zero
doubles and zero drops — but only ONE resolver ran, so the exactly-once
assertion was proving nothing, which is precisely why the "both resolvers were
live" guards are in the test. It now waits for the first reply before
releasing; both resolvers are live every run (byReply 496-744, byTeardown
9256-9504 over six runs).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(plain): make release() safe against a call already inside the object
The defect this PR reported as unfixable is fixable, and the argument that
said otherwise conflated two different races.
That argument ran: every mechanism that could save the racing call is a
member of the object, so the racing thread's first act would be to read
freed storage. That is true of a call that ENTERS after destruction. It is
false of a call ALREADY INSIDE the object, which is the defect actually
reproduced — a synchronous callMethod parked in its future wait, released
from a second thread, faulting on the next line it executes. That call took
its bookkeeping on the way in, while the object was provably alive, so
release() cannot fail to see it.
So PlainLogosObject carries a live-reference count next to the counter the
detector already added: 1 for the owner plus one per caller inside a public
entry point. release() tears down and then drops THE OWNER'S reference
instead of `delete this`; whoever drops the count to zero destroys the
object, which for a racing call is that call's own thread on its way out.
EntryGuard takes the reference before it touches anything else and drops it
after everything else, because the drop may BE the delete.
SAFE now: release() concurrent with any call that entered first, sync or
async, any number of threads; and release() re-entered from inside a call
or an event callback on the same thread (shipped behaviour, io thread).
STILL a caller error, and still diagnosed: STARTING a call at or after
release() — its first act is to increment a counter that may already be
freed, so nothing in the object can save it — and `delete obj` in place of
release() with a call in flight, where there is no destruction left to
defer. Both report and abort in debug builds whenever the object still
exists to notice; when the storage is already freed there is nothing left
to look at, and that residue is the documented contract.
Two consequences worth naming. m_conn is no longer reset by release(): the
parked caller's next act is `m_conn->cancelPending(...)`, and resetting a
shared_ptr while another thread reads it is a data race on the shared_ptr
itself. And the object — with its share of the connection — now outlives
release() by however long the slowest call still inside it takes, which is
bounded by that call's own timeout. release() itself still blocks on
nothing: 0ms with an 8000ms call in flight, unchanged.
release() and the destructor call an unguarded disconnectEventsImpl(),
because taking a reference during destruction would drop it again and
recurse into the delete.
VERIFIED by running, on macOS arm64, debug:
* The reproduction now exits 0 through the real host stack; on cf1b9b0 the
child dies by signal, 3 runs of 3.
* The deterministic twin (a connection double that never answers, so the
park needs no timing assumption): release() returns in 0ms with the call
parked, destroyed=0 at that moment, destroyed=1 after the caller leaves,
and the caller reaches its post-wait cancelPending. On cf1b9b0: exit 139,
with and without Guard Malloc.
* The tight version — the double answers with a pending sentinel so
release()'s notify wakes the parked caller inside the window — 400 rounds,
one destruction each, 0 double deletes. On cf1b9b0 that one is SILENT
without Guard Malloc and 139 with it, which is noted in the test.
* Both remaining misuses die with their named diagnostic; both fail on
cf1b9b0, where no diagnostic exists to match.
* No false alarms: 310/310 protocol_tests, and with the detector's
decrement removed by hand all four "not accused" tests abort on a
correct program (rc=134), which is what makes them detectors.
* Guard Malloc clean over SyncCallReleaseRace, IoFold, PlainObjectTeardown,
PlainCompletionSubLifetime, PlainCancelPendingRace, PlainWaiterReaping.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(plain): keep the Qt-free delivery vehicle alive as long as its callers
The delivery thread this PR added fixed the drop and introduced a new
use-after-free one moment later in the process's life.
DeliveryService was an ordinary function-local static, so it was constructed
on the FIRST async delivery — which means every object with static storage
constructed before that (i.e. everything constructed during dynamic
initialisation) is destroyed AFTER it. A delivery issued from such a
destructor posted into an io_context that had already run its own destructor,
on a thread that had already been joined. Reproduced with nothing but the
null-connection early-return path: SIGSEGV under Guard Malloc inside
scheduler::post_immediate_completion, reached from __cxa_finalize, 3 runs of
3; and without Guard Malloc, silently, as delivered=0 — the exact drop this
class exists to prevent, moved to a later moment. So "exactly once holds for
the whole life of a Qt-free process" was still untrue.
FIX: the service is never destroyed and registers no destructor — a
`new`-ed pointer behind the function-local static, with the destructor
DELETED so no future edit can reintroduce one — and its thread is detached.
There is now no state in which the vehicle is gone but callers remain. The
old destructor's own comment worried about a user callback blocking the join
at static-destruction time; with no join there is no such hang, and exit()
does not wait for a detached thread. Costs: one io_context and one thread in
a process that is ending, and a callback that is RUNNING at process exit can
be cut off — the same exposure a Qt slot has when the loop's thread goes.
The alternative (detect the destroyed service and deliver inline) was
rejected: inline delivery is the re-entrancy class this hop exists to
prevent, and "we are at static destruction, so no io thread is running" is
not knowable from inside postDelivery — the completing thread there can be
IoContextPool's.
ALSO IN THIS COMMIT, because it is the same file and the same claim:
* THE INLINE CHECK NOW MEASURES NESTING. Test 1 read d.total after
callMethodAsyncWithError returned and asserted it was zero, which is a
race against the delivery thread and not an inline check: 7 failures in
200 runs here (the review reported 3/200 plain, 2/40 under Guard Malloc),
every one of them with on-caller-thread=0 — i.e. nothing had actually run
inline. This file already says as much about its own tests 2 and 5. The
replacement is a thread-local depth marker raised around the issuing call
and read BY THE DELIVERING THREAD at delivery time: a callback that runs
inline is nested on the issuing thread and says so from inside itself,
with no shared state and no timing. Applied to tests 1, 2 and 5, where it
also strengthens 5 — "did a cancellation run from inside release()" is now
nesting rather than a thread comparison.
* A HARNESS LIFETIME BUG in the same file: QtFreeHost held its
IncomingCallHandler as a member, RpcServer keeps a raw pointer to it and
nothing joins the io thread, so a frame already read from the socket could
be dispatched into freed storage. SIGBUS on the io thread inside
dispatchIncoming, 1 run in 25 (1 in 5 under Guard Malloc) once the run got
long enough for the io thread to reach the queued frames. The handler is
now deliberately leaked, which is the shape that cannot lose that race.
CONTRACT WORDING. logos_object.h promised exactly-once unconditionally. It
now promises AT MOST once always, EXACTLY once whenever the callback has
somewhere to run, and enumerates the three process-level cases where it does
not: after ~QCoreApplication in a Qt process; in a process that constructs a
QCoreApplication and never RUNS its loop (queued onto a loop that never
turns — unfixable here, and it was covered by the old unconditional promise);
and in a process whose QCoreApplication was TRANSIENT, where the latch keeps
dropping for the rest of that process's life. That last one is the price of
the first: from inside postDelivery "the app is gone because we are shutting
down" and "a helper's app object went out of scope" are the same observation,
and guessing the other way would run user callbacks on a side thread during
every Qt host's teardown. A process with no QCoreApplication in its life is
NOT on the list — there delivery now holds through static destruction, with
the only residue being the process exiting before the delivery thread runs.
VERIFIED by running, on macOS arm64, debug:
* The after-main window is now a TEST: a static destructor issues a delivery
and reports through the process exit code, because no test case runs
there. On the pre-fix delivery service it fails 3/3 (exit 70,
delivered=0) and 3/3 under Guard Malloc (139). On this commit:
delivered=1, off the issuing thread, exit 0.
* The de-flaked test: 0 failures in 250 runs plain, 0 in 60 under Guard
Malloc (was 7/200 before).
* Whole no-Qt binary: 40/40 clean plain, 10/10 clean under Guard Malloc
(was 1/25 and 1/5 with the SIGBUS above). Process exit adds ~50ms and does
not hang.
* All 7 no-Qt tests still fail on cf1b9b0 (0 deliveries), so defect 5 is
still what it was.
* 310/310 protocol_tests, 3 runs of 3.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(noqt): record that the exactly-once gate is TWO gates, not one
Re-validating the release-race exactly-once test against the no-Qt delivery
vehicle turned up a correction to what this suite says about its own
mechanism. AsyncCall guards a duplicate delivery twice — claim()'s
compare-exchange, and the swap in takeCallback() that leaves a second caller
holding an empty std::function — and the note in tests/protocol/CMakeLists.txt
describes only the first.
Measured, on the no-Qt twin (20 rounds x 500 calls released mid-burst):
* CAS removed, swap intact: 0 doubled deliveries. This test, its Qt twin
and PlainCancelPendingRaceTest all stay GREEN. So a validation that
removes only the CAS proves nothing about the gate.
* both removed: 22 doubled deliveries, this test FAILS — while
the three per-path exactly-once tests stay green, which is the difference
between a detector and a pin.
Neither half is redundant: the CAS is what stops a second caller from also
erasing registries and cancelling timers, and the swap is what protects the
callback itself. Comment-only; no behaviour change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>