Every LiveHost fixture in tests/protocol (five of them: test_iofold,
test_plain_object_teardown, test_plain_waiter_reaping,
test_plain_completion_sub_lifetime, test_call_error_after_acquire) puts a
ModuleProxy on a worker QThread and publishes it through a PlainTransportHost,
and every one of them freed that host from the TEST thread:
m_host.reset(); // test thread
m_thread->quit();
m_thread->wait();
That is a use-after-free with a millisecond-wide window, in code five test
files share.
WHY. publishObject() connects a lambda to the proxy's eventResponse signal with
NO context object, so it is a direct connection and runs on whichever thread
emits. ModuleProxy always QUEUES that emission to its own thread — it must, or
QtRO source serialization races the reply socket — so the emitting thread is
always the worker. The lambda converts the payload and then calls fanOutEvent,
which locks the host's m_mu. Free the host on the test thread and that lock is
on a destroyed mutex. ~PlainTransportHost does disconnect the connection, which
covers an emission that has not started; a worker already inside the lambda is
not called back, it is simply running, and qvariantListToRpcList on a real
payload sits in front of the lock.
MOVING reset() AFTER quit()/wait() IS NOT THE FIX, and the obvious reason for
saying so is wrong, so here is the measured one. That order does stop the fault:
wait() joins the worker, so an in-flight lambda has finished, and it is clean
under Guard Malloc. It is clean because it throws the queue away —
QThread::quit() reaches QEventLoop::exit(), which sets the exit flag
SYNCHRONOUSLY from the calling thread instead of posting an event, so the
worker's loop stops at its next iteration and discards every emission still
queued behind it. Same specimen, same load: 273-383 of 960 events delivered,
silently. In a suite whose tests count deliveries that is the worse failure,
because nothing reports it. It also shuts the host down AFTER the proxy's
thread, which test_call_error_after_acquire needs the other way round.
THE FIX, in one shared place (live_host_teardown.h) rather than five copies,
because a copied pattern is what this was: destroy the host ON the proxy's
thread, via the SDK's own logos::runOnOwnerThread marshal. A QMetaCallEvent is
dispatched by the worker's event loop, so while it runs the worker is by
definition not inside any other slot. Qt dispatches equal-priority events FIFO,
so every emission queued before it runs first, against a live host; everything
after finds the connection already severed by ~PlainTransportHost. The io
thread, the third thread that reaches the host, is still covered by the drain
barrier ~PlainTransportHost already carries — the proxy thread is not the io
thread, so that barrier's running_in_this_thread() check still takes the
blocking path. The added blocking wait introduces no hang that was not already
there: the next two statements are quit()/wait() on the same thread, with no
timeout.
EVIDENCE. test_plain_host_event_teardown.cpp is the specimen, and it is a
detector: 24 wide events queued per round, teardown aimed at the trailing edge
of the first so the worker is inside the host's lambda. Validated the way this
directory validates detectors — against a real checkout of the code it replaces
(cf1b9b0), with the fixture's teardown as it was:
pre-fix, no detector: SIGABRT 3/3 runs ("mutex lock failed: Invalid
argument" out of a Qt event handler)
pre-fix, Guard Malloc: SIGSEGV 3/3 runs at plain_transport_host.cpp:354
in fanOutEvent, on the thread named "QThread", under
ModuleProxy::eventResponse
naive (quit/wait/reset): NO fault, either detector — and 273-383 of 960
emissions delivered; the rest silently dropped
fixed, no detector: 960/960 emitted, 0 after the free, 40/40 rounds
still draining when teardown began
fixed, Guard Malloc: same counts, clean
WHAT THIS IS NOT. The five fixtures do not fault on their own today: pre-fix,
their suites are clean under Guard Malloc (2/2 runs, 33 tests) because their
test bodies drain the burst before teardown. So this closes a live trap in
shared fixture code — one that two separate probes fell into by copying the
pattern — rather than a reproduced CI failure. It remains a candidate for the
suite's unexplained SIGSEGVs, not a proven cause, and it is stated that way in
the test file.
Full suite: 304/304 (302 before, +2 new). `nix build .#tests`: 304/304 via
ctest, 95s. The five affected suites are clean under Guard Malloc on the fixed
tree.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* perf(plain): fold the per-call waiter thread into a call object with its own clock
An async call on the plain transport used to be an OS thread whose entire job
was to be blockable: std::future cannot be waited on with a deadline AND a
cancel, so the waiter polled it in 25ms slices, parked on a condition variable
for the deferred half, and delivered. Three costs came with that — one thread
per pending RPC, a 25ms floor on teardown, and a registry-plus-reaping protocol
to stop finished threads accumulating, because a thread cannot join itself. The
TODO in callMethodAsyncWithError has said to fold it away since it was written.
A call is now a shared_ptr<AsyncCall>: state that the reply (delivered as a
handler rather than parked in a promise), a deadline, and cancellation race to
finish. Nothing captures `this`. Handlers hold a shared_ptr to their AsyncCall
and a weak_ptr to CallState, so "no handler touches a destroyed object" is true
by construction rather than by a barrier, and the join is replaced by ownership.
postToQtEventLoop is kept verbatim as the re-entrancy firebreak: all four
completion sites route through it, so no user callback ever runs on an Asio
stack.
Measured against pristine 0f26ffd, same probe compiled into both:
* 32 calls parked in a provider that will not answer: threads 5 -> 37 on
master, 6 -> 6 here (the 6th is the deadline clock below, not per-call);
* release() with those 32 in flight: 26-29ms -> 0ms;
* 24,000 calls resolved by their deadline and never answered: 8.5MB of
resident memory on master, 353 bytes per call growing strictly linearly,
against 0.95MB here that has stopped growing by 3,000 calls.
THE DEADLINE GETS ITS OWN THREAD, and that is the one decision worth arguing
with. Hanging the per-call timer off the connection's strand is the obvious
move and it has a real regression: IoContextPool runs ONE thread for the whole
process and this transport delivers user onEvent callbacks INLINE on it, so an
event handler that calls another module — ordinary module code — holds every
deadline in the process. Measured on that design: an onEvent handler making a
2000ms call delayed a 200ms deadline on a DIFFERENT connection to 2003ms, and a
handler that never returns meant the deadline never fired at all. A second io
thread does not fix it (user handlers are unbounded, so N blocked handlers need
N+1 threads); moving inline event delivery off the strand is a much larger
change to event ordering for every consumer; a Qt timer is strictly worse,
because a synchronous call from the Qt thread blocks that loop too. So:
DeadlineService, one thread process-wide, doing nothing but arming, cancelling
and firing timers. Both shapes are back to 200ms and 250ms, and are pinned.
Two things closed on the way, neither of them inherited:
* RpcConnection::cancelPending(). m_pendingCalls was emptied only by a decoded
reply and by fail()'s sweep, so a call resolved by its DEADLINE left its
registration there for the life of the connection — which outlives every
handle it hands out. That was true of the promise before this change too;
the fold would have made the orphan bigger, so it is closed rather than
passed on. The sync call path and getMethods withdraw theirs as well.
* A sentinel that arrives after its own deadline used to be filed under
CallState::deferred by a reply handler that had not noticed the call was
already resolved. deliver() therefore leaves the registries BEFORE the
exactly-once gate, not after it.
TESTS. tests/protocol/test_iofold.cpp is the evidence, and its detectors are
validated by an explicit inverted build rather than asserted:
cmake -S tests -B build-broken -DLOGOS_PROTOCOL_DETECTOR_INVERSIONS=ON
which removes the exactly-once CAS and puts the deadline back on the shared
io_context. In that build the deadline tests fail at 2002ms and never-fires
respectively, and the release-vs-replies race reports 6-16 double deliveries per
10,000 calls. Finding a race wide enough to be a RELIABLE exactly-once detector
took three attempts and the two rejected candidates are documented in the file:
the plain outcomes are weak detectors (one resolver, one deliver()), and
release-against-a-single-completion caught one double in seven runs. Teardown
against a burst of arriving replies is the one that works, because teardown
snapshots the whole in-flight map and then delivers with the lock released.
test_plain_waiter_publish_is_last.cpp is DELETED. It pinned exactly one rule —
publishFinishedWaiter() is a waiter thread's last access to the object, which
was the only reason stopAndJoinWaiters() could return while a reaper was still
mid-join. There are no waiter threads, no reaper and no publish list, so there
is no ordering left to pin; the property it protected is now structural.
test_plain_waiter_reaping.cpp is retargeted at CallState::inflight, keeping its
claims and losing its mechanism.
The four guarantees from #41 all re-measured on this branch: no thread growth
with in-flight calls, teardown 0ms with 32 outstanding, exactly one callback on
normal/deferred/timeout/cancellation counted per call over 10,000 calls
including a release race, and retention final 0 on both registries. Guard Malloc
clean over the lifetime suites (27 tests), with the completion-subscription
detector still faulting 3/3 on pristine master. ctest 296/296 (287 before, minus
3 deleted, plus 12); nix build .#tests 296/296.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(plain): say what cancelPending() actually guarantees, and pin it
rpc_connection.h claimed that "a caller that gives up before the reply arrives
calls cancelPending() and is never called back at all". It is not true.
dispatchIncoming copies the handler out of m_pendingCalls under m_mu and invokes
it with the mutex RELEASED, so a cancelPending() landing in that gap erases an
entry that is already gone and returns having stopped nothing — the handler then
runs to completion, after cancelPending() has returned.
THE CODE IS FINE; THE COMMENT WAS NOT, and the distinction it was blurring is
the load-bearing one:
* at-most-once INVOCATION of a registered handler is this layer's, and comes
from the extract-and-erase under m_mu — three contenders (dispatchIncoming,
fail()'s sweep, cancelPending) and only one can have it;
* exactly-once DELIVERY to the user is NOT this layer's. It is
AsyncCall::deliver()'s CAS, and nothing else.
Both callers of sendCallAsync() were checked, because a non-idempotent one would
have made this a bug rather than a comment. PlainLogosObject funnels every
outcome into deliver(). RpcConnection::sendCall()'s promise handler has no CAS
and needs none: only one contender ever reaches it, and fulfilling a future its
caller has already walked away from is a no-op. Same for the two cancelPending()
callers that are not deliver() — the sync callMethodWithError timeout and
getMethods().
Three comment sites corrected (ResultHandler, dispatchIncoming's Result arm,
cancelPending's own contract) and the claims moved out of prose into
tests/protocol/test_plain_cancel_pending_race.cpp, which BUILDS the interleaving
instead of racing for it: a stub connection reproduces dispatchIncoming's
extract-then-invoke and lets the test stand between the halves. Four tests — the
callback that fires after cancelPending() returns, the extracted reply racing
teardown (exactly one delivery), the promise-shaped handler in the same gap, and
a real RpcConnection pair proving the half of the old comment that IS true.
The exactly-once one goes RED under -DLOGOS_PROTOCOL_DETECTOR_INVERSIONS=ON:
2 deliveries for 1 call, deterministically rather than probabilistically. Six
tests now go red in that build, listed in tests/protocol/CMakeLists.txt. Full
suite 302/302.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(plain): take the fold's detector inversions out of the transport too
Three more compiled-in alternative implementations, same anti-pattern as
the one the base branch just lost: production source holding a second,
deliberately wrong version of its own contract so the suite could be
built once with the mechanism removed.
* deadlineContext() had LOGOS_PLAIN_DETECTOR_BREAK_DEADLINE_ISOLATION
returning IoContextPool::shared().ioContext() — the rejected design
DeadlineService exists to avoid. Kept DeadlineService::shared()
.context(); the io_context_pool.h include the fold added for that
branch alone goes with it.
* AsyncCall::claim() had LOGOS_PLAIN_DETECTOR_BREAK_ONCE storing
`delivered` and returning true unconditionally. Kept the CAS.
* AsyncCall::takeCallback() had the same macro returning a COPY of the
callback. Kept the swap.
And the CMake option that defined all three, whose surviving content —
which six tests are real detectors, and that the per-path exactly-once
tests are PINS rather than detectors because a call resolved once calls
deliver() once whatever guards it — moved into the note that replaces
it.
The comments now describe the validation that actually happened: a local
edit in a throwaway checkout, with the numbers each run produced. The
sub-order detector needs no edit at all, since it goes red on pristine
master.
302 tests pass, unchanged in count: no test deleted or weakened.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(iofold): count the provider's completion workers so none outlives the proxy
Fixes the Linux SIGSEGV (exit 139) in the "Run protocol tests" step.
`OmniProvider::defer` is a "multi" provider: it answers with a pending sentinel
and pushes the real completion later from a worker of its own. That worker calls
the EventCallback ModuleProxy handed the provider, and that listener captures
`this` RAW (module_proxy.cpp) — its first act is
`QMetaObject::invokeMethod(this, …, Qt::QueuedConnection)`, which dereferences
the QObject. The worker was spawned DETACHED, so nothing proved it had finished,
and `~LiveHost` runs `delete m_proxy` a couple of milliseconds after the last
spawn.
Nothing keeps the two apart. A round of ReleaseRacingAnInFlightCompletionIsSafe
ends when the CALLER is answered, and a release()d call is answered by teardown —
so the round can be over before the provider has run at all, and the proxy's
thread is still working through a backlog of `defer` calls while ~LiveHost is
already tearing down. Two to five workers were standing in the same frame at the
moment of the fault:
Thread "QThread" received signal SIGSEGV
QObject::thread() const
QMetaObject::invokeMethodImpl(QObject*, …)
ModuleProxy::ModuleProxy(...)::<lambda(const QString&, const QVariantList&)>
module_proxy.cpp:37
std::function<void(const QString&, const QList<QVariant>&)>::operator()
OmniProvider::callMethod(...)::<lambda()> test_iofold.cpp, in defer
std::thread::_State_impl<…>::_M_run()
WHY ONLY THE WHOLE-BINARY RUN. The corpse is left by the test that spawned the
worker and lands in whichever test runs NEXT — in CI, always
ReleaseFromInsideAnIoThreadEventCallbackDoesNotWedge, which follows the 300-round
ReleaseRacingAnInFlightCompletionIsSafe and its 300 `defer` calls. Under ctest
every test is its own process, so the worker dies with the process that owned the
proxy and there is nothing left to fault: `nix build '.#tests'` is green on the
unfixed tree, which is exactly how this hid.
Measured on Linux (aarch64, Qt 6.9.2, gcc 14.3, `nix build '.#tests'` artifact,
QT_QPA_PLATFORM=offscreen, five concurrent copies):
tree IoFoldTest.* whole binary
master (03842db) n/a (no test) 0/10
feat/plain-async-io-fold 6/50 (12%) 2/20 (10%)
+ harness-host-thread-affinity 10/50 (20%) 5/20 (25%)
this commit, on either tree 0/50 0/20
So it is latent HERE and merely widened by destroying the fixtures' host on the
proxy's thread: that order lets the queued `defer` backlog RUN instead of
discarding it with QThread::quit(), which is why the rate roughly doubles. The
defect and the fix both belong to this commit's tree.
macOS never opens the window — 3/3 clean whole-binary runs on the unfixed tree,
and 4/4 clean under Guard Malloc (which unmaps the freed page, so a late worker
would fault every time), which is why that half of the matrix stayed green.
A COUNT, NOT A JOIN. Joinable workers would hold their 8MB stacks until reaped —
400 outstanding in NormalAndDeferredCompletionsDeliverExactlyOnceAtVolume — and
reaping from the dispatch thread would block the very thread `defer` exists to
free. So they stay detached and the provider counts them, with the decrement and
its notify under one mutex so a drain() woken by it cannot return before the
worker has released that mutex.
~LiveHost drains AFTER `m_thread->wait()`: the proxy's event loop has stopped, so
no queued call can reach the provider any more and the worker set is FINAL —
before that point a drain could pass and the backlog spawn more. ~OmniProvider
drains too, because a worker's last act touches one of its members.
Verification, all on the fixed tree:
* `nix build '.#tests'` — 402/402, Linux and macOS (173s on macOS).
* 20 whole-binary Linux runs: 402/402 every time.
* Exactly-once, on the release-race shape rather than the per-path pins:
200,000 calls released mid-burst across those 20 runs (20 rounds x 500 each)
reported DOUBLE deliveries=0 dropped=0, and all 20 runs of
ReleaseRacingAnInFlightCompletionIsSafe were 300/300 with 0 doubles.
* Teardown stays fast: `release()` 0-2ms with 32 calls in flight, and the
added wait is on the FIXTURE, not on release() — it delays `delete m_proxy`
by however long a completion worker still had to sleep (<=2.4ms here).
* No production code touched, so "no user callback inline on an io thread" and
the deadline guarantees are unchanged.
Not touched: test_plain_completion_sub_order.cpp (InstantMultiModule) and
test_concurrent_dispatch.cpp spawn the same detached completion worker, and read
the callback member through a captured `this` on top of it. Neither has been
observed to fault.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(plain): the completion subscription must not outlive the object it points at
PlainLogosObject::ensureCompletionSub() registered the deferred-completion
handler with raw `this` captured. That handler is stored in the RpcConnection,
which is SHARED by every handle the connection hands out and outlives all of
them — release() says so itself, and ends in `delete this`. So a completion
event arriving across a release() ran a handler holding a dangling pointer, on
the io thread, on a path nothing joins: #41's waiter JOIN covers the per-call
waiter threads and nothing else.
The unsubscribe release() sends is real — RpcConnection::sendUnsubscribe erases
the entry under the connection's mutex — but it cannot close this, because
dispatchIncoming copies the handler out under that mutex and then invokes it
with the mutex dropped. An erase racing an already-copied handler changes
nothing about the invocation in flight.
Reproduced, not assumed. tests/protocol/test_plain_completion_sub_lifetime.cpp
widens the window with a large completion payload (the conversion between the
copy and the handler's first touch of the object) and aims release() into it
using a wildcard subscriber as a clock. On master:
* SIGSEGV under macOS Guard Malloc, 3/3 runs, faulting in
pthread_mutex_lock <- std::mutex::lock <- ensureCompletionSub()::$_0 <-
onEvent()::$_0 <- dispatchIncoming <- doRead <- IoContextPool's thread;
* without a detector, 4/5 runs die differently and just as fatally: the freed
mutex makes pthread_mutex_lock return EINVAL, std::mutex::lock() throws, and
the exception unwinds into doRead()'s catch, which fail()s the whole
connection. That is the per-round isConnected() assertion in the test.
The fix moves the rendezvous (mutex, condvar, completions map) into a
shared_ptr-held block and hands the handler a weak_ptr, so "no handler touches
a destroyed object" holds by construction: a handler that locks it keeps it
alive for one callback, one that cannot lock it does nothing. Nothing else in
the object was reachable from that handler, which is what keeps this to two
files; rpc_connection.h is untouched.
Verified after the fix: repro clean 10/10 plain and 3/3 under Guard Malloc,
with the same cadence and 24/24 releases still landing inside a dispatch — the
window is still exercised, it is just no longer a use-after-free. The control
(same storm, nothing released) is clean under the same detector on both sides,
so the detector is not objecting to the load. All four #41 guarantees re-measured
and unchanged: waiters joined (60/60 rounds), teardown 10-22ms against master's
6-27ms, exactly one callback on all four outcomes, registry final=1 after 200 /
600 / 800 / 1600 calls. Full suite 287/287, including nix build .#tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(plain): a concurrent first caller must WAIT for the completion subscription, not just see the flag
ensureCompletionSub() raised m_completionSubscribed under the rendezvous mutex
and then RELEASED that mutex before subscribing. Two threads entering
callMethod() on the same fresh object is enough: the second reads "subscribed",
builds its Call and puts it on the wire while the first has not enqueued the
Subscribe frame yet. A "multi" provider that answers such a call quickly emits
its completion into a subscription the host has not registered —
PlainTransportHost::fanOutEvent finds no sink for that connection and DROPS it —
and the caller waits out its whole timeout for a result that was computed and
thrown away.
A LOST COMPLETION, NOT A CRASH, which is why it survived: the failure looks like
a slow provider, arrives seconds after the code that caused it, and leaves
nothing behind.
PRE-EXISTING, not introduced by this branch: pristine master has the identical
flag-then-subscribe shape and reproduces at 42/400 two-thread first-call rounds
(this branch before the fix: 28/400; through the real host stack: 10/600 calls).
It ships here, as its own commit, because it is four lines in the very function
this PR rewrites and in the same subscription this PR is about.
The fix is std::call_once plus a release/acquire fast path. Serializing is the
whole of it: a second caller blocks until the first has both registered the
client-side callback and enqueued the Subscribe frame, and asio then keeps the
two posts in that order because the mutex supplies the happens-before edge its
strand guarantee is conditioned on.
Rejected: holding the rendezvous mutex across the subscribe (works, but makes
the io thread's completion handler wait on the connection's write path — that
mutex exists to hand a completion over, not to gate I/O); subscribing eagerly in
the constructor (kills the race outright but costs a Subscribe frame and a host
sink per handle, deferred call or not).
tests/protocol/test_plain_completion_sub_order.cpp pins both halves — the wire
order, observed at a provider that stamps every frame it receives, and the
consequence through PlainTransportHost with nothing instrumented at all. Both go
RED under -DLOGOS_PROTOCOL_DETECTOR_INVERSIONS=ON, which restores the pre-fix
shape: 5 of 5 broken runs failed (25-37 dropped completions per 250 rounds), 8
of 8 fixed runs were clean. Full suite 289/289.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(plain): validate the detectors against master, not a compiled-in inversion
The transport carried a second ensureCompletionSub() behind
LOGOS_PLAIN_DETECTOR_BREAK_SUB_ORDER: the pre-fix racy shape, reachable
from a -DLOGOS_PROTOCOL_DETECTOR_INVERSIONS=ON configure of the tests
tree, so the ordering tests could be shown to fail. That is the same
anti-pattern as the getenv() probes an earlier draft carried, wearing a
build flag instead — production source keeping a deliberately wrong
implementation of its own contract — and it does not belong in the PR.
Both detectors are validated by the stronger check anyway: this file
compiles unmodified on master, which still raises the flag under the
rendezvous mutex and drops it before subscribing, and still captures raw
`this` in the completion handler. Numbers now in the comments are from
that run, not from the synthetic build:
sub-order raw wire 18/26/27/28 of 250 rounds inverted, dropped and
timed out, four runs
sub-order real stack 6 to 10 of 500 calls timed out
sub-lifetime RED in 11 of 12 solo runs, connection dying at
round 6 in 9 of them
The 400/600-round figures the comments quoted were also stale: 66e0153
moved both tests to 250 rounds.
No test is deleted or weakened. The std::call_once fix and the weak_ptr
handler capture are untouched; only the alternative implementation and
the CMake option that reached it are gone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(plain): make the burst-drain experiment a burst on both platforms
BurstThatGoesIdleDrainsWithoutAnotherCall claimed to detect "reaping only
happens on the spawn path". It did not, on Linux, at any bound — because the
experiment was not a concurrent burst there.
"Issue 800 calls in a loop and hope they overlap" is a race between two rates:
spawning a std::thread, and a loopback RPC coming back. On macOS the first is
much cheaper and the burst is real (544-612 of 800 still outstanding when the
last call goes out). On Linux they are comparable, so the burst completes WHILE
IT IS BEING ISSUED and the spawn-path reaper — the very reaper this test is
supposed to be doing without — collects it: 0-237 outstanding, median 85, and
one run in 25 with the entire burst answered before the last call was issued.
Both arms then drew from overlapping distributions and no number could separate
them. 400 scored 25/25 on macOS and 0/40 on an idle Linux box; the 8 before it
failed correct code 96 runs in 160.
So gate the provider instead of arbitrating the race. Every call in the burst
invokes `gate`, which parks; dispatch is single-threaded, so the first arrival
holds up all 800 and not one reply exists until the test releases them. The
burst's own precondition is now asserted rather than assumed, off the provider's
reply counter and the registry size: 800/800 in flight, 0 replies, in all 760
runs of both arms at every load level.
The drain is then PACED, 16 at a time, which is the other half. Releasing all
800 at once replaces one scheduling artefact with another: a reaper that
collects a large batch sits in its join loop while everyone behind it publishes,
so on 6-core Linux correct code leaves 2-389 (macOS: 1). That is a real property
of the exit guard, not a defect, and it now says so in plain_logos_object.h —
but it is useless as a detector, since the defect is only ~2x it. Paced, the
residue is the last step's exit batch: measured over 380 runs, worst 1 on macOS
and 7 on Linux, and the unloaded Linux cell is the worst one.
The bound is therefore a constant again, and stated against the release step
rather than the burst, because that is what bounds the quantity: 4 * kRelease =
64. 9.1x above the worst correct-code value seen, 12.5x below the defect. With
the exit-guard reap removed the residue is 800 — kBurst exactly, not "several
hundred" — in every one of 380 runs on both platforms at every load level,
because nothing is left that can remove an entry. THE DEFECT IS NOW CAUGHT ON
LINUX: 200/200, where the previous bound caught 0/40.
Cannot hang, constructed three ways. The gate is bounded and opens itself on the
way out, so a release that never happens costs one budget and fails naming the
gate (20.2s, measured) instead of wedging the host 800 times over; a drain step
that stalls breaks the loop instead of spending fifty budgets; and a bail-out
with the burst still in flight now releases and pumps through a scope guard
before Deliveries dies — without it that path is a SIGABRT on a destroyed mutex
(3/3), which is a crash where a verdict is wanted.
800 live waiter threads is the new peak, up from ~150 on Linux. Lazily faulted:
peak RSS 29MiB Linux / 32MiB macOS.
nix build .#tests: 289/289 on both platforms.
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
BurstThatGoesIdleDrainsWithoutAnotherCall failed on ubuntu-latest at 21, then at
10 on a re-run, against EXPECT_LE(idle, 8u) — having scored 7 against that same
8 the run before. The change under review is not involved: the same source
compiles to a byte-identical object file with and without it.
WHAT THE RESIDUE IS. A waiter reaps only OTHERS, never itself, so what survives
an idle burst is whatever published after the FINAL reap: the last waiter to
finish has nobody behind it, and a waiter sitting in the join loop of its own
reap has not published yet while the batch it did not collect already has. That
is the size of the last exit batch, which is the scheduler's business.
AND NOTHING TAKES IT LATER, so this is not a window that was too short. Sampled
from 100ms to 25.6s after the burst went quiet the count does not move: 5->5 and
17->17 idle, 2->2 under 4x CPU oversubscription, and 24->24, 33->33, 49->49,
82->82, 123->123, 138->138, 168->168 under 32x — 12 runs, every one flat. It is
a residue, not a drain in progress.
MEASURED, 20 runs per cell, this 800-call burst, m_waiters when it goes idle:
this code reaping only on the spawn path
macOS idle 1 every run 572-723
Linux idle 1-80 (median 9) 4-168 (median 80)
Linux 2x CPU 1-145 (median 14) 16-527 (median 275)
Linux 4x CPU 1-104 (median 28) 10-504 (median 271)
Two things fall out of that, and the second is why this commit says more than
"the number was too small".
1. 8 WAS READ OFF THE macOS COLUMN. On Linux it sits under the MEDIAN of a
correct build — 35 of 60 unloaded runs of correct code exceed it — so the
test was failing correct code in most Linux runs. CI's 7 was luck.
2. THE ~600 THIS TEST IS DOCUMENTED AGAINST IS macOS-ONLY. On Linux the burst
is not concurrent: spawning 800 std::threads costs more than a loopback
ping, so most of it has already been collected by the SPAWN-path reaper
before the last call is issued, and the defect's own residue collapses
into the same range as a correct build's (4-168 idle, min 4). The two arms
overlap there at any bound, 8 included. This assertion is a coarse
retention check on Linux, not the detector for that defect.
THE NEW BOUND is kBurst/2 — the majority of the burst must have retired itself
with no further call — because the residue has no ceiling for a tighter
fraction to sit under. Worst per load level, 620 runs on a 6-core Linux box:
idle 152 (n=60) 2x 145 (n=20) 4x 172 (n=80) 8x 175 (n=160)
16x 278 (n=120) 32x 402 (n=60) 64x 317 (n=40)
Flat out to 8x, climbing after. A quarter of the burst (200) would have been
the original mistake in a new unit: it clears the worst by 1.14x, the same
ratio as 7-against-8. Half clears everything up to 16x by 1.44x and the worst
CI has ever produced (21) by 19x. The single run in 620 that scored 402, at 32x
oversubscription, is recorded in the comment rather than rounded away.
Both assertions in the test take the same expression, the second included: a
residue the follow-up call did not collect is the same retention bug, and a
tighter hard-coded number there would only move the magic constant somewhere
quieter.
Also corrects the retention note in plain_logos_object.h, which quoted "1-2
after a 2000-call burst" as though it were platform-independent.
THE DETECTOR, rebuilt with the defect this test exists to catch — the reap
dropped from the waiter's exit guard, leaving only the spawn path:
this assertion, macOS RED 10/10, 550-614 against 400
this assertion, Linux 16x RED 4/10, up to 645
this assertion, Linux idle GREEN 0/15, 25-208 — see below
publish-is-last, macOS RED 5/5
publish-is-last, Linux RED 8/8 (green 3/3 with the reap in place)
nix build '.#tests' fails its own checkPhase with the defect in
The third line is a real loss of Linux coverage in THIS assertion and it is
stated in the comment rather than glossed: on an unloaded Linux box no bound
that a correct build survives will catch it, because the burst is not
concurrent there. It costs the SUITE nothing — with the exit-guard reap gone,
PublishedWaiterDoesNotTouchTheRegistryAgain is RED deterministically on both
platforms, and it is that test, not this one, that pins the reap. If this one
ever has to be the detector again, the answer is to pace the provider so the
burst is concurrent on every platform, not to tighten the number.
No behaviour change: the only non-comment edit is the bound.
VERIFIED: nix build '.#tests' green on macOS (289/289, 69.6s) and Linux
(289/289, 78.3s).
(cherry picked from commit f147aed2c6)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(lp): lp_invoke_async can finally report a failure
lp_result_cb has always been documented as carrying an outcome —
"ok != 0 -> `json` is the result JSON value; ok == 0 -> `json` is the
canonical error object" — and the synchronous twin lp_invoke has always
honoured it (LP_ERR_UNAVAILABLE + out_error_json). lp_invoke_async did
not: it subscribed with the VALUE-ONLY invokeRemoteMethodAsync overload
and called back `cb(1, json, user_data)` with ok hard-coded to 1, so a
call to a module that cannot be acquired reached the callback as a
SUCCESS carrying a default-constructed value.
The fix is to pass a TWO-argument lambda, which is invocable only as
LogosAPIClient::AsyncResultErrorCallback and so binds to the
CallError-aware overload that already exists next to the value-only one.
The failure is then rendered with the same makeErrorJson the sync path
uses, so both entry points report the same event in the same shape.
The ABI is unchanged. lp_result_cb's (ok, json, user_data) signature
already reserves ok == 0 for exactly this; nothing had to grow a new
entry point, and every in-tree consumer already branches on `ok`
(logos-rust-sdk's async_call_trampoline even parses `message` out of the
canonical error object — code written against a contract the
implementation never kept).
Regression test: a matched pair over a REAL transport (plain TCP), not
the mock.
FAILING async call -> ok=0 {"code":"object_unavailable", ...}
SUCCEEDING async call -> ok=1 7
The first fails on the unfixed tree (ok=1, json "null"); the second
passes on both, so an over-eager "report failure everywhere" fix cannot
sneak through.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(protocol): report the failures that happen AFTER acquire, on both twins
#40 made lp_invoke_async able to report a failure, but only for the two
conditions produced ABOVE the transport: acquire failure and the unauthorized
sentinel. Everything the transport learns while the call is in flight was still
discarded — PlainLogosObject answered a bare QVariant() for a timeout and for
`ResultMessage.ok == false` alike, and LogosAPIConsumer hard-coded an empty
CallError next to it.
Two ordinary failures therefore still reported success on both entry points:
a TIMEOUT, and MODULE NOT LOADED against a host that is up (which is not an
acquire failure on the plain wire — requestObject hands back a handle for any
name over an open connection).
The information already exists: ResultMessage carries err/errCode, the futures
know they expired, QtRO knows its pending call never finished. It had nowhere to
go because LogosObject's callMethod returns a lone QVariant and its
callMethodAsync callback takes a lone QVariant.
Widening those virtuals would append a vtable slot to an installed, subclassed
interface, so instead this adds LogosObjectErrorChannel — a SIBLING interface
reached by dynamic_cast. LogosObject's size, layout and vtable are unchanged
(verified: a subclass compiled against the old and new headers emits the same
14-entry vtable with identical slot indices), and a transport that does not
implement it keeps today's behaviour.
logos_protocol.cpp needs no change: lp_invoke and lp_invoke_async already render
this CallError, so both twins gain the coverage together.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(protocol): stop the macOS flake that was sinking #41
Three real races the new CallErrorAfterAcquire suite exposed (and that
Copilot flagged on the QtRO half):
1. ~PlainTransportHost stopped the acceptor but did not quiesce the shared
Asio I/O thread. Server-side RpcConnections hold a raw IncomingCallHandler*
back to the host; a fail()/onConnectionClosed racing teardown freed the
handler mid-call. That is the macOS CI SIGSEGV in
AsyncSuccessStillReportsTheValue — it fires with no output of its own
because the previous live-host test's destructor left the heap corrupted.
Restore the I/O barrier that landed on the qtfree branches but never on
master (proven: 80/80 clean on the CI crash sequence that was ~2/50 before).
2. PlainLogosObject::callMethodAsync detached its per-call waiter while
capturing `this`. release()/delete this could then race the waiter.
Join waiters in the destructor/release, and register the thread under the
lock before it can outrun teardown.
3. QtRO async could deliver the user callback twice when the timeout timer
and the pending-call watcher finished around the same moment, violating
the exactly-once contract. Gate both paths (and the deferred-completion
arm) on one atomic.
Also drain queued onCall invokes after host.reset() in the #40 live-target
control, matching LiveHost's teardown discipline.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(protocol): the drain barrier must not dangle on its own timeout
Two defects in the barrier added by 1e9c934, both on the path it takes when
it fails:
std::promise<void> drained; // stack local
boost::asio::post(ioc, [&drained] { drained.set_value(); }); // by REFERENCE
fut.wait_for(std::chrono::seconds(5)); // result dropped
1. The wait is bounded, so on timeout this frame returns while the posted task
is still queued -- and the task holds a pointer to a destroyed stack object.
set_value() then writes to freed stack memory. The bound that stops a wedged
I/O thread hanging teardown introduced the exact class of use-after-free the
barrier exists to prevent. The promise is now a shared_ptr captured BY VALUE,
so the task keeps it alive whether or not anyone is still waiting.
2. The wait_for result was discarded. A timeout means the barrier did NOT hold
and we are about to free an IncomingCallHandler that a live connection may
still call back into -- the original crash, minus any way to know it
happened. It now warns, naming the consequence.
Neither is reachable while the I/O thread drains promptly, which is why the
suite is green either way; both matter precisely when it does not, which is
the only situation the barrier is for.
Tests: 270/270.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(protocol): teardown must not wait out the call it is abandoning
Joining the per-call waiters (rather than detaching them) closed a real
use-after-free: the waiter captures `this`, and release() used to `delete this`
underneath it. But joinWaiters() could only join. It had no way to ASK a waiter
to stop, so destroying a PlainLogosObject with a call in flight blocked for the
remainder of that call's timeout — up to 20s on the protocol default. A module
unloading mid-call stalled the unload for that long, on the releasing thread.
Measured, 8s call timeout, provider parked:
release() before after
future wait (site 1) 7804 ms 11 ms
deferred completion (site 2) 7703 ms 0 ms
Both blocking sites are now interruptible, and they need different treatment:
* the std::future wait cannot be interrupted at all, so it is SLICED: one
deadline computed up front, waited in 25ms increments, stop flag checked
between them. Teardown latency is one slice; the timeout the caller asked
for is unchanged, because the last slice ends exactly on the deadline. 25ms
is under two frames (so a module unload stays imperceptible) and costs 40
wakeups/second per in-flight call, which is nothing beside the Qt event loop
these threads already sit next to.
* awaitCompletion's condition_variable is interruptible by construction:
widen the predicate, notify_all. No latency floor at all — hence 0 ms. The
flag is published under m_completionMu so a waiter cannot evaluate the
predicate, decide to sleep, and then miss the notify.
A CANCELLED CALL STILL DELIVERS, EXACTLY ONCE. This is the part a naive fix
breaks: callMethodAsyncWithError and lp_invoke_async promise the callback fires
exactly once, so a waiter that simply returns on stop trades a bounded stall for
an unbounded hang in every caller awaiting it. Proven by building that naive
variant: it passes the latency test and fails three exactly-once tests with the
callback never arriving.
The code is "transport_error", from the existing vocabulary rather than a new
one, since these codes are the wire contract. logos_call_error.h defines it as
"the connection failed or was torn down mid-call", which is precisely what
happened — the consumer tore its own end down. The alternatives all misattribute
it: "object_unavailable" says the module is absent (it is not, and callers
re-acquire on that code), "call_failed" blames the peer for a dispatch it
performed fine, and "timeout" — what this used to report, after waiting the
deadline out — claims a deadline elapsed that did not. It is also already what
the wire produces for the same event seen from the other side (callErrorFromWire
maps TRANSPORT_CLOSED to transport_error).
Delivering during teardown is safe because postToQtEventLoop touches nothing
owned by the object: it is a free function taking the callback, value and error
BY VALUE, and the waiter copies objectName/method up front. That was already
true and is now load-bearing, so it is documented at the function. The queued
lambda runs after the object may be gone; everything the waiter reaches through
`this` runs before the join returns, which is why the join must stay.
Also closes the registration window it opens: a call arriving after the stop
would push a thread onto an m_waiters that teardown has already swapped out, so
it would never be joined. It is answered as cancelled instead.
The UAF is verified still closed under macOS Guard Malloc rather than ASan —
libclang_rt livelocks in its own initializer before main on this toolchain, for
both ASan and TSan, on a hello-world. Under Guard Malloc the race test is clean
across 5 runs and SIGSEGVs immediately when the join is turned back into a
detach, so the check is a real detector and not a vacuous pass.
Tests: 277/277 (was 270; 7 new). CallErrorAfterAcquireTest hammered 40x, 0
failures — it was ~2/50 flaky before this branch's earlier fixes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(protocol): a call that FINISHED must not park its thread for the object's life
The per-call waiters are joinable rather than detached, which is what closed the
use-after-free where release() deleted the object under a still-running waiter
(4f9d824), and they are interruptible, so teardown no longer waits out the call's
timeout (731e579). Both stay. What neither did was retire a waiter that had
FINISHED: m_waiters was only ever swap()ped, in stopAndJoinWaiters(), so an
exited-but-unjoined std::thread — whose stack and pthread struct are not
reclaimed until somebody joins it — stayed parked for the lifetime of the handle.
Measured against a live PlainTransportHost over TCP, every call completing
normally, one handle held throughout, before:
10000 calls m_waiters 300 -> 10300 rss +156.56 MiB 16417 B/call
30000 calls m_waiters 300 -> 30300 rss +469.28 MiB 16403 B/call
and the same through the production C ABI — one lp_client, N lp_invoke_async —
at +156.53 MiB. That path is why this matters: LogosAPIConsumer caches ONE
handle per module and reuses it for every async call, releasing it only on
eviction or teardown (cpp/logos_api_consumer.cpp:129 and :207), so a
long-lived module leaks per lp_invoke_async. The ~16KB constant is one page on
this 16KiB-page arm64 and will be smaller elsewhere; the UNBOUNDEDNESS is the
platform-independent part, and follows from m_waiters.size() rising 1:1 with
completed calls and only ever falling in teardown. Attribution: the retention
arrived with the join in 4f9d824, not with 731e579 — but 731e579 is what makes
the join permanent.
The registry is now KEYED, because a thread cannot join itself and so a waiter
can never retire its own entry. Each waiter publishes its id as its FINAL act (a
scope guard declared first, so it destructs last, covering all four exit paths),
and the next spawn — plus teardown — joins those ids and erases them. Joining a
thread that has already returned is a couple of syscalls. Same probe, same
workload, after:
10000 calls m_waiters 15 -> 16 rss +0.08 MiB 8 B/call
30000 calls m_waiters 16 -> 16 rss +0.06 MiB 2 B/call
10000 calls via lp_invoke_async rss +0.09 MiB 10 B/call
Retention is now bounded by the waiters that finish after the LAST spawn, i.e.
by peak in-flight concurrency — 16 at the in-flight window above, and exactly 1
when calls are issued sequentially — instead of by call count.
THE DEADLOCK THIS SHAPE INVITES is a reaper that joins while holding m_waiterMu,
against a waiter blocked on m_waiterMu trying to publish. It is avoided by
construction rather than by argument: nothing is joined with a lock held, in the
reaper or in teardown, whatever a waiter does on its way out. Proven by building
the naive variant that does join under the lock — the new hammer wedges it, with
the main thread in reapFinishedWaiters -> pthread_join and a waiter in
publishFinishedWaiter -> mutex wait, and the test's watchdog names the cause
instead of letting CI hang.
Teardown's guarantee is restated rather than weakened. It is not "every waiter
has been joined by the time stopAndJoinWaiters() returns" — a waiter a
concurrent reaper is mid-join on is no longer in the map — but the thing that
guarantee was ever for: NO WAITER TOUCHES THE OBJECT AFTER IT RETURNS. An entry
leaves m_waiters only once its thread has published, and publishing is that
thread's last access.
The TODO above the waiter still stands: the real fix is to fold the wait into
the shared Asio io_context and have no thread per pending RPC at all. This makes
the interim honest; it does not replace that.
Two more things review turned up, folded in here:
* The two wait sites resolved stop-vs-result in OPPOSITE directions.
waitForResult tested the stop flag BEFORE polling, so an already-ready
future was still reported as transport_error, while awaitCompletion
deliberately preferred a completion that had landed — and both were
commented as intentional. One rule now, applied to both: AN ANSWER ALREADY
IN HAND BEATS A CONCURRENT STOP, and the stop only decides what happens when
there is nothing to hand over. The callback fires either way
(postToQtEventLoop copies everything it delivers), so the only thing a stop
can change is what the callback SAYS — and manufacturing transport_error
while the true answer sits in the future reports a failure that did not
happen, to callers that re-acquire, retry and log on that code. Preferring
the answer costs nothing, since it is already there: the flag is still
checked before every sleep, so the teardown-latency bound is unchanged.
* CORRECTION to 731e579's message, which claimed it "closes the registration
window" where a call arriving after the stop would never be joined. That
branch is unreachable in defined behaviour: m_stopping is raised only by
teardown, so any thread that can read it inside callMethodAsyncWithError is
already calling a method on an object whose destructor is running — the load
is itself the use-after-free, reproduced as a SIGSEGV on that commit and on
its parent alike, and nothing inside that function can repair it. The guard
is harmless and stays (one predictable branch, and it fails safe with one
callback), but its comment now says what it is instead of claiming a fix it
does not make.
Verified by running, with every check first shown to FAIL on unfixed code:
* Retention: the probe above, plus a committed regression test that reads
m_waiters out of the live object through the explicit-instantiation access
hole ([temp.spec] does not check access on an explicit instantiation's
template arguments) — so the code under test keeps its private state, with
no friend, no test-only accessor and no `#define private public`. 200
sequential completed calls keep 1 waiter; without pruning they keep 200.
* Exactly-once on all four paths — normal completion, timeout, cancellation
and the deferred-completion (pending-sentinel) arm — counted PER CALL so a
dropped one and a doubled one cannot cancel out, plus the 60-round
release-during-call race. Shown to catch a cancelled path that returns
silently (3 failures) rather than delivering.
* Teardown latency unchanged from 731e579: 10-17ms with an in-flight 8000ms
call and 0-1ms mid-defer, against 15ms / 1ms on that commit.
* The UAF stays closed: 11 teardown + reaping tests clean under macOS Guard
Malloc (ASan/TSan remain unusable on this toolchain).
* Full suite 281/281 twice, `nix build .#tests` green (281/281 in the
sandbox), CallErrorAfterAcquireTest hammered 40x clean.
* fix(protocol): a burst that goes quiet must not wait for a call that never comes
378d889 retired finished waiters, but from ONE site: the async-call spawn path.
So whatever finishes after the LAST spawn is never reaped, and a module that
bursts and then goes idle parks it all until the handle dies. Measured on
378d889, one handle, 2000 concurrent calls, every one delivered:
after 2000 completed calls, IDLE: m_waiters=1428 rss=+24.17 MiB
after ONE further call: m_waiters=1 rss=+ 1.92 MiB
The unbounded-per-call class was gone; this is what it left behind, and the
second line is the whole diagnosis — the corpses go the instant anything calls
again, so the reaper works and simply never runs. LogosAPIConsumer caches one
handle per module and never releases it between calls, so "bursts, then quiet"
is not a corner case: it is a UI that fans out on a refresh and then waits for
the user.
A finishing waiter now reaps the OTHER finished waiters before publishing
itself, so a burst drains as it completes. Same probe, same workload:
after 2000 completed calls, IDLE: m_waiters=1 rss=+ 1.88 MiB
THE BOUND IS ONE, NOT ZERO, and by construction rather than by luck: a waiter
can only reap OTHERS (a thread cannot join itself), so the last one to finish
has nobody behind it to collect it. Anything that publishes after the final
reap survives too, which is why 12 runs of the probe gave 1 eleven times and 2
once. Those go on the next call, or in teardown. Retention now tracks neither
call count nor peak concurrency — the sequential and in-flight-16 numbers move
from "15 -> 16 waiters" to "1 -> 1" — and the memory figures are unchanged
against 378d889 where they were already flat: 10k sequential +0.00 MiB, 10k at
16 in flight +0.06 MiB, 30k +0.09 MiB, and 10k through the production C ABI
(one lp_client, N lp_invoke_async) +0.09 MiB / 10 B per call, the same as
378d889 reported.
THE ORDER IS THE SAFETY ARGUMENT. Reap first, publish last, never the reverse:
* Publishing is what makes a waiter joinable BY ANOTHER WAITER. Reaping first
keeps that relation one-way — unpublished threads join published ones,
published ones join nobody — so it has no cycles. Inverted, two waiters
publishing in the same instant can each take the other's thread out of
m_waiters and then join it; both are already out of the registry, so
teardown does not even wait for them. Built that variant: pthread_join
detects the cycle and throws, the half-drained thread vector then destroys
a still-joinable thread, and the process aborts — the EXISTING hammer
(ReapingRacesPublishingWithoutDeadlocking) catches it 5 runs out of 5, with
the stack showing two waiters inside FinishOnExit joining each other.
* While a waiter is unpublished it is still in m_waiters, so a concurrent
teardown joins it and the object cannot be destroyed under the reap. Once
published, a reaper may take its thread out of the map and release() may
`delete this` — and a reaper on the CALLER's thread (the spawn path) is one
teardown neither knows about nor waits for, so a post-publish touch of
m_waiterMu is a use-after-free on a member mutex. That path needs a caller
still issuing calls while another thread releases, which this class already
treats as caller-side UB, so it is stated as an argument; the cycle above is
what the tests actually demonstrate.
Two corrections to 378d889, which this change makes load-bearing rather than
cosmetic. NOT amended into it — it is pushed, and a commit that misstates its
own reasoning is better read alongside the correction than rewritten.
* plain_logos_object.h:107-109 said reapFinishedWaiters() is "called on every
async spawn ... and from stopAndJoinWaiters()". It is not, and never was,
called from stopAndJoinWaiters(): teardown does its own id-independent
brute-force join, which is precisely why it needs no cooperation from the
reaper. Harmless behaviourally, wrong in a mechanism whose entire argument
is who joins what and when. The comment now names the two real callers —
the spawn path and, as of this commit, every waiter on its way out.
* 378d889's message presented "the join is outside the lock" as THE property
that prevents the reaper deadlock, "proven by construction" by its hammer.
That is overstated, in a way that would let the guarantee be refactored
away with the suite still green. TWO independent properties each suffice:
joining only PUBLISHED ids (a published waiter never needs m_waiterMu
again, so it cannot be the thread being shut out), and joining outside the
lock. The hammer only wedges when BOTH are gone. Measured, on top of this
change: the variant that joins under the lock but KEEPS the published-only
filter passes ReapingRacesPublishingWithoutDeadlocking in 293/297/290ms
across three runs and the whole reaping suite besides, while the variant
that joins everything under the lock trips the watchdog at 60s. So a later
"simplification" that moves the join inside the lock would ship green. Both
properties are kept, and the comment now says which one the test is
actually testing.
The TODO above the waiter still stands: the real fix is to fold the wait into
the shared Asio io_context and have no thread per pending RPC at all. This
makes the interim honest; it does not replace it.
Verified by running, each check first shown to FAIL on unfixed code:
* Retention: the burst probe above, plus a committed regression test that
reads m_waiters out of the live object through the explicit-instantiation
access hole. 800 concurrent completed calls, then IDLE with NO further
call: 1 waiter left, 20 runs out of 20. On 378d889 the same test leaves
610 of 800 and fails. The pre-existing sequential and in-flight tests are
unchanged and still pass.
* The UAF stays closed — the check that matters most here, because this adds
an object access late in the waiter's life. 9 reaping/teardown-race tests
plus the 7-test teardown suite clean under macOS Guard Malloc (ASan is
unusable on this box: it hangs in its own initializer). DETECTOR VALIDATED
both ways: turning teardown's join back into a detach SIGSEGVs under Guard
Malloc on the release-during-call hammer (exit 139), and the specific
inversion this change risks — reaping AFTER publishing — aborts as
described above.
* No deadlock: reap-vs-publish hammered 20x (1600 calls in 40 overlapping
bursts each), plus 60 rounds of teardown landing from another thread while
the tail of a burst retires itself, plus 6x600-call bursts checking that
LIVE OS threads (task_threads, which counts wedges and not corpses) come
back to baseline every round. Clean; the watchdog names the cause if it
ever is not.
* Exactly-once on all four paths — normal, timeout, cancellation, deferred
sentinel — counted per call. Each detector validated with a broken build:
dropping the cancelled callback fails 4 tests, dropping the timeout one
fails its test, and double-delivering the normal/deferred arm fails those.
* Teardown latency unchanged from 378d889: 1-25ms with an in-flight 8000ms
call and 0ms mid-defer across 5 runs, against 2-21ms / 0ms on that commit —
the same one-wait-slice (25ms) bound, since a waiter's extra work happens
after it has stopped waiting.
* Full suite 282/282 three times, `nix build .#tests` green,
CallErrorAfterAcquireTest hammered 40x clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(protocol): pin publishing as a waiter's LAST access to the object
PlainLogosObject's waiters are joinable, interruptible and reaped, and all
three rest on one ordering rule that nothing in the suite could see:
~FinishOnExit() {
self->reapFinishedWaiters(); // others, never itself
self->publishFinishedWaiter(id); // strictly last
}
reapFinishedWaiters() erases published entries from m_waiters under m_waiterMu
and joins those threads OUTSIDE it. stopAndJoinWaiters() swaps m_waiters under
the same lock and brute-force joins whatever it got. So a waiter that a
concurrent reaper is mid-join on is NOT in teardown's map, and teardown can
return — with release() going straight on to `delete this` — while that waiter
is still unwinding. stopAndJoinWaiters() already says this in as many words:
the guarantee is not "everything is joined when this returns" but "no waiter
touches this object after this returns". Publishing being last is the entire
reason the second sentence is true, so one member access below it is a
use-after-free, and moving the publish above the reap is a join cycle.
THE DEFECT SHIPS GREEN. Rebuild plain_logos_object.cpp with a single object
read after the publish and the whole of PlainObjectTeardownTest and
PlainWaiterReapingTest passes, 10 runs out of 10, cleanly under Guard Malloc.
That is not a hole in those suites. No SUPPORTED caller can provoke it: under
calls-in-flight-plus-release, every waiter is still joined transitively,
because a waiter leaves m_waiters only via teardown (which joins it) or a
reaper, and a reaper is either another waiter — itself in m_waiters until after
its join returns — or the async-spawn path, whose join completes before the
call returns. The one uncovered reaper is the spawn path racing a concurrent
release(), and calling a method on an object another thread is releasing is
caller-side UB that faults on correct code too. A test built on that race would
be red on green code, so it is not a usable detector.
SO STOP RACING AND OBSERVE. tests/protocol/test_plain_waiter_publish_is_last.cpp
drives a real PlainLogosObject through a scripted RpcConnectionBase — no socket,
no host, no event-loop timing, and the test decides exactly when the call's
future is satisfied — and watches the accesses in two halves.
* THE STATE. The object is placement-newed into an mmap'd two-page arena, put
down so a page boundary lands at m_waiterMu: the members teardown
coordinates on go on the second page, everything else on the first. The
first page is mprotect(PROT_NONE)'d for exactly as long as a waiter runs,
and a SIGSEGV/SIGBUS handler RECORDS each access — address, thread, and how
many ids were published at that instant — then unprotects so the access
proceeds. Nothing crashes; the access is evidence. A correct waiter touches
that page zero times: objectName and method are copied into the closure
precisely so it needs nothing from the object. Four rounds, one per exit
path out of the lambda (answered, rejected, timed out, cancelled), since all
four end in the same guard.
* THE REGISTRY, which that page cannot cover because publishing has to reach
it. Caught with bait, using the reaper's own shape: reapFinishedWaiters()
joins outside m_waiterMu, so a waiter that has picked up somebody else's
finished thread sits in that join holding nothing — a window the test holds
open as long as it likes, because the thread being joined is one the test
planted and keeps parked. Plant bait 1; let the call finish; the exit guard
reaps, takes it, parks. Plant bait 2 at leisure. Release bait 1; the waiter
finishes its reap and publishes. Bait 2 must still be registered. Bait 1
doubles as a check that the reap really does join with the lock free.
Neither half is probabilistic. A third test proves the detector can fire at all,
so the two "this counter stayed at zero" assertions are not vacuous.
MEASURED, rebuilding the file under test with each defect (caught/runs):
defect below publishFinishedWaiter() new teardown+reaping
------------------------------------ --- ----------------
read m_objectName 40/40 0/10
read m_conn 10/10 0/10
read m_completions 10/10 0/10
read m_completionSubscribed 10/10 0/10
lock m_mu 10/10 0/10
call reapFinishedWaiters() again 20/20 2/2
read m_stopping 0/10 0/10
(publish moved ABOVE the reap) 0/5 12/15
no defect — 8f0c60f 0/40 0/10
The one gap is m_stopping, the single member sharing the registry's page, which
cannot be guarded without guarding the publish. The inverted order is left to
the reaping suite's hammer, which has it covered. Runtime 0.9-1.0s for all
three tests; clean 40/40 on 8f0c60f, and clean 3/3 under Guard Malloc
(MALLOC_PROTECT_BEFORE=1, banner confirmed) — the test never touches freed
memory itself, which is the other half of not being built on UB. No Guard
Malloc needed to detect anything: mprotect and the bait are the detectors.
Also: nix build .#tests 100% (285/285), the full binary 285/285, and
CallErrorAfterAcquireTest 40/40.
CORRECTIONS to measurements claimed earlier on this branch. All three were
overstated in the same direction — a single sample read as a constant:
* "ReapingRacesPublishingWithoutDeadlocking aborts the process, 5 runs out of
5" (plain_logos_object.cpp, and 378d889's message) is 12 runs in 15, ~80%.
It is a race detector, so one green run of it proves nothing — which is
exactly the argument for the deterministic suite added here. Corrected in
the comment.
* C-ABI retention was reported as "+0.09 MiB / 10 B per call" for 10k
lp_invoke_async on one client (8f0c60f's message). ~6 B/call. Same
conclusion — flat — different arithmetic.
* The burst retention figures 1428 (2000 calls, idle) and 610 of 800 came
back as 1421 and 599 on re-measure of the same build. Race-dependent, same
magnitude, which is why the tests assert a bound and not a value. Noted in
test_plain_waiter_reaping.cpp so the next reader does not treat them as
reproducible constants.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(protocol): make the publish-is-last probe fail loudly, and state the rule it actually checks
Three defects in cb015f5's regression test, plus the corrections that commit's
own CORRECTIONS section still owed. No behaviour change: the diff against
8f0c60f under cpp/ is comment lines only, verified by filtering the diff.
(a) THE BAIL-OUT PATH HUNG INSTEAD OF FAILING, which is the one that can stall
CI. PublishedWaiterDoesNotTouchTheRegistryAgain plants parked "bait" threads
behind gates and registers them in m_waiters. An ASSERT that fires before
gate1.open() — ASSERT_TRUE(tookBait1) is the obvious one — returns from the
function, and then ~PlainLogosObject blocks forever joining a thread nobody
will release. Reproduced by removing the reap from the exit guard: the
assertion PRINTS and the run still ends as a timeout kill, exit 124, with no
test result at all.
The gates are now opened by a scope guard on every exit path, and declared
BEFORE the GuardedObject so they outlive the teardown that joins the threads
parked on them. Same break, after: the same assertion, exit 1, 10.0s — which
is the probe's own tryWithRegistry budget and not a hang.
This is the shape a future refactor trips, not a hypothetical: the TODO
above the waiter (fold the wait into the shared Asio io_context) moves where
reaping happens, which is exactly the edit that makes tookBait1 false.
Both tests also stopped capturing their delivery counter by reference. On a
bail-out the cancelled call's callback is delivered on a later event-loop
iteration, i.e. after the frame is gone — a real use-after-free on the way
out of a failing test in a file about use-after-free. Owned by the callback
now.
(b) THE STATE ASSERTION WAS STRICTER THAN THE INVARIANT. It asserted
accessCount() == 0; the rule is only "no access AFTER the publish", and the
fault handler already stamps each access with how many ids were published at
that instant, so it can tell them apart.
Not hypothetical either. On the deferred/"multi" path a CORRECT waiter calls
awaitCompletion() (plain_logos_object.cpp:338), which locks m_completionMu
and reads m_completions and m_objectName — all on the guarded page, all
before it publishes. cb015f5 was green only because none of its four rounds
returned a pending sentinel, and the header's claim that "a correct waiter
touches that page ZERO times, before the publish or after" was true only of
the non-deferred rounds.
So: a fifth round drives the pending-sentinel path (ScriptedConn now answers
with the sentinel; nothing pushes the completion, so awaitCompletion runs out
its deadline), and the assertion narrowed to accesses stamped published >= 1.
PROVEN BOTH WAYS on this tree — with the old accessCount() == 0 predicate the
new round fails on correct code, naming offset 144 with "0 waiter id(s)
already published"; with the narrowed one the suite is 30/30 clean.
The round cannot pass vacuously: it REQUIRES at least one recorded access, so
a machine slow enough to turn it into a plain timeout fails it instead of
quietly proving nothing. Each round also now asserts m_finishedWaiters is
empty before arming, which is what makes "published >= 1" mean "after THIS
waiter's publish".
Two things guard against the narrowing being a quiet disarm:
* the handler now re-arms. It could not before (the faulting instruction
re-runs immediately), so the observing thread does it — it polls the
registry anyway and never touches the guarded page. Without it the first
legitimate access disarms the detector for the whole round.
* the catch rates were re-measured, not assumed. They are unchanged.
(c) TWO OVERSTATED NUMBERS, in the section whose whole point was to stop
overstating. Fixed where they live; cb015f5 is pushed and is not rewritten.
* "C-ABI retention ~6 B/call, not 10" replaced one sample with another.
10k lp_invoke_async on one lp_client, run ten times: 0, 5, 5, 5, 7, 7, 8,
10, 13, 10 bytes/call (mean 7.0). Ten more, run here: 3, 11, 8, 5, 8, 10,
13, 8, 3, 10 (mean 7.9). One distribution, range 0-13; both 6 and 10 sit
inside it and 8f0c60f's arithmetic (0.09 MiB / 10k) was not wrong.
THE HONEST STATEMENT IS THAT IT IS FLAT: indistinguishable from zero, RSS
noise and not a per-call rate. Recorded in test_plain_waiter_reaping.cpp
beside the other retention figures, where the next person to quote one
will see it.
* the table cell "reapFinishedWaiters() again ... 2/2" for the older
teardown+reaping suites was a two-run sample printed beside 10-40 run
samples. Re-measured over 30 runs: 9/30 here, 12/30 on another 30-run
sample — roughly one run in three, matching what reapFinishedWaiters'
own comment already said ("about one run in four"). The cell now reads
9/30, and the table says to read that column as rates and the left-hand
one as deterministic.
ALSO STATED PLAINLY, because it was overstated in review: the window where
"teardown returns while a reaped waiter is still unwinding" is NOT reachable by
a supported caller. A waiter leaves m_waiters only via teardown (which joins it)
or via a reaper, and that reaper is either another waiter — still registered
itself, since it reaps before it publishes, so teardown joins it and therefore
waits out the join it is in — or the async-spawn path, whose join completes
before the call returns. The only uncovered reaper is the spawn path racing a
concurrent release(), which is caller-side UB on any version of this class.
So publish-is-last is an invariant the design rests on and documents, not a
lurking use-after-free. This suite pins it against future edits; it does not
close an open hole. The file header, both failure messages and the comment in
plain_logos_object.cpp now say that instead of implying otherwise.
MEASURED AFTER THE CHANGE, rebuilding plain_logos_object.cpp with each defect
below publishFinishedWaiter() and running the suite (caught/runs), beside the
numbers from before it:
defect before after
------ ------ -----
read m_objectName 25/25 25/25
lock m_mu 25/25 25/25
write m_completions under m_completionMu 25/25 25/25
call reapFinishedWaiters() again (bait) 20/20 20/20
read m_conn - 10/10
read m_completions - 10/10
read m_completionSubscribed - 10/10
read m_stopping (declared blind spot) 0/10 0/10
publish moved ABOVE the reap (delegated) 0/5 0/5
no defect 0/30 0/30
Nothing moved, including the two declared blind spots — a narrowing that had
started catching or stopped catching something would show here. cb015f5 reported
40/40 for m_objectName from a longer run; 25/25 is this run, not a regression.
WHERE THE DEFERRED ROUND IS WEAKER, said here rather than left to be found: on
that one round the post-publish half is best-effort. A legitimate access opens
the page, the re-arm is a syscall behind, and a defect firing a microsecond
later slips through — measured with every round forced to run, the other four
catch a post-publish m_objectName read 5/5 and the deferred round 0/5, and a
variant that spins on the re-arm instead of polling records 4-24 accesses per
round and still catches it 0/5. It costs nothing: FinishOnExit is ONE piece of
code shared by all five exit paths, so the same defect is the same defect on
every round and the other four catch it deterministically. The deferred round is
there to keep the assertion honest about correct code, not to add a fifth copy
of the same detection.
Verified: PlainWaiterPublishIsLastTest 30/30 clean, the three waiter suites
15/15, the full binary 285/285, `nix build .#tests` 100% (285/285),
CallErrorAfterAcquireTest 40/40. Suite runtime 1.3-1.5s for the three tests
(0.9-1.0s before — the deferred round waits out a 400ms completion deadline).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
