mirror of
https://github.com/logos-co/logos-protocol.git
synced 2026-08-27 12:01:15 +00:00
0f26ffdeef
* fix(lp): lp_invoke_async can finally report a failure
lp_result_cb has always been documented as carrying an outcome —
"ok != 0 -> `json` is the result JSON value; ok == 0 -> `json` is the
canonical error object" — and the synchronous twin lp_invoke has always
honoured it (LP_ERR_UNAVAILABLE + out_error_json). lp_invoke_async did
not: it subscribed with the VALUE-ONLY invokeRemoteMethodAsync overload
and called back `cb(1, json, user_data)` with ok hard-coded to 1, so a
call to a module that cannot be acquired reached the callback as a
SUCCESS carrying a default-constructed value.
The fix is to pass a TWO-argument lambda, which is invocable only as
LogosAPIClient::AsyncResultErrorCallback and so binds to the
CallError-aware overload that already exists next to the value-only one.
The failure is then rendered with the same makeErrorJson the sync path
uses, so both entry points report the same event in the same shape.
The ABI is unchanged. lp_result_cb's (ok, json, user_data) signature
already reserves ok == 0 for exactly this; nothing had to grow a new
entry point, and every in-tree consumer already branches on `ok`
(logos-rust-sdk's async_call_trampoline even parses `message` out of the
canonical error object — code written against a contract the
implementation never kept).
Regression test: a matched pair over a REAL transport (plain TCP), not
the mock.
FAILING async call -> ok=0 {"code":"object_unavailable", ...}
SUCCEEDING async call -> ok=1 7
The first fails on the unfixed tree (ok=1, json "null"); the second
passes on both, so an over-eager "report failure everywhere" fix cannot
sneak through.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(protocol): report the failures that happen AFTER acquire, on both twins
#40 made lp_invoke_async able to report a failure, but only for the two
conditions produced ABOVE the transport: acquire failure and the unauthorized
sentinel. Everything the transport learns while the call is in flight was still
discarded — PlainLogosObject answered a bare QVariant() for a timeout and for
`ResultMessage.ok == false` alike, and LogosAPIConsumer hard-coded an empty
CallError next to it.
Two ordinary failures therefore still reported success on both entry points:
a TIMEOUT, and MODULE NOT LOADED against a host that is up (which is not an
acquire failure on the plain wire — requestObject hands back a handle for any
name over an open connection).
The information already exists: ResultMessage carries err/errCode, the futures
know they expired, QtRO knows its pending call never finished. It had nowhere to
go because LogosObject's callMethod returns a lone QVariant and its
callMethodAsync callback takes a lone QVariant.
Widening those virtuals would append a vtable slot to an installed, subclassed
interface, so instead this adds LogosObjectErrorChannel — a SIBLING interface
reached by dynamic_cast. LogosObject's size, layout and vtable are unchanged
(verified: a subclass compiled against the old and new headers emits the same
14-entry vtable with identical slot indices), and a transport that does not
implement it keeps today's behaviour.
logos_protocol.cpp needs no change: lp_invoke and lp_invoke_async already render
this CallError, so both twins gain the coverage together.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(protocol): stop the macOS flake that was sinking #41
Three real races the new CallErrorAfterAcquire suite exposed (and that
Copilot flagged on the QtRO half):
1. ~PlainTransportHost stopped the acceptor but did not quiesce the shared
Asio I/O thread. Server-side RpcConnections hold a raw IncomingCallHandler*
back to the host; a fail()/onConnectionClosed racing teardown freed the
handler mid-call. That is the macOS CI SIGSEGV in
AsyncSuccessStillReportsTheValue — it fires with no output of its own
because the previous live-host test's destructor left the heap corrupted.
Restore the I/O barrier that landed on the qtfree branches but never on
master (proven: 80/80 clean on the CI crash sequence that was ~2/50 before).
2. PlainLogosObject::callMethodAsync detached its per-call waiter while
capturing `this`. release()/delete this could then race the waiter.
Join waiters in the destructor/release, and register the thread under the
lock before it can outrun teardown.
3. QtRO async could deliver the user callback twice when the timeout timer
and the pending-call watcher finished around the same moment, violating
the exactly-once contract. Gate both paths (and the deferred-completion
arm) on one atomic.
Also drain queued onCall invokes after host.reset() in the #40 live-target
control, matching LiveHost's teardown discipline.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(protocol): the drain barrier must not dangle on its own timeout
Two defects in the barrier added by 1e9c934, both on the path it takes when
it fails:
std::promise<void> drained; // stack local
boost::asio::post(ioc, [&drained] { drained.set_value(); }); // by REFERENCE
fut.wait_for(std::chrono::seconds(5)); // result dropped
1. The wait is bounded, so on timeout this frame returns while the posted task
is still queued -- and the task holds a pointer to a destroyed stack object.
set_value() then writes to freed stack memory. The bound that stops a wedged
I/O thread hanging teardown introduced the exact class of use-after-free the
barrier exists to prevent. The promise is now a shared_ptr captured BY VALUE,
so the task keeps it alive whether or not anyone is still waiting.
2. The wait_for result was discarded. A timeout means the barrier did NOT hold
and we are about to free an IncomingCallHandler that a live connection may
still call back into -- the original crash, minus any way to know it
happened. It now warns, naming the consequence.
Neither is reachable while the I/O thread drains promptly, which is why the
suite is green either way; both matter precisely when it does not, which is
the only situation the barrier is for.
Tests: 270/270.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(protocol): teardown must not wait out the call it is abandoning
Joining the per-call waiters (rather than detaching them) closed a real
use-after-free: the waiter captures `this`, and release() used to `delete this`
underneath it. But joinWaiters() could only join. It had no way to ASK a waiter
to stop, so destroying a PlainLogosObject with a call in flight blocked for the
remainder of that call's timeout — up to 20s on the protocol default. A module
unloading mid-call stalled the unload for that long, on the releasing thread.
Measured, 8s call timeout, provider parked:
release() before after
future wait (site 1) 7804 ms 11 ms
deferred completion (site 2) 7703 ms 0 ms
Both blocking sites are now interruptible, and they need different treatment:
* the std::future wait cannot be interrupted at all, so it is SLICED: one
deadline computed up front, waited in 25ms increments, stop flag checked
between them. Teardown latency is one slice; the timeout the caller asked
for is unchanged, because the last slice ends exactly on the deadline. 25ms
is under two frames (so a module unload stays imperceptible) and costs 40
wakeups/second per in-flight call, which is nothing beside the Qt event loop
these threads already sit next to.
* awaitCompletion's condition_variable is interruptible by construction:
widen the predicate, notify_all. No latency floor at all — hence 0 ms. The
flag is published under m_completionMu so a waiter cannot evaluate the
predicate, decide to sleep, and then miss the notify.
A CANCELLED CALL STILL DELIVERS, EXACTLY ONCE. This is the part a naive fix
breaks: callMethodAsyncWithError and lp_invoke_async promise the callback fires
exactly once, so a waiter that simply returns on stop trades a bounded stall for
an unbounded hang in every caller awaiting it. Proven by building that naive
variant: it passes the latency test and fails three exactly-once tests with the
callback never arriving.
The code is "transport_error", from the existing vocabulary rather than a new
one, since these codes are the wire contract. logos_call_error.h defines it as
"the connection failed or was torn down mid-call", which is precisely what
happened — the consumer tore its own end down. The alternatives all misattribute
it: "object_unavailable" says the module is absent (it is not, and callers
re-acquire on that code), "call_failed" blames the peer for a dispatch it
performed fine, and "timeout" — what this used to report, after waiting the
deadline out — claims a deadline elapsed that did not. It is also already what
the wire produces for the same event seen from the other side (callErrorFromWire
maps TRANSPORT_CLOSED to transport_error).
Delivering during teardown is safe because postToQtEventLoop touches nothing
owned by the object: it is a free function taking the callback, value and error
BY VALUE, and the waiter copies objectName/method up front. That was already
true and is now load-bearing, so it is documented at the function. The queued
lambda runs after the object may be gone; everything the waiter reaches through
`this` runs before the join returns, which is why the join must stay.
Also closes the registration window it opens: a call arriving after the stop
would push a thread onto an m_waiters that teardown has already swapped out, so
it would never be joined. It is answered as cancelled instead.
The UAF is verified still closed under macOS Guard Malloc rather than ASan —
libclang_rt livelocks in its own initializer before main on this toolchain, for
both ASan and TSan, on a hello-world. Under Guard Malloc the race test is clean
across 5 runs and SIGSEGVs immediately when the join is turned back into a
detach, so the check is a real detector and not a vacuous pass.
Tests: 277/277 (was 270; 7 new). CallErrorAfterAcquireTest hammered 40x, 0
failures — it was ~2/50 flaky before this branch's earlier fixes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(protocol): a call that FINISHED must not park its thread for the object's life
The per-call waiters are joinable rather than detached, which is what closed the
use-after-free where release() deleted the object under a still-running waiter
(4f9d824), and they are interruptible, so teardown no longer waits out the call's
timeout (731e579). Both stay. What neither did was retire a waiter that had
FINISHED: m_waiters was only ever swap()ped, in stopAndJoinWaiters(), so an
exited-but-unjoined std::thread — whose stack and pthread struct are not
reclaimed until somebody joins it — stayed parked for the lifetime of the handle.
Measured against a live PlainTransportHost over TCP, every call completing
normally, one handle held throughout, before:
10000 calls m_waiters 300 -> 10300 rss +156.56 MiB 16417 B/call
30000 calls m_waiters 300 -> 30300 rss +469.28 MiB 16403 B/call
and the same through the production C ABI — one lp_client, N lp_invoke_async —
at +156.53 MiB. That path is why this matters: LogosAPIConsumer caches ONE
handle per module and reuses it for every async call, releasing it only on
eviction or teardown (cpp/logos_api_consumer.cpp:129 and :207), so a
long-lived module leaks per lp_invoke_async. The ~16KB constant is one page on
this 16KiB-page arm64 and will be smaller elsewhere; the UNBOUNDEDNESS is the
platform-independent part, and follows from m_waiters.size() rising 1:1 with
completed calls and only ever falling in teardown. Attribution: the retention
arrived with the join in 4f9d824, not with 731e579 — but 731e579 is what makes
the join permanent.
The registry is now KEYED, because a thread cannot join itself and so a waiter
can never retire its own entry. Each waiter publishes its id as its FINAL act (a
scope guard declared first, so it destructs last, covering all four exit paths),
and the next spawn — plus teardown — joins those ids and erases them. Joining a
thread that has already returned is a couple of syscalls. Same probe, same
workload, after:
10000 calls m_waiters 15 -> 16 rss +0.08 MiB 8 B/call
30000 calls m_waiters 16 -> 16 rss +0.06 MiB 2 B/call
10000 calls via lp_invoke_async rss +0.09 MiB 10 B/call
Retention is now bounded by the waiters that finish after the LAST spawn, i.e.
by peak in-flight concurrency — 16 at the in-flight window above, and exactly 1
when calls are issued sequentially — instead of by call count.
THE DEADLOCK THIS SHAPE INVITES is a reaper that joins while holding m_waiterMu,
against a waiter blocked on m_waiterMu trying to publish. It is avoided by
construction rather than by argument: nothing is joined with a lock held, in the
reaper or in teardown, whatever a waiter does on its way out. Proven by building
the naive variant that does join under the lock — the new hammer wedges it, with
the main thread in reapFinishedWaiters -> pthread_join and a waiter in
publishFinishedWaiter -> mutex wait, and the test's watchdog names the cause
instead of letting CI hang.
Teardown's guarantee is restated rather than weakened. It is not "every waiter
has been joined by the time stopAndJoinWaiters() returns" — a waiter a
concurrent reaper is mid-join on is no longer in the map — but the thing that
guarantee was ever for: NO WAITER TOUCHES THE OBJECT AFTER IT RETURNS. An entry
leaves m_waiters only once its thread has published, and publishing is that
thread's last access.
The TODO above the waiter still stands: the real fix is to fold the wait into
the shared Asio io_context and have no thread per pending RPC at all. This makes
the interim honest; it does not replace that.
Two more things review turned up, folded in here:
* The two wait sites resolved stop-vs-result in OPPOSITE directions.
waitForResult tested the stop flag BEFORE polling, so an already-ready
future was still reported as transport_error, while awaitCompletion
deliberately preferred a completion that had landed — and both were
commented as intentional. One rule now, applied to both: AN ANSWER ALREADY
IN HAND BEATS A CONCURRENT STOP, and the stop only decides what happens when
there is nothing to hand over. The callback fires either way
(postToQtEventLoop copies everything it delivers), so the only thing a stop
can change is what the callback SAYS — and manufacturing transport_error
while the true answer sits in the future reports a failure that did not
happen, to callers that re-acquire, retry and log on that code. Preferring
the answer costs nothing, since it is already there: the flag is still
checked before every sleep, so the teardown-latency bound is unchanged.
* CORRECTION to 731e579's message, which claimed it "closes the registration
window" where a call arriving after the stop would never be joined. That
branch is unreachable in defined behaviour: m_stopping is raised only by
teardown, so any thread that can read it inside callMethodAsyncWithError is
already calling a method on an object whose destructor is running — the load
is itself the use-after-free, reproduced as a SIGSEGV on that commit and on
its parent alike, and nothing inside that function can repair it. The guard
is harmless and stays (one predictable branch, and it fails safe with one
callback), but its comment now says what it is instead of claiming a fix it
does not make.
Verified by running, with every check first shown to FAIL on unfixed code:
* Retention: the probe above, plus a committed regression test that reads
m_waiters out of the live object through the explicit-instantiation access
hole ([temp.spec] does not check access on an explicit instantiation's
template arguments) — so the code under test keeps its private state, with
no friend, no test-only accessor and no `#define private public`. 200
sequential completed calls keep 1 waiter; without pruning they keep 200.
* Exactly-once on all four paths — normal completion, timeout, cancellation
and the deferred-completion (pending-sentinel) arm — counted PER CALL so a
dropped one and a doubled one cannot cancel out, plus the 60-round
release-during-call race. Shown to catch a cancelled path that returns
silently (3 failures) rather than delivering.
* Teardown latency unchanged from 731e579: 10-17ms with an in-flight 8000ms
call and 0-1ms mid-defer, against 15ms / 1ms on that commit.
* The UAF stays closed: 11 teardown + reaping tests clean under macOS Guard
Malloc (ASan/TSan remain unusable on this toolchain).
* Full suite 281/281 twice, `nix build .#tests` green (281/281 in the
sandbox), CallErrorAfterAcquireTest hammered 40x clean.
* fix(protocol): a burst that goes quiet must not wait for a call that never comes
378d889 retired finished waiters, but from ONE site: the async-call spawn path.
So whatever finishes after the LAST spawn is never reaped, and a module that
bursts and then goes idle parks it all until the handle dies. Measured on
378d889, one handle, 2000 concurrent calls, every one delivered:
after 2000 completed calls, IDLE: m_waiters=1428 rss=+24.17 MiB
after ONE further call: m_waiters=1 rss=+ 1.92 MiB
The unbounded-per-call class was gone; this is what it left behind, and the
second line is the whole diagnosis — the corpses go the instant anything calls
again, so the reaper works and simply never runs. LogosAPIConsumer caches one
handle per module and never releases it between calls, so "bursts, then quiet"
is not a corner case: it is a UI that fans out on a refresh and then waits for
the user.
A finishing waiter now reaps the OTHER finished waiters before publishing
itself, so a burst drains as it completes. Same probe, same workload:
after 2000 completed calls, IDLE: m_waiters=1 rss=+ 1.88 MiB
THE BOUND IS ONE, NOT ZERO, and by construction rather than by luck: a waiter
can only reap OTHERS (a thread cannot join itself), so the last one to finish
has nobody behind it to collect it. Anything that publishes after the final
reap survives too, which is why 12 runs of the probe gave 1 eleven times and 2
once. Those go on the next call, or in teardown. Retention now tracks neither
call count nor peak concurrency — the sequential and in-flight-16 numbers move
from "15 -> 16 waiters" to "1 -> 1" — and the memory figures are unchanged
against 378d889 where they were already flat: 10k sequential +0.00 MiB, 10k at
16 in flight +0.06 MiB, 30k +0.09 MiB, and 10k through the production C ABI
(one lp_client, N lp_invoke_async) +0.09 MiB / 10 B per call, the same as
378d889 reported.
THE ORDER IS THE SAFETY ARGUMENT. Reap first, publish last, never the reverse:
* Publishing is what makes a waiter joinable BY ANOTHER WAITER. Reaping first
keeps that relation one-way — unpublished threads join published ones,
published ones join nobody — so it has no cycles. Inverted, two waiters
publishing in the same instant can each take the other's thread out of
m_waiters and then join it; both are already out of the registry, so
teardown does not even wait for them. Built that variant: pthread_join
detects the cycle and throws, the half-drained thread vector then destroys
a still-joinable thread, and the process aborts — the EXISTING hammer
(ReapingRacesPublishingWithoutDeadlocking) catches it 5 runs out of 5, with
the stack showing two waiters inside FinishOnExit joining each other.
* While a waiter is unpublished it is still in m_waiters, so a concurrent
teardown joins it and the object cannot be destroyed under the reap. Once
published, a reaper may take its thread out of the map and release() may
`delete this` — and a reaper on the CALLER's thread (the spawn path) is one
teardown neither knows about nor waits for, so a post-publish touch of
m_waiterMu is a use-after-free on a member mutex. That path needs a caller
still issuing calls while another thread releases, which this class already
treats as caller-side UB, so it is stated as an argument; the cycle above is
what the tests actually demonstrate.
Two corrections to 378d889, which this change makes load-bearing rather than
cosmetic. NOT amended into it — it is pushed, and a commit that misstates its
own reasoning is better read alongside the correction than rewritten.
* plain_logos_object.h:107-109 said reapFinishedWaiters() is "called on every
async spawn ... and from stopAndJoinWaiters()". It is not, and never was,
called from stopAndJoinWaiters(): teardown does its own id-independent
brute-force join, which is precisely why it needs no cooperation from the
reaper. Harmless behaviourally, wrong in a mechanism whose entire argument
is who joins what and when. The comment now names the two real callers —
the spawn path and, as of this commit, every waiter on its way out.
* 378d889's message presented "the join is outside the lock" as THE property
that prevents the reaper deadlock, "proven by construction" by its hammer.
That is overstated, in a way that would let the guarantee be refactored
away with the suite still green. TWO independent properties each suffice:
joining only PUBLISHED ids (a published waiter never needs m_waiterMu
again, so it cannot be the thread being shut out), and joining outside the
lock. The hammer only wedges when BOTH are gone. Measured, on top of this
change: the variant that joins under the lock but KEEPS the published-only
filter passes ReapingRacesPublishingWithoutDeadlocking in 293/297/290ms
across three runs and the whole reaping suite besides, while the variant
that joins everything under the lock trips the watchdog at 60s. So a later
"simplification" that moves the join inside the lock would ship green. Both
properties are kept, and the comment now says which one the test is
actually testing.
The TODO above the waiter still stands: the real fix is to fold the wait into
the shared Asio io_context and have no thread per pending RPC at all. This
makes the interim honest; it does not replace it.
Verified by running, each check first shown to FAIL on unfixed code:
* Retention: the burst probe above, plus a committed regression test that
reads m_waiters out of the live object through the explicit-instantiation
access hole. 800 concurrent completed calls, then IDLE with NO further
call: 1 waiter left, 20 runs out of 20. On 378d889 the same test leaves
610 of 800 and fails. The pre-existing sequential and in-flight tests are
unchanged and still pass.
* The UAF stays closed — the check that matters most here, because this adds
an object access late in the waiter's life. 9 reaping/teardown-race tests
plus the 7-test teardown suite clean under macOS Guard Malloc (ASan is
unusable on this box: it hangs in its own initializer). DETECTOR VALIDATED
both ways: turning teardown's join back into a detach SIGSEGVs under Guard
Malloc on the release-during-call hammer (exit 139), and the specific
inversion this change risks — reaping AFTER publishing — aborts as
described above.
* No deadlock: reap-vs-publish hammered 20x (1600 calls in 40 overlapping
bursts each), plus 60 rounds of teardown landing from another thread while
the tail of a burst retires itself, plus 6x600-call bursts checking that
LIVE OS threads (task_threads, which counts wedges and not corpses) come
back to baseline every round. Clean; the watchdog names the cause if it
ever is not.
* Exactly-once on all four paths — normal, timeout, cancellation, deferred
sentinel — counted per call. Each detector validated with a broken build:
dropping the cancelled callback fails 4 tests, dropping the timeout one
fails its test, and double-delivering the normal/deferred arm fails those.
* Teardown latency unchanged from 378d889: 1-25ms with an in-flight 8000ms
call and 0ms mid-defer across 5 runs, against 2-21ms / 0ms on that commit —
the same one-wait-slice (25ms) bound, since a waiter's extra work happens
after it has stopped waiting.
* Full suite 282/282 three times, `nix build .#tests` green,
CallErrorAfterAcquireTest hammered 40x clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(protocol): pin publishing as a waiter's LAST access to the object
PlainLogosObject's waiters are joinable, interruptible and reaped, and all
three rest on one ordering rule that nothing in the suite could see:
~FinishOnExit() {
self->reapFinishedWaiters(); // others, never itself
self->publishFinishedWaiter(id); // strictly last
}
reapFinishedWaiters() erases published entries from m_waiters under m_waiterMu
and joins those threads OUTSIDE it. stopAndJoinWaiters() swaps m_waiters under
the same lock and brute-force joins whatever it got. So a waiter that a
concurrent reaper is mid-join on is NOT in teardown's map, and teardown can
return — with release() going straight on to `delete this` — while that waiter
is still unwinding. stopAndJoinWaiters() already says this in as many words:
the guarantee is not "everything is joined when this returns" but "no waiter
touches this object after this returns". Publishing being last is the entire
reason the second sentence is true, so one member access below it is a
use-after-free, and moving the publish above the reap is a join cycle.
THE DEFECT SHIPS GREEN. Rebuild plain_logos_object.cpp with a single object
read after the publish and the whole of PlainObjectTeardownTest and
PlainWaiterReapingTest passes, 10 runs out of 10, cleanly under Guard Malloc.
That is not a hole in those suites. No SUPPORTED caller can provoke it: under
calls-in-flight-plus-release, every waiter is still joined transitively,
because a waiter leaves m_waiters only via teardown (which joins it) or a
reaper, and a reaper is either another waiter — itself in m_waiters until after
its join returns — or the async-spawn path, whose join completes before the
call returns. The one uncovered reaper is the spawn path racing a concurrent
release(), and calling a method on an object another thread is releasing is
caller-side UB that faults on correct code too. A test built on that race would
be red on green code, so it is not a usable detector.
SO STOP RACING AND OBSERVE. tests/protocol/test_plain_waiter_publish_is_last.cpp
drives a real PlainLogosObject through a scripted RpcConnectionBase — no socket,
no host, no event-loop timing, and the test decides exactly when the call's
future is satisfied — and watches the accesses in two halves.
* THE STATE. The object is placement-newed into an mmap'd two-page arena, put
down so a page boundary lands at m_waiterMu: the members teardown
coordinates on go on the second page, everything else on the first. The
first page is mprotect(PROT_NONE)'d for exactly as long as a waiter runs,
and a SIGSEGV/SIGBUS handler RECORDS each access — address, thread, and how
many ids were published at that instant — then unprotects so the access
proceeds. Nothing crashes; the access is evidence. A correct waiter touches
that page zero times: objectName and method are copied into the closure
precisely so it needs nothing from the object. Four rounds, one per exit
path out of the lambda (answered, rejected, timed out, cancelled), since all
four end in the same guard.
* THE REGISTRY, which that page cannot cover because publishing has to reach
it. Caught with bait, using the reaper's own shape: reapFinishedWaiters()
joins outside m_waiterMu, so a waiter that has picked up somebody else's
finished thread sits in that join holding nothing — a window the test holds
open as long as it likes, because the thread being joined is one the test
planted and keeps parked. Plant bait 1; let the call finish; the exit guard
reaps, takes it, parks. Plant bait 2 at leisure. Release bait 1; the waiter
finishes its reap and publishes. Bait 2 must still be registered. Bait 1
doubles as a check that the reap really does join with the lock free.
Neither half is probabilistic. A third test proves the detector can fire at all,
so the two "this counter stayed at zero" assertions are not vacuous.
MEASURED, rebuilding the file under test with each defect (caught/runs):
defect below publishFinishedWaiter() new teardown+reaping
------------------------------------ --- ----------------
read m_objectName 40/40 0/10
read m_conn 10/10 0/10
read m_completions 10/10 0/10
read m_completionSubscribed 10/10 0/10
lock m_mu 10/10 0/10
call reapFinishedWaiters() again 20/20 2/2
read m_stopping 0/10 0/10
(publish moved ABOVE the reap) 0/5 12/15
no defect — 8f0c60f 0/40 0/10
The one gap is m_stopping, the single member sharing the registry's page, which
cannot be guarded without guarding the publish. The inverted order is left to
the reaping suite's hammer, which has it covered. Runtime 0.9-1.0s for all
three tests; clean 40/40 on 8f0c60f, and clean 3/3 under Guard Malloc
(MALLOC_PROTECT_BEFORE=1, banner confirmed) — the test never touches freed
memory itself, which is the other half of not being built on UB. No Guard
Malloc needed to detect anything: mprotect and the bait are the detectors.
Also: nix build .#tests 100% (285/285), the full binary 285/285, and
CallErrorAfterAcquireTest 40/40.
CORRECTIONS to measurements claimed earlier on this branch. All three were
overstated in the same direction — a single sample read as a constant:
* "ReapingRacesPublishingWithoutDeadlocking aborts the process, 5 runs out of
5" (plain_logos_object.cpp, and 378d889's message) is 12 runs in 15, ~80%.
It is a race detector, so one green run of it proves nothing — which is
exactly the argument for the deterministic suite added here. Corrected in
the comment.
* C-ABI retention was reported as "+0.09 MiB / 10 B per call" for 10k
lp_invoke_async on one client (8f0c60f's message). ~6 B/call. Same
conclusion — flat — different arithmetic.
* The burst retention figures 1428 (2000 calls, idle) and 610 of 800 came
back as 1421 and 599 on re-measure of the same build. Race-dependent, same
magnitude, which is why the tests assert a bound and not a value. Noted in
test_plain_waiter_reaping.cpp so the next reader does not treat them as
reproducible constants.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(protocol): make the publish-is-last probe fail loudly, and state the rule it actually checks
Three defects in cb015f5's regression test, plus the corrections that commit's
own CORRECTIONS section still owed. No behaviour change: the diff against
8f0c60f under cpp/ is comment lines only, verified by filtering the diff.
(a) THE BAIL-OUT PATH HUNG INSTEAD OF FAILING, which is the one that can stall
CI. PublishedWaiterDoesNotTouchTheRegistryAgain plants parked "bait" threads
behind gates and registers them in m_waiters. An ASSERT that fires before
gate1.open() — ASSERT_TRUE(tookBait1) is the obvious one — returns from the
function, and then ~PlainLogosObject blocks forever joining a thread nobody
will release. Reproduced by removing the reap from the exit guard: the
assertion PRINTS and the run still ends as a timeout kill, exit 124, with no
test result at all.
The gates are now opened by a scope guard on every exit path, and declared
BEFORE the GuardedObject so they outlive the teardown that joins the threads
parked on them. Same break, after: the same assertion, exit 1, 10.0s — which
is the probe's own tryWithRegistry budget and not a hang.
This is the shape a future refactor trips, not a hypothetical: the TODO
above the waiter (fold the wait into the shared Asio io_context) moves where
reaping happens, which is exactly the edit that makes tookBait1 false.
Both tests also stopped capturing their delivery counter by reference. On a
bail-out the cancelled call's callback is delivered on a later event-loop
iteration, i.e. after the frame is gone — a real use-after-free on the way
out of a failing test in a file about use-after-free. Owned by the callback
now.
(b) THE STATE ASSERTION WAS STRICTER THAN THE INVARIANT. It asserted
accessCount() == 0; the rule is only "no access AFTER the publish", and the
fault handler already stamps each access with how many ids were published at
that instant, so it can tell them apart.
Not hypothetical either. On the deferred/"multi" path a CORRECT waiter calls
awaitCompletion() (plain_logos_object.cpp:338), which locks m_completionMu
and reads m_completions and m_objectName — all on the guarded page, all
before it publishes. cb015f5 was green only because none of its four rounds
returned a pending sentinel, and the header's claim that "a correct waiter
touches that page ZERO times, before the publish or after" was true only of
the non-deferred rounds.
So: a fifth round drives the pending-sentinel path (ScriptedConn now answers
with the sentinel; nothing pushes the completion, so awaitCompletion runs out
its deadline), and the assertion narrowed to accesses stamped published >= 1.
PROVEN BOTH WAYS on this tree — with the old accessCount() == 0 predicate the
new round fails on correct code, naming offset 144 with "0 waiter id(s)
already published"; with the narrowed one the suite is 30/30 clean.
The round cannot pass vacuously: it REQUIRES at least one recorded access, so
a machine slow enough to turn it into a plain timeout fails it instead of
quietly proving nothing. Each round also now asserts m_finishedWaiters is
empty before arming, which is what makes "published >= 1" mean "after THIS
waiter's publish".
Two things guard against the narrowing being a quiet disarm:
* the handler now re-arms. It could not before (the faulting instruction
re-runs immediately), so the observing thread does it — it polls the
registry anyway and never touches the guarded page. Without it the first
legitimate access disarms the detector for the whole round.
* the catch rates were re-measured, not assumed. They are unchanged.
(c) TWO OVERSTATED NUMBERS, in the section whose whole point was to stop
overstating. Fixed where they live; cb015f5 is pushed and is not rewritten.
* "C-ABI retention ~6 B/call, not 10" replaced one sample with another.
10k lp_invoke_async on one lp_client, run ten times: 0, 5, 5, 5, 7, 7, 8,
10, 13, 10 bytes/call (mean 7.0). Ten more, run here: 3, 11, 8, 5, 8, 10,
13, 8, 3, 10 (mean 7.9). One distribution, range 0-13; both 6 and 10 sit
inside it and 8f0c60f's arithmetic (0.09 MiB / 10k) was not wrong.
THE HONEST STATEMENT IS THAT IT IS FLAT: indistinguishable from zero, RSS
noise and not a per-call rate. Recorded in test_plain_waiter_reaping.cpp
beside the other retention figures, where the next person to quote one
will see it.
* the table cell "reapFinishedWaiters() again ... 2/2" for the older
teardown+reaping suites was a two-run sample printed beside 10-40 run
samples. Re-measured over 30 runs: 9/30 here, 12/30 on another 30-run
sample — roughly one run in three, matching what reapFinishedWaiters'
own comment already said ("about one run in four"). The cell now reads
9/30, and the table says to read that column as rates and the left-hand
one as deterministic.
ALSO STATED PLAINLY, because it was overstated in review: the window where
"teardown returns while a reaped waiter is still unwinding" is NOT reachable by
a supported caller. A waiter leaves m_waiters only via teardown (which joins it)
or via a reaper, and that reaper is either another waiter — still registered
itself, since it reaps before it publishes, so teardown joins it and therefore
waits out the join it is in — or the async-spawn path, whose join completes
before the call returns. The only uncovered reaper is the spawn path racing a
concurrent release(), which is caller-side UB on any version of this class.
So publish-is-last is an invariant the design rests on and documents, not a
lurking use-after-free. This suite pins it against future edits; it does not
close an open hole. The file header, both failure messages and the comment in
plain_logos_object.cpp now say that instead of implying otherwise.
MEASURED AFTER THE CHANGE, rebuilding plain_logos_object.cpp with each defect
below publishFinishedWaiter() and running the suite (caught/runs), beside the
numbers from before it:
defect before after
------ ------ -----
read m_objectName 25/25 25/25
lock m_mu 25/25 25/25
write m_completions under m_completionMu 25/25 25/25
call reapFinishedWaiters() again (bait) 20/20 20/20
read m_conn - 10/10
read m_completions - 10/10
read m_completionSubscribed - 10/10
read m_stopping (declared blind spot) 0/10 0/10
publish moved ABOVE the reap (delegated) 0/5 0/5
no defect 0/30 0/30
Nothing moved, including the two declared blind spots — a narrowing that had
started catching or stopped catching something would show here. cb015f5 reported
40/40 for m_objectName from a longer run; 25/25 is this run, not a regression.
WHERE THE DEFERRED ROUND IS WEAKER, said here rather than left to be found: on
that one round the post-publish half is best-effort. A legitimate access opens
the page, the re-arm is a syscall behind, and a defect firing a microsecond
later slips through — measured with every round forced to run, the other four
catch a post-publish m_objectName read 5/5 and the deferred round 0/5, and a
variant that spins on the re-arm instead of polling records 4-24 accesses per
round and still catches it 0/5. It costs nothing: FinishOnExit is ONE piece of
code shared by all five exit paths, so the same defect is the same defect on
every round and the other four catch it deterministically. The deferred round is
there to keep the assertion honest about correct code, not to add a fifth copy
of the same detection.
Verified: PlainWaiterPublishIsLastTest 30/30 clean, the three waiter suites
15/15, the full binary 285/285, `nix build .#tests` 100% (285/285),
CallErrorAfterAcquireTest 40/40. Suite runtime 1.3-1.5s for the three tests
(0.9-1.0s before — the deferred round waits out a 400ms completion deadline).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>