logos-co/logos-protocol
1
0
Fork 0
mirror of https://github.com/logos-co/logos-protocol.git synced 2026-08-30 21:41:10 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(protocol): a call that FINISHED must not park its thread for the object's life

Browse Source 
The per-call waiters are joinable rather than detached, which is what closed the
use-after-free where release() deleted the object under a still-running waiter
(4f9d824), and they are interruptible, so teardown no longer waits out the call's
timeout (731e579). Both stay. What neither did was retire a waiter that had
FINISHED: m_waiters was only ever swap()ped, in stopAndJoinWaiters(), so an
exited-but-unjoined std::thread — whose stack and pthread struct are not
reclaimed until somebody joins it — stayed parked for the lifetime of the handle.

Measured against a live PlainTransportHost over TCP, every call completing
normally, one handle held throughout, before:

    10000 calls   m_waiters   300 -> 10300   rss +156.56 MiB   16417 B/call
    30000 calls   m_waiters   300 -> 30300   rss +469.28 MiB   16403 B/call

and the same through the production C ABI — one lp_client, N lp_invoke_async —
at +156.53 MiB. That path is why this matters: LogosAPIConsumer caches ONE
handle per module and reuses it for every async call, releasing it only on
eviction or teardown (cpp/logos_api_consumer.cpp:129 and :207), so a
long-lived module leaks per lp_invoke_async. The ~16KB constant is one page on
this 16KiB-page arm64 and will be smaller elsewhere; the UNBOUNDEDNESS is the
platform-independent part, and follows from m_waiters.size() rising 1:1 with
completed calls and only ever falling in teardown. Attribution: the retention
arrived with the join in 4f9d824, not with 731e579 — but 731e579 is what makes
the join permanent.

The registry is now KEYED, because a thread cannot join itself and so a waiter
can never retire its own entry. Each waiter publishes its id as its FINAL act (a
scope guard declared first, so it destructs last, covering all four exit paths),
and the next spawn — plus teardown — joins those ids and erases them. Joining a
thread that has already returned is a couple of syscalls. Same probe, same
workload, after:

    10000 calls   m_waiters    15 -> 16      rss +0.08 MiB         8 B/call
    30000 calls   m_waiters    16 -> 16      rss +0.06 MiB         2 B/call
    10000 calls via lp_invoke_async          rss +0.09 MiB        10 B/call

Retention is now bounded by the waiters that finish after the LAST spawn, i.e.
by peak in-flight concurrency — 16 at the in-flight window above, and exactly 1
when calls are issued sequentially — instead of by call count.

THE DEADLOCK THIS SHAPE INVITES is a reaper that joins while holding m_waiterMu,
against a waiter blocked on m_waiterMu trying to publish. It is avoided by
construction rather than by argument: nothing is joined with a lock held, in the
reaper or in teardown, whatever a waiter does on its way out. Proven by building
the naive variant that does join under the lock — the new hammer wedges it, with
the main thread in reapFinishedWaiters -> pthread_join and a waiter in
publishFinishedWaiter -> mutex wait, and the test's watchdog names the cause
instead of letting CI hang.

Teardown's guarantee is restated rather than weakened. It is not "every waiter
has been joined by the time stopAndJoinWaiters() returns" — a waiter a
concurrent reaper is mid-join on is no longer in the map — but the thing that
guarantee was ever for: NO WAITER TOUCHES THE OBJECT AFTER IT RETURNS. An entry
leaves m_waiters only once its thread has published, and publishing is that
thread's last access.

The TODO above the waiter still stands: the real fix is to fold the wait into
the shared Asio io_context and have no thread per pending RPC at all. This makes
the interim honest; it does not replace that.

Two more things review turned up, folded in here:

  * The two wait sites resolved stop-vs-result in OPPOSITE directions.
    waitForResult tested the stop flag BEFORE polling, so an already-ready
    future was still reported as transport_error, while awaitCompletion
    deliberately preferred a completion that had landed — and both were
    commented as intentional. One rule now, applied to both: AN ANSWER ALREADY
    IN HAND BEATS A CONCURRENT STOP, and the stop only decides what happens when
    there is nothing to hand over. The callback fires either way
    (postToQtEventLoop copies everything it delivers), so the only thing a stop
    can change is what the callback SAYS — and manufacturing transport_error
    while the true answer sits in the future reports a failure that did not
    happen, to callers that re-acquire, retry and log on that code. Preferring
    the answer costs nothing, since it is already there: the flag is still
    checked before every sleep, so the teardown-latency bound is unchanged.

  * CORRECTION to 731e579's message, which claimed it "closes the registration
    window" where a call arriving after the stop would never be joined. That
    branch is unreachable in defined behaviour: m_stopping is raised only by
    teardown, so any thread that can read it inside callMethodAsyncWithError is
    already calling a method on an object whose destructor is running — the load
    is itself the use-after-free, reproduced as a SIGSEGV on that commit and on
    its parent alike, and nothing inside that function can repair it. The guard
    is harmless and stays (one predictable branch, and it fails safe with one
    callback), but its comment now says what it is instead of claiming a fix it
    does not make.

Verified by running, with every check first shown to FAIL on unfixed code:

  * Retention: the probe above, plus a committed regression test that reads
    m_waiters out of the live object through the explicit-instantiation access
    hole ([temp.spec] does not check access on an explicit instantiation's
    template arguments) — so the code under test keeps its private state, with
    no friend, no test-only accessor and no `#define private public`. 200
    sequential completed calls keep 1 waiter; without pruning they keep 200.
  * Exactly-once on all four paths — normal completion, timeout, cancellation
    and the deferred-completion (pending-sentinel) arm — counted PER CALL so a
    dropped one and a doubled one cannot cancel out, plus the 60-round
    release-during-call race. Shown to catch a cancelled path that returns
    silently (3 failures) rather than delivering.
  * Teardown latency unchanged from 731e579: 10-17ms with an in-flight 8000ms
    call and 0-1ms mid-defer, against 15ms / 1ms on that commit.
  * The UAF stays closed: 11 teardown + reaping tests clean under macOS Guard
    Malloc (ASan/TSan remain unusable on this toolchain).
  * Full suite 281/281 twice, `nix build .#tests` green (281/281 in the
    sandbox), CallErrorAfterAcquireTest hammered 40x clean.
This commit is contained in:
Dario Gabriel Lipicar
2026-08-05 20:24:11 -03:00
parent 731e579064
commit 378d889a1d
4 changed files with 778 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cpp/implementations/plain/plain_logos_object.cpp
+151 -35
View File
@@ -44,6 +44,25 @@ constexpr std::chrono::milliseconds kWaitSlice(25);
enum class WaitOutcome { Ready, TimedOut, Cancelled };
// ── the one rule both waits follow ──────────────────────────────────────────
//
// AN ANSWER ALREADY IN HAND BEATS A CONCURRENT STOP. The stop only decides what
// happens when there is nothing to hand over.
//
// The two sites used to resolve this in opposite directions — this one tested
// the flag before polling, so an already-ready future was still reported as
// transport_error, while awaitCompletion deliberately preferred a completion
// that had landed. Both were commented as deliberate, and they cannot both be
// right, so: the callback fires either way (postToQtEventLoop copies everything
// it delivers, precisely so a released handle costs it nothing), which means the
// only thing a stop can change is what the callback SAYS. Reporting
// transport_error while the true answer sits in the future is a failure that did
// not happen, and that code is not inert — callers re-acquire, retry and log on
// it. Preferring the answer is also free: it is already there, so nothing waits
// for it. The teardown-latency bound is untouched, because the flag is still
// checked before every sleep, and a stop with no answer in hand still wins
// immediately.
//
// The interruptible form of `fut.wait_for(milliseconds(timeoutMs))`.
//
// The overall deadline is computed once, so slicing does not stretch the
@@ -54,27 +73,23 @@ WaitOutcome waitForResult(std::future<ResultMessage>& fut, int timeoutMs,
using clock = std::chrono::steady_clock;
const auto deadline = clock::now() + std::chrono::milliseconds(timeoutMs);
for (;;) {
// Checked BEFORE sleeping, so a stop that already happened costs
// Poll FIRST, with a zero wait: an answer in hand beats both a
// concurrent stop and the deadline. This is also what makes a
// non-positive timeout behave as the single unsliced wait_for did —
// one poll, then give up — and what reports a future that went ready
// during the last slice.
if (fut.wait_for(clock::duration::zero()) == std::future_status::ready)
return WaitOutcome::Ready;
// Checked before sleeping, so a stop that already happened costs
// nothing, and after every slice, so one that arrives mid-wait costs at
// most kWaitSlice. A pending stop beats a result that landed in the
// same tick on purpose: the caller has released the handle and is no
// longer interested in the answer.
// most kWaitSlice.
if (stopping.load(std::memory_order_acquire))
return WaitOutcome::Cancelled;
const auto remaining = deadline - clock::now();
const bool expired = remaining <= clock::duration::zero();
// At (or past) the deadline, poll once with a zero wait rather than
// giving up blind — that is what the single unsliced wait_for did for a
// non-positive timeout, and a future that is already ready must still
// be reported as ready.
const auto slice = expired
? clock::duration::zero()
: std::min<clock::duration>(kWaitSlice, remaining);
if (fut.wait_for(slice) == std::future_status::ready)
return WaitOutcome::Ready;
if (expired)
if (remaining <= clock::duration::zero())
return WaitOutcome::TimedOut;
fut.wait_for(std::min<clock::duration>(kWaitSlice, remaining));
}
}
@@ -129,22 +144,87 @@ void PlainLogosObject::stopWaiters()
m_completionCv.notify_all();
}
void PlainLogosObject::publishFinishedWaiter(std::uint64_t id)
{
std::lock_guard<std::mutex> g(m_waiterMu);
m_finishedWaiters.push_back(id);
}
void PlainLogosObject::reapFinishedWaiters()
{
// Only ids a waiter published are taken, and publishing is that waiter's
// last act — so everything moved into `done` has already stopped touching
// this object, and joining it is effectively instant.
std::vector<std::thread> done;
{
std::lock_guard<std::mutex> g(m_waiterMu);
std::vector<std::uint64_t> keep;
for (const std::uint64_t id : m_finishedWaiters) {
const auto it = m_waiters.find(id);
if (it == m_waiters.end())
continue; // teardown already took this one
if (it->second.get_id() == std::this_thread::get_id()) {
// Unreachable today — callbacks are delivered on the Qt event
// loop, so a waiter thread never re-enters this class — but a
// thread that joined itself would terminate the process, and
// this is one comparison. Leave it registered; teardown, which
// runs on somebody else's thread, will collect it.
keep.push_back(id);
continue;
}
done.push_back(std::move(it->second));
m_waiters.erase(it);
}
m_finishedWaiters.swap(keep);
}
// Joined with NO lock held. Not just hygiene — this is THE deadlock this
// whole mechanism can introduce: a reaper holding m_waiterMu while it joins
// a waiter that is itself blocked on m_waiterMu trying to publish would
// wedge the process. Holding no lock across a join makes that impossible by
// construction rather than by argument, whatever a waiter does on its way
// out. stopAndJoinWaiters() keeps the same discipline for the same reason.
for (auto& t : done) {
if (t.joinable())
t.join();
}
}
void PlainLogosObject::stopAndJoinWaiters()
{
stopWaiters();
std::vector<std::thread> waiters;
std::map<std::uint64_t, std::thread> waiters;
{
std::lock_guard<std::mutex> g(m_waiterMu);
waiters.swap(m_waiters);
}
// Joined with NO lock held: a waiter on its way out still takes
// m_completionMu (awaitCompletion) and m_waiterMu is what a concurrent
// callMethodAsyncWithError needs to see the stop flag.
for (auto& t : waiters) {
// m_completionMu (awaitCompletion) and then m_waiterMu (to publish), and
// m_waiterMu is also what a concurrent callMethodAsyncWithError needs in
// order to see the stop flag.
//
// Everything outstanding is joined by id-independent brute force, so this
// needs no cooperation from the reaper: a waiter that publishes while this
// loop runs simply leaves a stale id behind, and its thread is joined here
// anyway.
//
// A waiter that a concurrent reaper is in the middle of joining is NOT in
// this map, and that is still safe. The invariant is not "every waiter has
// been joined by the time this returns" but the thing that invariant was
// ever for: NO WAITER TOUCHES THIS OBJECT AFTER THIS RETURNS. An entry
// leaves m_waiters only once its thread has published, and publishing is
// that thread's last access — all it has left to do is unwind.
for (auto& entry : waiters) {
std::thread& t = entry.second;
if (t.joinable())
t.join();
}
// Cleared after the joins, so the stale ids just described go too. Nothing
// can be added afterwards: m_stopping is set, so no new waiter registers.
{
std::lock_guard<std::mutex> g(m_waiterMu);
m_finishedWaiters.clear();
}
}
QVariant PlainLogosObject::callMethod(const QString& authToken,
@@ -252,8 +332,9 @@ QVariant PlainLogosObject::awaitCompletion(const QString& callId, int timeoutMs,
|| m_stopping.load(std::memory_order_relaxed);
});
// A completion that actually landed beats a concurrent stop: there is a real
// answer in hand, so hand it over rather than manufacture an error.
// A completion that actually landed beats a concurrent stop — the same rule
// the future wait follows (see waitForResult): there is a real answer in
// hand, so hand it over rather than manufacture an error.
const auto it = m_completions.find(callId);
if (it != m_completions.end()) {
const QVariant result = it->second;
@@ -359,33 +440,66 @@ void PlainLogosObject::callMethodAsyncWithError(const QString& authToken,
// io_context (the connection already runs on it) so we don't spin
// up a thread per pending RPC.
//
// The thread is JOINed in stopAndJoinWaiters() (destructor / release), not
// detached: capturing `this` for awaitCompletion / m_stopping is
// only safe while the object is alive, and release() used to
// The thread is JOINed — in reapFinishedWaiters() once it has finished, or
// in stopAndJoinWaiters() (destructor / release) if teardown gets there
// first — never detached: capturing `this` for awaitCompletion /
// m_stopping is only safe while the object is alive, and release() used to
// `delete this` while a waiter could still be mid-flight.
const std::string objectName = m_objectName;
const std::string method = methodName.toStdString();
// Retire the previous calls' waiters before adding one. Done HERE rather
// than by the waiters themselves because a thread cannot join itself; done
// BEFORE taking m_waiterMu because it joins, and joining under that lock is
// the deadlock described in reapFinishedWaiters().
reapFinishedWaiters();
// Register under the lock BEFORE the thread can outrun release(): a
// detach-then-push left a window where delete this raced the waiter.
{
std::lock_guard<std::mutex> g(m_waiterMu);
if (m_stopping.load(std::memory_order_acquire)) {
// Teardown has already swapped m_waiters out, so a thread pushed
// now would never be joined — exactly the dangling waiter this
// whole mechanism exists to prevent. Answer as a cancelled call
// instead, which keeps the exactly-once contract either way.
// Refuse rather than register: teardown has already swapped
// m_waiters out, so a thread pushed now would never be joined.
//
// To be honest about what this branch is: it is NOT a reachable
// window that got closed. m_stopping is raised only by teardown
// (release() / the destructor), so a thread that can read it as
// true here is already calling a method on an object whose
// destructor is running — this very load is the use-after-free, and
// nothing inside this function can repair that. Reproduced as a
// SIGSEGV, on this branch and on its parent alike. It is kept
// because it costs one predictable branch on a path that already
// does a socket write, and because failing this way — one callback,
// with the same error a cancelled call gets — is strictly better
// than pushing a thread nobody will ever join, should some future
// caller of stopWaiters() make the state legitimately observable.
postToQtEventLoop(std::move(callback), QVariant(),
callErrorReleased(objectName, method));
return;
}
m_waiters.emplace_back([this, objectName, fut, timeoutMs, methodName, method,
callback = std::move(callback)]() mutable {
const std::uint64_t waiterId = m_nextWaiterId++;
std::thread waiter([this, waiterId, objectName, fut, timeoutMs, methodName, method,
callback = std::move(callback)]() mutable {
// Everything reached through `this` below (m_stopping,
// awaitCompletion's m_completionMu / m_completions) is safe only
// because stopAndJoinWaiters() joins this thread before the object
// dies. Everything handed to postToQtEventLoop is a COPY, because
// that delivery happens after this thread has returned — i.e.
// possibly after the object is gone. Keep it that way.
// because this thread is joined before the object dies — by the
// reaper if it finishes first, by stopAndJoinWaiters() otherwise.
// Everything handed to postToQtEventLoop is a COPY, because that
// delivery happens after this thread has returned — i.e. possibly
// after the object is gone. Keep it that way.
//
// Declared FIRST so it destructs LAST: publishing this waiter's id
// is what permits somebody else to join and drop it, so it must
// come after every access to the object, on every exit path
// (four returns below, plus anything that throws). What runs after
// it is the unwinding of the captures above, none of which belongs
// to the object: a string, a shared_ptr to the call's future, and a
// callback that has already been moved out.
struct PublishOnExit {
PlainLogosObject* self;
std::uint64_t id;
~PublishOnExit() { self->publishFinishedWaiter(id); }
} publishOnExit{this, waiterId};
const WaitOutcome outcome = waitForResult(*fut, timeoutMs, m_stopping);
if (outcome == WaitOutcome::Cancelled) {
// A cancelled call still DELIVERS, exactly once. Returning
@@ -425,6 +539,7 @@ void PlainLogosObject::callMethodAsyncWithError(const QString& authToken,
}
postToQtEventLoop(std::move(callback), std::move(value), std::move(err));
});
m_waiters.emplace(waiterId, std::move(waiter));
}
}
@@ -518,7 +633,8 @@ void PlainLogosObject::release()
// object under a still-running waiter — and merely joining them made
// release() block for the rest of the call's timeout, so they are asked to
// stop first. Each abandoned call still delivers its callback, once, with
// callErrorReleased.
// callErrorReleased. Waiters that already finished were reaped as the
// calls after them were issued; this collects whatever is left.
disconnectEvents();
stopAndJoinWaiters();
m_conn.reset();