logos-co/logos-protocol
1
0
Fork 0
mirror of https://github.com/logos-co/logos-protocol.git synced 2026-08-27 12:01:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

perf(plain): fold the per-call waiter thread into a call object with its own clock (#46)

Browse Source 
* perf(plain): fold the per-call waiter thread into a call object with its own clock

An async call on the plain transport used to be an OS thread whose entire job
was to be blockable: std::future cannot be waited on with a deadline AND a
cancel, so the waiter polled it in 25ms slices, parked on a condition variable
for the deferred half, and delivered. Three costs came with that — one thread
per pending RPC, a 25ms floor on teardown, and a registry-plus-reaping protocol
to stop finished threads accumulating, because a thread cannot join itself. The
TODO in callMethodAsyncWithError has said to fold it away since it was written.

A call is now a shared_ptr<AsyncCall>: state that the reply (delivered as a
handler rather than parked in a promise), a deadline, and cancellation race to
finish. Nothing captures `this`. Handlers hold a shared_ptr to their AsyncCall
and a weak_ptr to CallState, so "no handler touches a destroyed object" is true
by construction rather than by a barrier, and the join is replaced by ownership.
postToQtEventLoop is kept verbatim as the re-entrancy firebreak: all four
completion sites route through it, so no user callback ever runs on an Asio
stack.

Measured against pristine 0f26ffd, same probe compiled into both:

  * 32 calls parked in a provider that will not answer: threads 5 -> 37 on
    master, 6 -> 6 here (the 6th is the deadline clock below, not per-call);
  * release() with those 32 in flight: 26-29ms -> 0ms;
  * 24,000 calls resolved by their deadline and never answered: 8.5MB of
    resident memory on master, 353 bytes per call growing strictly linearly,
    against 0.95MB here that has stopped growing by 3,000 calls.

THE DEADLINE GETS ITS OWN THREAD, and that is the one decision worth arguing
with. Hanging the per-call timer off the connection's strand is the obvious
move and it has a real regression: IoContextPool runs ONE thread for the whole
process and this transport delivers user onEvent callbacks INLINE on it, so an
event handler that calls another module — ordinary module code — holds every
deadline in the process. Measured on that design: an onEvent handler making a
2000ms call delayed a 200ms deadline on a DIFFERENT connection to 2003ms, and a
handler that never returns meant the deadline never fired at all. A second io
thread does not fix it (user handlers are unbounded, so N blocked handlers need
N+1 threads); moving inline event delivery off the strand is a much larger
change to event ordering for every consumer; a Qt timer is strictly worse,
because a synchronous call from the Qt thread blocks that loop too. So:
DeadlineService, one thread process-wide, doing nothing but arming, cancelling
and firing timers. Both shapes are back to 200ms and 250ms, and are pinned.

Two things closed on the way, neither of them inherited:

  * RpcConnection::cancelPending(). m_pendingCalls was emptied only by a decoded
    reply and by fail()'s sweep, so a call resolved by its DEADLINE left its
    registration there for the life of the connection — which outlives every
    handle it hands out. That was true of the promise before this change too;
    the fold would have made the orphan bigger, so it is closed rather than
    passed on. The sync call path and getMethods withdraw theirs as well.
  * A sentinel that arrives after its own deadline used to be filed under
    CallState::deferred by a reply handler that had not noticed the call was
    already resolved. deliver() therefore leaves the registries BEFORE the
    exactly-once gate, not after it.

TESTS. tests/protocol/test_iofold.cpp is the evidence, and its detectors are
validated by an explicit inverted build rather than asserted:

  cmake -S tests -B build-broken -DLOGOS_PROTOCOL_DETECTOR_INVERSIONS=ON

which removes the exactly-once CAS and puts the deadline back on the shared
io_context. In that build the deadline tests fail at 2002ms and never-fires
respectively, and the release-vs-replies race reports 6-16 double deliveries per
10,000 calls. Finding a race wide enough to be a RELIABLE exactly-once detector
took three attempts and the two rejected candidates are documented in the file:
the plain outcomes are weak detectors (one resolver, one deliver()), and
release-against-a-single-completion caught one double in seven runs. Teardown
against a burst of arriving replies is the one that works, because teardown
snapshots the whole in-flight map and then delivers with the lock released.

test_plain_waiter_publish_is_last.cpp is DELETED. It pinned exactly one rule —
publishFinishedWaiter() is a waiter thread's last access to the object, which
was the only reason stopAndJoinWaiters() could return while a reaper was still
mid-join. There are no waiter threads, no reaper and no publish list, so there
is no ordering left to pin; the property it protected is now structural.
test_plain_waiter_reaping.cpp is retargeted at CallState::inflight, keeping its
claims and losing its mechanism.

The four guarantees from #41 all re-measured on this branch: no thread growth
with in-flight calls, teardown 0ms with 32 outstanding, exactly one callback on
normal/deferred/timeout/cancellation counted per call over 10,000 calls
including a release race, and retention final 0 on both registries. Guard Malloc
clean over the lifetime suites (27 tests), with the completion-subscription
detector still faulting 3/3 on pristine master. ctest 296/296 (287 before, minus
3 deleted, plus 12); nix build .#tests 296/296.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(plain): say what cancelPending() actually guarantees, and pin it

rpc_connection.h claimed that "a caller that gives up before the reply arrives
calls cancelPending() and is never called back at all". It is not true.
dispatchIncoming copies the handler out of m_pendingCalls under m_mu and invokes
it with the mutex RELEASED, so a cancelPending() landing in that gap erases an
entry that is already gone and returns having stopped nothing — the handler then
runs to completion, after cancelPending() has returned.

THE CODE IS FINE; THE COMMENT WAS NOT, and the distinction it was blurring is
the load-bearing one:

  * at-most-once INVOCATION of a registered handler is this layer's, and comes
    from the extract-and-erase under m_mu — three contenders (dispatchIncoming,
    fail()'s sweep, cancelPending) and only one can have it;
  * exactly-once DELIVERY to the user is NOT this layer's. It is
    AsyncCall::deliver()'s CAS, and nothing else.

Both callers of sendCallAsync() were checked, because a non-idempotent one would
have made this a bug rather than a comment. PlainLogosObject funnels every
outcome into deliver(). RpcConnection::sendCall()'s promise handler has no CAS
and needs none: only one contender ever reaches it, and fulfilling a future its
caller has already walked away from is a no-op. Same for the two cancelPending()
callers that are not deliver() — the sync callMethodWithError timeout and
getMethods().

Three comment sites corrected (ResultHandler, dispatchIncoming's Result arm,
cancelPending's own contract) and the claims moved out of prose into
tests/protocol/test_plain_cancel_pending_race.cpp, which BUILDS the interleaving
instead of racing for it: a stub connection reproduces dispatchIncoming's
extract-then-invoke and lets the test stand between the halves. Four tests — the
callback that fires after cancelPending() returns, the extracted reply racing
teardown (exactly one delivery), the promise-shaped handler in the same gap, and
a real RpcConnection pair proving the half of the old comment that IS true.

The exactly-once one goes RED under -DLOGOS_PROTOCOL_DETECTOR_INVERSIONS=ON:
2 deliveries for 1 call, deterministically rather than probabilistically. Six
tests now go red in that build, listed in tests/protocol/CMakeLists.txt. Full
suite 302/302.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(plain): take the fold's detector inversions out of the transport too

Three more compiled-in alternative implementations, same anti-pattern as
the one the base branch just lost: production source holding a second,
deliberately wrong version of its own contract so the suite could be
built once with the mechanism removed.

  * deadlineContext() had LOGOS_PLAIN_DETECTOR_BREAK_DEADLINE_ISOLATION
    returning IoContextPool::shared().ioContext() — the rejected design
    DeadlineService exists to avoid. Kept DeadlineService::shared()
    .context(); the io_context_pool.h include the fold added for that
    branch alone goes with it.
  * AsyncCall::claim() had LOGOS_PLAIN_DETECTOR_BREAK_ONCE storing
    `delivered` and returning true unconditionally. Kept the CAS.
  * AsyncCall::takeCallback() had the same macro returning a COPY of the
    callback. Kept the swap.

And the CMake option that defined all three, whose surviving content —
which six tests are real detectors, and that the per-path exactly-once
tests are PINS rather than detectors because a call resolved once calls
deliver() once whatever guards it — moved into the note that replaces
it.

The comments now describe the validation that actually happened: a local
edit in a throwaway checkout, with the numbers each run produced. The
sub-order detector needs no edit at all, since it goes red on pristine
master.

302 tests pass, unchanged in count: no test deleted or weakened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(iofold): count the provider's completion workers so none outlives the proxy

Fixes the Linux SIGSEGV (exit 139) in the "Run protocol tests" step.

`OmniProvider::defer` is a "multi" provider: it answers with a pending sentinel
and pushes the real completion later from a worker of its own. That worker calls
the EventCallback ModuleProxy handed the provider, and that listener captures
`this` RAW (module_proxy.cpp) — its first act is
`QMetaObject::invokeMethod(this, …, Qt::QueuedConnection)`, which dereferences
the QObject. The worker was spawned DETACHED, so nothing proved it had finished,
and `~LiveHost` runs `delete m_proxy` a couple of milliseconds after the last
spawn.

Nothing keeps the two apart. A round of ReleaseRacingAnInFlightCompletionIsSafe
ends when the CALLER is answered, and a release()d call is answered by teardown —
so the round can be over before the provider has run at all, and the proxy's
thread is still working through a backlog of `defer` calls while ~LiveHost is
already tearing down. Two to five workers were standing in the same frame at the
moment of the fault:

    Thread "QThread" received signal SIGSEGV
    QObject::thread() const
    QMetaObject::invokeMethodImpl(QObject*, …)
    ModuleProxy::ModuleProxy(...)::<lambda(const QString&, const QVariantList&)>
                                                       module_proxy.cpp:37
    std::function<void(const QString&, const QList<QVariant>&)>::operator()
    OmniProvider::callMethod(...)::<lambda()>           test_iofold.cpp, in defer
    std::thread::_State_impl<…>::_M_run()

WHY ONLY THE WHOLE-BINARY RUN. The corpse is left by the test that spawned the
worker and lands in whichever test runs NEXT — in CI, always
ReleaseFromInsideAnIoThreadEventCallbackDoesNotWedge, which follows the 300-round
ReleaseRacingAnInFlightCompletionIsSafe and its 300 `defer` calls. Under ctest
every test is its own process, so the worker dies with the process that owned the
proxy and there is nothing left to fault: `nix build '.#tests'` is green on the
unfixed tree, which is exactly how this hid.

Measured on Linux (aarch64, Qt 6.9.2, gcc 14.3, `nix build '.#tests'` artifact,
QT_QPA_PLATFORM=offscreen, five concurrent copies):

    tree                          IoFoldTest.*     whole binary
    master (03842db)              n/a (no test)    0/10
    feat/plain-async-io-fold       6/50  (12%)     2/20  (10%)
    + harness-host-thread-affinity 10/50 (20%)     5/20  (25%)
    this commit, on either tree     0/50            0/20

So it is latent HERE and merely widened by destroying the fixtures' host on the
proxy's thread: that order lets the queued `defer` backlog RUN instead of
discarding it with QThread::quit(), which is why the rate roughly doubles. The
defect and the fix both belong to this commit's tree.

macOS never opens the window — 3/3 clean whole-binary runs on the unfixed tree,
and 4/4 clean under Guard Malloc (which unmaps the freed page, so a late worker
would fault every time), which is why that half of the matrix stayed green.

A COUNT, NOT A JOIN. Joinable workers would hold their 8MB stacks until reaped —
400 outstanding in NormalAndDeferredCompletionsDeliverExactlyOnceAtVolume — and
reaping from the dispatch thread would block the very thread `defer` exists to
free. So they stay detached and the provider counts them, with the decrement and
its notify under one mutex so a drain() woken by it cannot return before the
worker has released that mutex.

~LiveHost drains AFTER `m_thread->wait()`: the proxy's event loop has stopped, so
no queued call can reach the provider any more and the worker set is FINAL —
before that point a drain could pass and the backlog spawn more. ~OmniProvider
drains too, because a worker's last act touches one of its members.

Verification, all on the fixed tree:
  * `nix build '.#tests'` — 402/402, Linux and macOS (173s on macOS).
  * 20 whole-binary Linux runs: 402/402 every time.
  * Exactly-once, on the release-race shape rather than the per-path pins:
    200,000 calls released mid-burst across those 20 runs (20 rounds x 500 each)
    reported DOUBLE deliveries=0 dropped=0, and all 20 runs of
    ReleaseRacingAnInFlightCompletionIsSafe were 300/300 with 0 doubles.
  * Teardown stays fast: `release()` 0-2ms with 32 calls in flight, and the
    added wait is on the FIXTURE, not on release() — it delays `delete m_proxy`
    by however long a completion worker still had to sleep (<=2.4ms here).
  * No production code touched, so "no user callback inline on an io thread" and
    the deadline guarantees are unchanged.

Not touched: test_plain_completion_sub_order.cpp (InstantMultiModule) and
test_concurrent_dispatch.cpp spawn the same detached completion worker, and read
the callback member through a captured `this` on top of it. Neither has been
observed to fault.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Dario Lipicar
2026-08-12 16:23:57 -03:00
committed by GitHub
co-authored by Claude Opus 5
parent 4a20e99260
commit 7be3a6b856
8 changed files with 2878 additions and 1769 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cpp/implementations/plain/plain_logos_object.cpp
+556 -413
View File
File diff suppressed because it is too large Load Diff
cpp/implementations/plain/plain_logos_object.h
+91 -127
View File
@@ -12,12 +12,15 @@
#include <memory>
#include <mutex>
#include <string>
#include <thread>
#include <utility>
#include <vector>
namespace logos::plain {
// One in-flight async call. Defined in the .cpp — nothing outside needs its
// shape, and keeping it there keeps Boost.Asio out of this header.
struct AsyncCall;
// -----------------------------------------------------------------------------
// PlainLogosObject — consumer-side LogosObject backed by the plain-C++
// RPC runtime. Identical public shape to LocalLogosObject / RemoteLogosObject
@@ -70,55 +73,83 @@ public:
void release() override;
quintptr id() const override;
private:
// The deferred ("multi") completion rendezvous, in a block that is OWNED by
// the object but does not DIE with it.
public:
// ── the shared call state ────────────────────────────────────────────────
//
// A multi provider returns a pending sentinel (logos::pendingCallKey) from
// callMethod and later pushes the real result as a logos::callCompleteEvent
// event keyed by callId. We subscribe to that event EAGERLY (before any
// call can defer) so a completion racing ahead of the waiter is buffered,
// then block the caller until the matching callId lands. The completion
// arrives on the connection's IO thread; the caller waits on another
// thread — mu/cv bridge them.
// Everything an OFF-THREAD handler can reach lives here rather than on the
// handle, and the handle's own lifetime stops mattering to those handlers.
//
// WHY IT IS A SEPARATE BLOCK. The subscription handler lives in the
// RpcConnection, which is SHARED by every PlainLogosObject the connection
// hands out and outlives all of them (see release()). RpcConnection copies
// a handler out of its map under its own mutex and then invokes it with
// that mutex RELEASED — so the unsubscribe release() sends cannot reach a
// handler already in flight on the io thread, and nothing joins that
// thread. When the handler captured `this`, a completion arriving across a
// release() wrote to a freed object; reproduced as a SIGSEGV under Guard
// Malloc, in test_plain_completion_sub_lifetime.cpp.
// That split is not decoration. release() ends in `delete this`, and
// LogosObject's ABI is frozen (logos_object.h) — the handle crosses module
// boundaries as a raw pointer, so it cannot itself become shared-owned.
// But nothing a handler touches goes through the handle: no virtual, no
// id(), not even its address. So the STATE becomes shared-owned and the
// facade stays exactly as it was. Handlers hold a shared_ptr to the
// per-call AsyncCall and a weak_ptr to this block; the last one to run
// drops the last share, whenever that is.
//
// Putting the rendezvous behind a shared_ptr and handing the handler a
// weak_ptr makes "no handler touches a destroyed object" true by
// construction: a handler that locks it keeps it alive for the length of
// one callback, and one that cannot lock it does nothing. Nothing else in
// this object is reachable from the handler, which is what keeps the fix
// this small.
struct CompletionRendezvous {
std::mutex mu;
std::condition_variable cv;
// This is the same block the completion-subscription lifetime fix
// introduced (it was CompletionRendezvous: mutex, condvar, completions),
// widened to carry the in-flight calls the fold moved off their threads.
// The guarantee it exists for is unchanged and is now load-bearing for
// three handlers instead of one: NO HANDLER TOUCHES A DESTROYED OBJECT,
// true by construction rather than by a barrier.
struct CallState {
std::mutex mu;
// The SYNC path's rendezvous, unchanged in kind: callMethodWithError
// still parks its own caller's thread here, because a synchronous call
// has to block someone and that someone is the caller.
std::condition_variable cv;
std::map<QString, QVariant> completions;
// In-flight ASYNC calls, keyed by the call's wire id. This is the whole
// retention story on the handle now: an entry exists exactly while its
// call is outstanding and is erased by the one delivery it gets. No
// waiter registry, no publish list, no reaping, nothing that survives a
// completed call.
std::map<std::uint64_t, std::shared_ptr<AsyncCall>> inflight;
// Second index over the same calls, for the ones a "multi" provider
// deferred: the completion event is keyed by the provider's callId
// string, not by our numeric id.
std::map<QString, std::shared_ptr<AsyncCall>> deferred;
// Set once by teardown, never cleared. Written under `mu` so the
// condition-variable side cannot miss it, and atomic so the lock-free
// readers do not have to take the mutex.
std::atomic<bool> stopping{false};
};
// Bring the completion subscription up, ONCE, and — the part that is not
// the same thing — make every other caller wait until it is actually up.
private:
// Deferred ("multi") completion rendezvous. A multi provider returns a
// pending sentinel (logos::pendingCallKey) from callMethod and later pushes
// the real result as a logos::callCompleteEvent event keyed by callId. We
// subscribe to that event EAGERLY (before any call can defer) so a completion
// racing ahead of the caller is buffered, then either resolve the waiting
// AsyncCall directly (async) or wake the parked caller (sync).
//
// The window this closes is a LOST COMPLETION, not a crash. The flag used to
// be raised under the rendezvous mutex and the mutex DROPPED before the
// subscribe, so a second caller could read "subscribed", build its Call and
// put it on the wire while the Subscribe frame had not been enqueued yet. A
// "multi" provider that answers such a call quickly emits its completion
// into a subscription the host has not registered — PlainTransportHost::
// fanOutEvent finds no sink for that connection and DROPS it — and the
// caller then waits out its full timeout for a result that was computed and
// thrown away. Measured on pristine master, four runs: 18 to 28 of 250
// two-thread first-call rounds inverted on the wire, every one of them a
// dropped completion and a timed-out caller; 6 to 10 of 500 calls through
// the real host stack.
// The completion arrives on the connection's IO thread, and the subscription
// holds a weak_ptr to CallState — never `this`. That subscription lives in
// the RpcConnection, which is SHARED by every PlainLogosObject the
// connection hands out and outlives all of them (see release()), and
// dispatchIncoming copies the handler out under its own mutex and invokes it
// with that mutex RELEASED — so the unsubscribe release() sends cannot reach
// a handler already in flight, and nothing joins the io thread. With `this`
// captured, a completion arriving across a release() wrote to a freed
// object; reproduced as a SIGSEGV under Guard Malloc, in
// test_plain_completion_sub_lifetime.cpp.
//
// ONCE, and — the part that is not the same thing — with every other caller
// WAITING until it is actually up. The flag used to be raised under
// CallState::mu and the mutex DROPPED before the subscribe, so a second
// caller could read "subscribed", build its Call and put it on the wire
// while the Subscribe frame had not been enqueued yet. A "multi" provider
// that answers such a call quickly emits its completion into a subscription
// the host has not registered — PlainTransportHost::fanOutEvent finds no
// sink for that connection and DROPS it — and the caller then waits out its
// full timeout for a result that was computed and thrown away. Measured on
// pristine master, four runs: 18 to 28 of 250 two-thread first-call rounds
// inverted on the wire, every one a dropped completion and a timed-out
// caller; 6 to 10 of 500 calls through the real host stack.
//
// Ordering, once the two are serialized, is a property of asio and not of
// luck: handlers posted to a strand run in the order they were posted when
@@ -136,53 +167,30 @@ private:
const QString& methodName = QString(),
logos::CallError* err = nullptr);
// Ask every in-flight waiter to give up, then join them, then return.
// Raise the stop flag, then cancel every outstanding async call — each of
// which delivers its callback, once, with callErrorReleased — and wake the
// synchronous caller if one is parked.
//
// The JOIN is what makes the waiters safe at all: they capture `this` (they
// read m_stopping and call awaitCompletion), and callMethodAsync used to
// DETACH them, so release()/delete racing an in-flight wait was a
// use-after-free. But joining alone means teardown blocks for whatever is
// left of the call's timeout — up to 20s on the protocol default — because
// a waiter has no reason to return early. Hence the stop first: it costs
// one wait slice instead, and a cancelled call still delivers its callback
// exactly once (with an error), because dropping it would turn the stall
// into a permanent hang in the caller awaiting it.
void stopAndJoinWaiters();
// Raise the stop flag and wake anything parked on the rendezvous cv. Split
// out because the flag has to be published under the rendezvous mutex (see
// the .cpp).
void stopWaiters();
// Join and drop the waiters that have already FINISHED, so a handle that
// outlives its calls does not accumulate them. TWO call sites, which
// between them cover both shapes of traffic:
//
// * every async spawn — a call pays for the corpses of earlier ones;
// * every waiter as it finishes, BEFORE it publishes its own id — so a
// burst drains itself instead of parking until the next call, which for
// a module that bursts and goes quiet may never come.
//
// NOT called from stopAndJoinWaiters(): teardown joins by id-independent
// brute force and needs no published list. (It used to say otherwise here;
// it never did.) Cheap either way: a join on an already-returned thread is
// a couple of syscalls, and only ids a waiter itself published are touched.
void reapFinishedWaiters();
// A waiter's FINAL act — see the scope guard in callMethodAsyncWithError.
// After this returns, that thread never touches the object again, which is
// what makes it safe for someone else to join and drop it. Nothing the
// waiter does may follow it, its own reap least of all.
void publishFinishedWaiter(std::uint64_t id);
// WHAT REPLACED THE JOIN. In-flight calls used to be threads that captured
// `this`, so teardown had to prove none of them was still running before
// `delete this`, and the only tool for that was joining threads it first had
// to ask to stop (a wait slice at best). Nothing captures `this` any more: a
// handler holds a shared_ptr to its AsyncCall and a weak_ptr to CallState.
// Teardown therefore waits for NOTHING — not the io thread, not a wait slice
// — which also means it cannot deadlock when release() is called from inside
// an event callback running on the single io thread (the shape
// remote_transport.cpp documents as real). It stays O(in-flight calls).
void stopAndCancelCalls();
std::string m_objectName;
std::shared_ptr<RpcConnectionBase> m_conn;
std::mutex m_mu;
std::vector<std::pair<QString, EventCallback>> m_subs;
// Never null and never reseated: the object owns exactly one rendezvous for
// its whole life, and the only other references are the weak_ptr the
// subscription handler holds and whatever a handler has momentarily locked.
std::shared_ptr<CompletionRendezvous> m_completion =
std::make_shared<CompletionRendezvous>();
// Never null and never reseated: the object owns exactly one state block for
// its whole life, and the only other references are the weak_ptrs its
// handlers hold and whatever one of them has momentarily locked.
std::shared_ptr<CallState> m_state{std::make_shared<CallState>()};
// "The Subscribe frame is on the strand." Stored with RELEASE after
// subscribeToCompletions() returns and read with ACQUIRE on the fast path,
// so a caller that skips the once_flag still inherits the edge that orders
@@ -191,50 +199,6 @@ private:
// What makes a concurrent first caller WAIT rather than sail past a flag
// that has been raised but not yet honoured. The whole fix is this member.
std::once_flag m_completionSubOnce;
// The waiter registry. KEYED, not a plain vector, because a thread cannot
// join itself: a waiter can therefore never retire its own entry, and a
// vector left only one moment to clear it — teardown — so every completed
// call parked a finished-but-unjoined thread (~one page of resident memory
// each) for the whole life of the handle. The production shape is one
// cached handle per module reused for every call (logos_api_consumer.cpp),
// so that grew without bound. Now a waiter publishes its id into
// m_finishedWaiters as its last act, and both the next spawn and every
// OTHER waiter on its way out join and erase it: see reapFinishedWaiters().
//
// Retention tracks neither call count nor peak concurrency. A burst drains
// as it completes, because each waiter reaps the ones that finished before
// it. What survives an idle handle is only what published after the last
// reap — at minimum the last waiter to finish, which by construction has
// nobody behind it to collect it. The next call, or teardown, takes those.
//
// That remainder is the size of the last exit batch, NOT a small constant,
// and nothing collects it while the handle stays idle: sampled out to 25.6s
// it does not move. What sets it is how SERIALIZED the exits are, because a
// waiter reaps and only then publishes: exits that interleave one after
// another leave 1, while a batch that becomes runnable together leaves most
// of itself, since the reaper that collects a large batch sits in its join
// loop (no lock held) while everyone behind it publishes and finds nobody to
// collect them. Measured on the drain of an 800-call burst that was fully
// outstanding before any reply — the worst case for this, and the shape
// test_plain_waiter_reaping.cpp builds deliberately — with all 800 answered
// at once: 1 every run on 10-core macOS, but 2-389 on a 6-core Linux box. It
// does not scale with the CALL COUNT, which is the claim; "1-2" was a
// one-platform reading of it, and a burst answered at any pace at all (16 at
// a time is enough, measured) leaves 1-6 on both.
//
// The real fix is still the TODO in callMethodAsyncWithError — fold the
// wait into the shared Asio io_context and have no thread per pending RPC
// at all. This makes the interim honest, it does not replace that.
std::mutex m_waiterMu;
std::map<std::uint64_t, std::thread> m_waiters;
std::vector<std::uint64_t> m_finishedWaiters;
std::uint64_t m_nextWaiterId = 0;
// Read lock-free by the sliced future wait and under the rendezvous mutex
// by awaitCompletion's predicate; written under that same mutex so the
// condition-variable side cannot miss it. Never cleared — an object that
// has begun tearing down does not come back.
std::atomic<bool> m_stopping{false};
};
} // namespace logos::plain
cpp/implementations/plain/rpc_connection.h
+114 -13
View File
@@ -41,6 +41,40 @@ namespace logos::plain {
class RpcConnectionBase {
public:
using ErrorHandler = std::function<void(const std::string& reason)>;
// A reply, handed over as it arrives instead of parked in a promise.
//
// Invoked AT MOST ONCE per call, from one of three places, and a caller has
// to answer for all three because they are not on the same thread:
// * the connection's strand (io thread) when the peer's Result frame is
// decoded — the normal path;
// * an arbitrary caller thread inside fail(), which sweeps every pending
// call when the connection is torn down (stop(), ~PlainTransport-
// Connection, RpcServer::stop());
// * INLINE on the calling thread, inside sendCallAsync itself, when the
// connection is already stopped.
// It must therefore not block and must not run user code directly — see
// postToQtEventLoop in plain_logos_object.cpp.
//
// AT MOST ONCE is a property of the REGISTRATION, and it is weaker than it
// sounds. Three things contend for a registered handler — dispatchIncoming,
// fail()'s sweep and cancelPending() — and the extract-and-erase under m_mu
// lets exactly one of them have it, so no handler is ever invoked twice.
//
// What that does NOT buy is a cancel that arrives in time. dispatchIncoming
// copies the handler out under m_mu and invokes it with the mutex RELEASED,
// so a cancelPending() landing in that gap erases nothing and the handler
// runs to completion AFTER cancelPending() has already returned. A caller
// that gives up must therefore be able to absorb one more call. Both callers
// here are:
// * PlainLogosObject funnels every outcome into AsyncCall::deliver(),
// whose CAS makes the later arrival a no-op — that CAS, and nothing at
// this layer, is what makes DELIVERY to the user exactly-once;
// * sendCall()'s promise handler cannot be reached twice at all (only one
// contender ever gets it) and fulfilling a future its caller has already
// walked away from is a no-op.
// test_plain_cancel_pending_race.cpp builds that interleaving by hand rather
// than racing for it, and pins both.
using ResultHandler = std::function<void(ResultMessage)>;
virtual ~RpcConnectionBase() = default;
@@ -49,8 +83,40 @@ public:
virtual bool isOpen() const = 0;
virtual std::future<ResultMessage> sendCall(CallMessage msg) = 0;
// The same send, completion-driven. sendCall() is now a thin wrapper over
// this one (it fulfils a promise from the handler), so there is exactly one
// registration path and the two cannot drift.
virtual void sendCallAsync(CallMessage msg, ResultHandler handler) = 0;
virtual std::future<MethodsResultMessage> sendMethods(MethodsMessage msg) = 0;
// Forget a pending Call or Methods registration whose caller has given up.
//
// THIS IS A RETENTION FIX, and it closes a hole that predates the async
// rework. m_pendingCalls / m_pendingMethods are emptied by exactly two
// events: a decoded reply carrying that id, and fail()'s teardown sweep. A
// call that is resolved by its DEADLINE and never answered is in neither,
// so its registration — a promise, or now a handler holding the caller's
// std::function — stayed in the map for the whole life of the connection.
// Measured against pristine master with a server that never answers: 8.5MB
// of resident memory over 24,000 orphaned calls, 353 bytes each, growing
// strictly linearly with the call count. And the connection outlives every
// handle it hands out, so nothing else was ever going to collect it.
//
// Erasing is the right semantic and not merely a cleanup: the caller has
// already been told the call timed out, so a reply arriving afterwards must
// be dropped, which is exactly what an absent registration does.
//
// Safe to call at any time and from any thread, including for an id that
// has already been answered (the erase simply finds nothing). Ids come from
// nextId() and are unique across BOTH maps, so one entry point covers them.
//
// BEST EFFORT AGAINST A REPLY ALREADY IN FLIGHT, and deliberately not more.
// It withdraws a REGISTRATION; it does not stop a handler dispatchIncoming
// has already taken out of the map. Returning from this is therefore not a
// guarantee of silence — see ResultHandler for who has to absorb the
// difference and how.
virtual void cancelPending(uint64_t id) = 0;
virtual void sendSubscribe(SubscribeMessage msg,
std::function<void(EventMessage)> callback) = 0;
virtual void sendUnsubscribe(UnsubscribeMessage msg) = 0;
@@ -89,8 +155,15 @@ public:
bool isOpen() const override { return !m_stopped.load(); }
std::future<ResultMessage> sendCall(CallMessage msg) override;
void sendCallAsync(CallMessage msg, ResultHandler handler) override;
std::future<MethodsResultMessage> sendMethods(MethodsMessage msg) override;
void cancelPending(uint64_t id) override {
std::lock_guard<std::mutex> g(m_mu);
m_pendingCalls.erase(id);
m_pendingMethods.erase(id);
}
void sendSubscribe(SubscribeMessage msg,
std::function<void(EventMessage)> callback) override;
void sendUnsubscribe(UnsubscribeMessage msg) override;
@@ -131,9 +204,10 @@ private:
std::deque<std::vector<uint8_t>> m_writeQueue;
bool m_writing = false;
// Outgoing-pending maps
// Outgoing-pending maps. Calls hold a HANDLER rather than a promise: the
// promise is one possible handler (see sendCall), not the mechanism.
std::mutex m_mu;
std::map<uint64_t, std::shared_ptr<std::promise<ResultMessage>>> m_pendingCalls;
std::map<uint64_t, ResultHandler> m_pendingCalls;
std::map<uint64_t, std::shared_ptr<std::promise<MethodsResultMessage>>> m_pendingMethods;
using EventKey = std::pair<std::string, std::string>; // object, event
@@ -225,16 +299,24 @@ void RpcConnection<Stream>::dispatchIncoming(AnyMessage msg)
using T = std::decay_t<decltype(m)>;
if constexpr (std::is_same_v<T, ResultMessage>) {
std::shared_ptr<std::promise<ResultMessage>> p;
ResultHandler h;
{
std::lock_guard<std::mutex> g(m_mu);
auto it = m_pendingCalls.find(m.id);
if (it != m_pendingCalls.end()) {
p = std::move(it->second);
h = std::move(it->second);
m_pendingCalls.erase(it);
}
}
if (p) p->set_value(std::forward<decltype(m)>(m));
// Erased under the lock BEFORE the call, so this, fail()'s sweep and
// cancelPending() cannot all get the same handler — that is what
// makes INVOCATION at-most-once at this layer, and it is the whole
// of what this layer promises. It is NOT a cancellation barrier: the
// call below runs with m_mu released, so a cancelPending() racing it
// finds the entry already gone, erases nothing, and returns while
// this handler is still running. Exactly-once DELIVERY belongs to the
// handler — see ResultHandler.
if (h) h(std::forward<decltype(m)>(m));
} else if constexpr (std::is_same_v<T, MethodsResultMessage>) {
std::shared_ptr<std::promise<MethodsResultMessage>> p;
@@ -303,19 +385,35 @@ RpcConnection<Stream>::sendCall(CallMessage msg)
{
auto p = std::make_shared<std::promise<ResultMessage>>();
auto f = p->get_future();
// The promise is now just one shape of handler. Everything the future path
// relied on — registration under m_mu before the write, the stopped
// early-out, fail()'s sweep — lives in sendCallAsync and is shared verbatim.
sendCallAsync(std::move(msg), [p](ResultMessage r) {
try { p->set_value(std::move(r)); } catch (...) {}
});
return f;
}
template <typename Stream>
void RpcConnection<Stream>::sendCallAsync(CallMessage msg, ResultHandler handler)
{
if (!handler) return;
if (m_stopped.load()) {
// Answered INLINE, on the caller's thread. That is the same shape the
// future path had (it set the promise before returning it), and it is
// why every handler in this codebase has to be non-blocking and has to
// hand user code off to the Qt loop rather than run it here.
ResultMessage r;
r.id = msg.id; r.ok = false;
r.err = "connection stopped"; r.errCode = "TRANSPORT_CLOSED";
p->set_value(std::move(r));
return f;
handler(std::move(r));
return;
}
{
std::lock_guard<std::mutex> g(m_mu);
m_pendingCalls[msg.id] = p;
m_pendingCalls[msg.id] = std::move(handler);
}
writeFrame(encodeFrame(*m_codec, AnyMessage{std::move(msg)}));
return f;
}
template <typename Stream>
@@ -471,8 +569,8 @@ void RpcConnection<Stream>::fail(const std::string& reason)
bool expected = false;
if (!m_stopped.compare_exchange_strong(expected, true)) return;
// Fail every pending promise with a transport-level error.
std::map<uint64_t, std::shared_ptr<std::promise<ResultMessage>>> calls;
// Fail every pending call with a transport-level error.
std::map<uint64_t, ResultHandler> calls;
std::map<uint64_t, std::shared_ptr<std::promise<MethodsResultMessage>>> methods;
ErrorHandler errCb;
{
@@ -482,10 +580,13 @@ void RpcConnection<Stream>::fail(const std::string& reason)
errCb.swap(m_error);
m_eventCallbacks.clear();
}
for (auto& [id, p] : calls) {
for (auto& [id, h] : calls) {
ResultMessage r; r.id = id; r.ok = false;
r.err = reason; r.errCode = "TRANSPORT_ERROR";
try { p->set_value(std::move(r)); } catch (...) {}
// Runs on WHATEVER THREAD called stop() — usually not the io thread.
// Handlers are written for that (see ResultHandler); the try/catch is
// the same containment the promise sweep already had.
try { h(std::move(r)); } catch (...) {}
}
for (auto& [id, p] : methods) {
MethodsResultMessage r; r.id = id; r.ok = false; r.err = reason;