"token" was overloaded (auth tokens, cgo handles, lexer tokens) and didn't say
what it is — a per-call correlation id linking an outgoing {.ffiHost.} call to
the answer that arrives later (possibly from another thread). Renamed across the
runtime (ffi_host / ffi_context), the macro, the exported C ABI (FFIHostFn,
<lib>_host_complete), the Go trampoline, and the tests; regenerated bindings.
The unrelated request-path cgo.Handle result-slot (also informally called a
"token" in go.nim comments) is left as-is — different mechanism.
16 host unit tests + the examples/host_demo Go round-trip stay green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Increment 4: the exported C surface for host callbacks, plus an end-to-end
test that the host can answer from a different thread than the FFI loop.
- declareLibrary now emits two exportc/cdecl procs on every library's
FFIContext (like the event ABI):
<lib>_register_host_fn(ctx, name, fn, userData)
<lib>_host_complete(ctx, token, ret, msg, len)
(the `name` param is spelled `hostFnName` to dodge the macros.name capture
under quote, same class as the existing id/ret collisions.)
- c.nim emits the FFIHostFn typedef + both declarations into <lib>.h
(guarded, format-agnostic), and the timer header is regenerated.
- Verified: the built timer lib exports both symbols.
The e2e (test_ffi_host_e2e) drives the real bridge: a {.ffi.} handler awaits a
{.ffiHost.} call; the host fn (invoked on the FFI thread, non-blocking) hands
the work to a worker thread, which answers via the completion path. The result
resolves on the loop thread and round-trips correctly (orc+refc). It calls the
underlying registerHostFn/completeHostCall directly, since the exported shims
need an --app:lib build; those shims are verified by the symbol check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Increment 3: the {.ffiHost.} pragma. A bodyless
proc fetchToken(key: string): Future[Result[string, string]] {.ffiHost.}
expands into an async proc that resolves the thread-local host registry +
pending table, looks the fn up by snake_case wire name, allocates a token,
invokes the host with the raw request bytes, and awaits the answer.
This is the inverse of {.ffi.} and the first end-to-end use of the registry
(increment 1) + completion bridge (increment 2). First slice is deliberately
narrow — raw ABI, one string param, Future[Result[string, string]] — to prove
the round-trip with zero serialization; struct params/returns and the
{.ffiHost: cbor.} format arg are follow-ups.
The body reads two new threadvars (ffiCurrentHostRegistry / ffiCurrentPendingTable)
set by ffiThreadBody alongside ffiCurrentEventRegistry, so the user's signature
stays ctx-free. The host fn is invoked synchronously before the await, while the
string arg is still alive (honouring the "req valid only for the call" contract).
5 macro tests pass under orc+refc; host + ffi_context suites stay green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Increment 2: wires the host-call machinery into the running FFI thread so a
host answer (delivered from any thread) resolves the chronos Future an awaiting
handler is blocked on.
- FFICompletionQueue (ffi_host.nim): a GC-free intrusive queue. host_complete
pushes c_malloc'd nodes from any thread; the FFI thread drains, copies the
payload into GC memory, completes the future by token, and frees the node.
- FFIContext gains hostRegistry / pendingTable / completionQueue, init'd and
deinit'd alongside the event registry.
- completeHostCall parks the answer and fires the EXISTING reqSignal — no second
ThreadSignalPtr needed; the loop drains completions every iteration, on the
loop thread (chronos single-thread invariant).
- On shutdown the loop failAllPendings first, so a handler awaiting a host
answer that never arrives can't hang the allFutures(pending) drain.
4 new queue unit tests (10 total) pass under orc+refc; the 19 ffi_context
integration tests stay green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First increment of typed host callbacks (roadmap #1): the data-structure
layer, independent of the FFI thread and the macro so it can be unit-tested
in isolation.
- FFIHostRegistry: wire-name -> (host fn ptr, userData). A missing entry is a
normal outcome (the imported proc errors), never a crash — never-crash
policy. nil fn unregisters.
- FFIPendingTable: monotonic token -> the chronos Future an awaiting
{.ffiHost.} proc is blocked on. completePending drops unknown/double
completions; failAllPending errors every outstanding future on teardown so
no awaiting handler is abandoned.
Both lock-guarded so a host thread and the FFI thread can touch them
concurrently; futures are only ever completed on the FFI thread. 6 unit tests
pass under orc and refc.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A "kitchen sink" {.ffi.} object spanning every supported field shape — all
integer widths, both floats, bool, string, sequences (scalars / strings /
floats / nested structs), Option/Maybe, and a nested struct by value — is sent
in as a C-POD and returned as a typed C-POD, then checked field-for-field
against the Nim-native result.
This is the native-path complement to the existing CBOR coverage (test_serial
for the codec, test_wire_compat for the bytes): it pins nimToPod ->
*NativeExport -> clonePod/podToNim of the typed return for the whole type
matrix. Compiling also proves the native-POD codegen accepts every type. Passes
under orc + refc and clean under ASAN.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Commit f3206c3 split each FFI export into two distinctly-named Nim procs
(`<name>CborExport` / `<name>NativeExport`, and the ctor variants), so the
bare user name now resolves only to the Nim-native helper. The C-shape
integration tests still invoked the CBOR entry points by the bare name and
no longer compiled. Point those call sites at the `*CborExport` /
`*CborCtorExport` procs; the Nim-native `waitFor <name>(lib, ...)` calls keep
the bare name on purpose.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
