Merge eb62813af592edee882af663a47359507a1455ed into c000a8467dfc81af043bbb1f11d1da03570e5128

docs/design-abi-format.md

@@ -0,0 +1,75 @@
+# Design: ABI format selection (`raw` vs `cbor`)
+
+Status: **direction decided, open for refinement.** How a library author tells the
+compiler which wire format a boundary-crossing proc uses. Applies to the whole
+pragma family (`{.ffi.}`, `{.ffiHost.}`, `{.ffiCtor.}`, `{.ffiDtor.}`).
+
+## The two formats
+
+- **`raw`** — native, zero-serialization C-POD ABI. Same-process; the request/
+  result is a C struct passed/cast directly under the deep-copy +
+  callback-lifetime ownership rule. The common path.
+- **`cbor`** — CBOR-encoded buffer. For IPC / generic / cross-language callers
+  where serialization is required anyway.
+
+## Decision
+
+**A1 — per-proc pragma argument** (chosen). The format is an argument on the
+pragma, read by the macro at compile time:
+
+```nim
+proc echo(req: EchoReq): Future[Result[EchoResp, string]] {.ffi: raw.}
+proc echo(req: EchoReq): Future[Result[EchoResp, string]] {.ffi: cbor.}
+
+proc fetchProfile(id: string): Future[Result[Profile, string]] {.ffiHost: cbor.}
+```
+
+Matches the existing `{.ffiEvent: "wire_name".}` precedent (pragmas already take
+args in this codebase). Local, granular, lets one library mix both formats.
+
+**C2 — library-wide default** (chosen, layered on top). Set the default once at
+`declareLibrary`; a per-proc pragma arg overrides it:
+
+```nim
+declareLibrary("my_app", MyApp, defaultAbiFormat = raw)
+# every {.ffi.}/{.ffiHost.}/… is `raw` unless it says otherwise
+```
+
+So the common case stays terse, with per-proc control when needed.
+
+**Rejected:**
+- **Global compile flag** (`-d:ffiFormat=…`) — discarded. All-or-nothing,
+  action-at-a-distance, can't mix formats in one library. `defaultAbiFormat`
+  replaces its only virtue (a single default) without the downsides.
+- Distinct pragma names (`{.ffiCbor.}`, `{.ffiHostCbor.}`) — combinatorial
+  explosion across the pragma family.
+- Format on the data type (`type T {.ffi: cbor.}`) — misattributes a transport
+  property to the data; breaks when one type is used by both a raw and a cbor
+  proc.
+
+## Sketch
+
+```nim
+type AbiFormat* = enum
+  raw    # native zero-serialization C-POD (same-process)
+  cbor   # CBOR buffer (IPC / generic)
+
+# two overloads, like ffiEvent: no-arg form defaults to the library default
+macro ffi*(prc: untyped): untyped = ...                       # uses defaultAbiFormat
+macro ffi*(fmt: static[AbiFormat], prc: untyped): untyped = ...  # explicit override
+```
+
+## Open questions (for further discussion)
+
+- **Enum name / spelling.** `AbiFormat` with values `raw` / `cbor`? (`raw` over
+  `native` per current preference.)
+- **Resolution order.** proc pragma arg → `defaultAbiFormat` → built-in fallback
+  (`raw`?). Confirm the fallback when `declareLibrary` sets no default.
+- **Symbol emission.** Today `{.ffi.}` emits BOTH `<name>` and `<name>_cbor`. Does
+  a per-proc format mean we emit only the chosen symbol, or keep emitting both
+  and let the format arg pick the *primary*? (Leaning: emit only the chosen one;
+  `<name>_cbor` suffix stays only when both are explicitly wanted.)
+- **Codegen impact.** Each generator (c/go/cpp/rust/swift/kotlin) must honour the
+  per-proc format when emitting wrappers + registration symbols.
+- **Future A3.** Whether to later expose an orthogonal `{.ffi, wire: cbor.}`
+  transport pragma that composes across the family (deferred; A1+C2 first).

docs/design-host-callbacks.md

@@ -0,0 +1,151 @@
+# Design: typed host callbacks (`{.ffiHost.}`)
+
+Status: **draft / in progress.** Roadmap item #1 from [future-work.md](future-work.md).
+
+## Goal
+
+Let a Nim `{.ffi.}` handler call **back into the host language** for typed data
+and `await` the result:
+
+```nim
+# Declared in the library, implemented by the host (no Nim body):
+proc fetchProfile(userId: string): Future[Result[Profile, string]] {.ffiHost.}
+
+proc myAppLogin(app: MyApp, req: LoginReq): Future[Result[Session, string]] {.ffi.} =
+  let profile = (await fetchProfile(req.userId)).valueOr:
+    return err("host fetch failed: " & error)
+  return ok(openSession(profile))
+```
+
+This is the inverse of events (which are lib → host, fire-and-forget). It is the
+"a lower layer needs to read from a higher one" case from logos-delivery #3865.
+
+## Why it's not just "events backwards"
+
+Events invoke a host `FFICallBack` **synchronously on the FFI thread** and
+ignore any return value. A host *call* must return data, and the host may take
+arbitrary time / answer on its own thread. The chronos `Future` the Nim handler
+awaits can only be completed **on the FFI (event-loop) thread**. So the result
+has to be marshaled back across the thread boundary — exactly the reverse of the
+existing request path:
+
+```
+host → lib request : reqChannel.trySend + reqSignal.fireSync → FFI loop → processRequest → reply callback
+lib → host call    : hostFn(token, req) … host works … <lib>_host_complete(token, result)
+                     → completionQueue.push + completionSignal.fireSync → FFI loop → fut.complete(result)
+```
+
+The completion path reuses the same primitive (`ThreadSignalPtr` + an SPSC/MPSC
+queue) that `reqSignal`/`reqChannel` already use (`ffi/ffi_context.nim`).
+
+## Moving parts
+
+### 1. Host-function registry (per context)
+A small registry mirroring `FFIEventRegistry` (`ffi/ffi_events.nim`): maps a wire
+name (`"fetch_profile"`) to a `(FFIHostFn, userData)`. The host registers an
+implementation at runtime; a nil/missing entry makes the imported proc resolve
+to `err("host fn '<name>' not registered")` rather than crash (never-crash
+policy).
+
+### 2. In-flight completion table (per context)
+`token: uint64 → Completer`, where `Completer` holds the pending chronos
+`Future` and a slot for the raw result bytes. Tokens are monotonic per context.
+Guarded by a lock; only the FFI thread completes futures.
+
+### 3. Completion bridge (FFI thread integration)
+- New `completionSignal: ThreadSignalPtr` + `completionQueue` on `FFIContext`.
+- `<lib>_host_complete(...)` (called from the host thread) pushes `(token, ret,
+  bytes)` onto the queue and fires `completionSignal`.
+- The FFI loop (`ffiThreadBody`) additionally waits on `completionSignal`; on
+  wake it drains the queue and, for each entry, looks up the token and
+  `fut.complete(decodedResult)` — on the loop thread, satisfying chronos.
+
+### 4. The `{.ffiHost.}` macro
+From a bodyless `proc <name>(args…): Future[Result[T, string]] {.ffiHost.}`,
+emit a normal async Nim proc whose body:
+1. marshals `args` into a request buffer (native POD first; CBOR variant later),
+2. allocates a token + registers a `Completer` (Future) in the in-flight table,
+3. looks up the host fn for `"<name>"`; if absent → `return err(...)`,
+4. invokes `hostFn(token, reqMsg, reqLen, userData)`,
+5. `return await completer.fut` (decoded to `Result[T, string]`).
+
+Note the same dual-proc spirit as `{.ffi.}`: in-process Nim callers could later
+get a directly-injectable implementation, but the foreign path goes through the
+registry.
+
+### 5. ABI + codegen (per language)
+Exported symbols (added to `c.nim` and the other generators):
+```c
+typedef void (*FFIHostFn)(uint64_t token, const char *req, size_t reqLen, void *userData);
+int  <lib>_register_host_fn(void *ctx, const char *name, FFIHostFn fn, void *userData);
+int  <lib>_host_complete(void *ctx, uint64_t token, int ret, const char *msg, size_t len);
+```
+Each generator then emits an idiomatic wrapper: register a closure, and on
+completion call `<lib>_host_complete`. (Out of scope for the first slice — C ABI
++ a C e2e test prove the mechanism first.)
+
+## Host consumption (per language)
+
+The raw contract a host satisfies, and the rule that shapes the wrappers:
+
+1. Register `fn` under a name. When the Nim handler `await`s the imported proc,
+   the library invokes `fn` **on the FFI thread** with a `token` + marshaled
+   args.
+2. `fn` **must return immediately** — it sits on the chronos event-loop thread,
+   so it captures the `token`, kicks the real work onto the host's own executor,
+   and returns.
+3. When the work finishes (any thread, any time later), the host calls
+   `<lib>_host_complete(ctx, token, ret, msg, len)`, which enqueues + signals the
+   FFI loop to complete the awaited `Future`. The `token` is what decouples
+   "invoked on the FFI thread" from "answered later on the host's thread."
+
+The generated wrapper hides token, threading hop, and marshaling — the host dev
+writes a normal function in the language's async idiom. The wrapper's trampoline
+does three things: decode `req` → typed args, run the closure **on the host
+executor** (never inline on the FFI thread), encode the result and call
+`<lib>_host_complete`.
+
+```go
+// Go — trampoline spawns a goroutine, then host_complete
+node.SetFetchProfile(func(userID string) (Profile, error) { return db.Lookup(userID) })
+```
+```swift
+// Swift — trampoline launches a Task
+node.fetchProfile = { userID in try await db.lookup(userID) }
+```
+```kotlin
+// Kotlin — JNI trampoline launches a coroutine
+node.setFetchProfile { userID -> db.lookup(userID) }
+```
+```rust
+// Rust — closure returning a future, driven on the host runtime
+node.set_fetch_profile(|userId| async move { db.lookup(&userId).await });
+```
+
+**The gotcha the wrappers exist to enforce:** a binding that ran the closure
+inline in `FFIHostFn` would stall the event loop (and deadlock if the closure
+re-entered the library). Each language needs its own trampoline to hop onto its
+executor — that's the real work of increment 5, not a shared shim.
+
+## Threading / safety notes
+
+- Futures completed **only** on the FFI thread (drain runs there). `host_complete`
+  from any thread only enqueues + signals.
+- A `host_complete` for an unknown/expired token is dropped with a debug log (a
+  late/double completion must not crash) — never-crash policy.
+- Context teardown must fail every outstanding `Completer`
+  (`err("context shutting down")`) and drain the queue so no future is abandoned
+  (matches the existing in-flight `pending` drain in `ffiThreadBody`).
+- Re-entrancy: an imported call happens *inside* a `{.ffi.}` handler already on
+  the FFI thread; it must `await` (yield the loop) so the loop keeps draining —
+  it must never block the thread waiting on the host.
+
+## Increments
+
+1. **Registry + in-flight table** (pure data structures + unit tests) ← first
+2. Completion bridge on `FFIContext` (signal + queue + loop drain + teardown)
+3. `{.ffiHost.}` macro (native POD marshaling, string args/results first)
+4. C ABI codegen + a C end-to-end test (Nim handler calls a C-provided host fn)
+5. Idiomatic wrappers in the per-language generators
+6. CBOR variant + structured (`{.ffi.}`-typed) args/results
+```

docs/future-work.md

@@ -0,0 +1,65 @@
+# nim-ffi — future work
+
+Ideas for making nim-ffi a best-in-class FFI solution for exposing Nim to any
+platform. Captured from design discussion; not yet scheduled unless linked to a
+branch/PR.
+
+## Foundation: the dual-proc design
+
+A `{.ffi.}` / `{.ffiCtor.}` / `{.ffiDtor.}` proc compiles into **two** procs that
+share the source name:
+
+1. a normal, fully-typed Nim proc (the user's body) — callable in-process with
+   zero serialization, and unit-testable without any FFI; and
+2. an `{.exportc, cdecl, dynlib.}` wrapper with the `(ctx, cb, ud, …)` ABI that
+   foreign callers bind.
+
+Nim disambiguates by overload resolution (see `ffi/internal/ffi_macro.nim`, the
+note at the `cExportProcName` definition). Most items below build on this: the
+same source can serve an in-process Nim caller and a foreign caller over the C
+ABI, choosing the transport per call site.
+
+## Roadmap (priority order)
+
+### 1. Typed bidirectional calls — host-provided functions the Nim side can `await`  ⬅ in progress
+Today data flows lib → host as events (raw/CBOR). The inverse is missing: a Nim
+`{.ffi.}` proc calling **back into** the host language for typed data and
+awaiting the result — the "a lower layer needs to read from a higher one" case
+(logos-delivery issue #3865). A `{.ffiHost.}`-style annotation turns a
+bodyless typed Nim proc into a call that marshals to a host-registered function
+pointer and resolves a chronos `Future` when the host calls back. Reuses the
+event machinery (registry + `ThreadSignalPtr` bridging into chronos). This is
+the feature that changes what people can *build* with nim-ffi.
+
+### 2. Richer error model than `string`
+`Result[T, string]` crosses today. Allow `Result[T, E]` where `E` is a typed
+`{.ffi.}` struct, so every language surfaces structured errors (codes, fields)
+instead of parsing text. Small change to the macro's return handling.
+
+### 3. Streaming / multi-shot results
+A proc that yields *many* values (an `AsyncStream`) mapping to host-native
+iterators: Kotlin `Flow`, Swift `AsyncSequence`, Rust `Stream`, JS async
+iterators. Turns nim-ffi from RPC into a reactive core.
+
+### 4. ABI self-descriptor symbol
+Export `<lib>_abi_descriptor()` returning the schema (CBOR/JSON) so a host can
+validate compatibility at load time. Addresses the deferred CBOR wire-versioning
+concern.
+
+## Cross-cutting decisions
+
+- **ABI format selection** ([design-abi-format.md](design-abi-format.md)) — per-proc
+  pragma arg `{.ffi: raw.}` / `{.ffi: cbor.}` (default native/`raw`), with a
+  library-wide `declareLibrary(defaultAbiFormat = …)` override. Global compile
+  flag discarded. Applies to `{.ffiHost.}` too. Direction decided, design only.
+
+## Adjacent / parallel tracks (already discussed elsewhere)
+
+- **seq/Option + multi-struct param marshaling parity** for the Swift (#59) and
+  Kotlin (#60) generators — `go.nim` is the reference (it already does this).
+- **Typed events on Swift/Kotlin** — the JNI-thread-attach-into-JVM case for
+  Kotlin is the hard part.
+- **Async idiom mapping** — `Future[T]` → Promise / `async`/`await` / `suspend`
+  / `impl Future`, so callers `await` instead of blocking on a semaphore.
+- **WASM Component Model (WIT) emitter** — emit a `.wit` so any host consumes the
+  interface without bespoke glue.

examples/host_demo/go_bindings/.gitignore

@@ -0,0 +1,3 @@
+*.dylib
+*.so
+example/example

examples/host_demo/go_bindings/Makefile

@@ -0,0 +1,35 @@
+# Build the Nim dylib next to the generated Go package and run the host-callback
+# example.
+#
+#   make run     # build libhost_demo + run the example
+#   make clean
+#
+# The generated package's cgo directives use ${SRCDIR}, so the library only has
+# to sit in this directory (-L/-rpath point here). It is compiled from the repo
+# root so the vendored Nimble dependencies resolve.
+
+REPO_ROOT := $(abspath ../../..)
+NIM_SRC   := $(REPO_ROOT)/examples/host_demo/host_demo.nim
+
+UNAME_S := $(shell uname -s)
+ifeq ($(UNAME_S),Darwin)
+  LIBNAME := libhost_demo.dylib
+else
+  LIBNAME := libhost_demo.so
+endif
+
+NIMFLAGS := --mm:orc -d:chronicles_log_level=WARN --app:lib --noMain \
+            --nimMainPrefix:libhost_demo
+
+.PHONY: all run clean
+
+all: $(LIBNAME)
+
+$(LIBNAME):
+	cd $(REPO_ROOT) && nim c $(NIMFLAGS) -o:$(CURDIR)/$(LIBNAME) $(NIM_SRC)
+
+run: $(LIBNAME)
+	cd example && go run .
+
+clean:
+	rm -f $(LIBNAME) example/example

examples/host_demo/go_bindings/example/go.mod

@@ -0,0 +1,7 @@
+module example
+
+go 1.21
+
+require host_demo v0.0.0
+
+replace host_demo => ../

examples/host_demo/go_bindings/example/main.go

@@ -0,0 +1,37 @@
+// Go example for a {.ffiHost.} host callback.
+//
+// `fetchToken` is implemented HERE (the Go app) and registered with
+// SetFetchToken. When we call UseToken, the Nim library calls back into this Go
+// closure for a token — the closure runs on a goroutine the generated wrapper
+// spawns (never blocking the FFI thread) and answers via host_complete.
+package main
+
+import (
+	"fmt"
+	"log"
+
+	hd "host_demo"
+)
+
+func main() {
+	node, err := hd.NewHost_demo()
+	if err != nil {
+		log.Fatalf("create: %v", err)
+	}
+	defer node.Destroy()
+
+	// The host's implementation of the {.ffiHost.} fetchToken.
+	node.SetFetchToken(func(key string) (string, error) {
+		return "TOK-" + key, nil
+	})
+
+	res, err := node.UseToken("session")
+	if err != nil {
+		log.Fatalf("useToken: %v", err)
+	}
+	fmt.Printf("result: %s\n", res)
+	if res != "token[TOK-session]" {
+		log.Fatalf("unexpected result: %q", res)
+	}
+	fmt.Println("OK")
+}

examples/host_demo/go_bindings/go.mod

@@ -0,0 +1,3 @@
+module host_demo
+
+go 1.21

examples/host_demo/go_bindings/host_demo.go

@@ -0,0 +1,173 @@
+// Code generated by nim-ffi Go codegen. DO NOT EDIT.
+package host_demo
+
+/*
+#cgo CFLAGS: -I${SRCDIR}
+#cgo LDFLAGS: -L${SRCDIR} -lhost_demo -Wl,-rpath,${SRCDIR}
+#include "host_demo.h"
+#include <stdlib.h>
+#include <string.h>
+#include <pthread.h>
+
+extern void host_demoGoEvent(int ret, char* msg, size_t len, void* userData);
+extern void host_demoHostTrampoline(uint64_t callId, char* req, size_t reqLen, void* userData);
+static int host_demoRegisterHost(void* ctx, const char* name, void* ud) {
+  return host_demo_register_host_fn(ctx, name, (FFIHostFn)host_demoHostTrampoline, ud);
+}
+
+typedef struct {
+  int ret; char* msg; size_t len; int done;
+  pthread_mutex_t mu; pthread_cond_t cv;
+} Host_demoResp;
+
+static Host_demoResp* host_demoRespNew() {
+  Host_demoResp* r = (Host_demoResp*)calloc(1, sizeof(Host_demoResp));
+  pthread_mutex_init(&r->mu, NULL); pthread_cond_init(&r->cv, NULL);
+  return r;
+}
+static void host_demoRespFree(Host_demoResp* r) {
+  if (!r) return;
+  if (r->msg) free(r->msg);
+  pthread_mutex_destroy(&r->mu); pthread_cond_destroy(&r->cv); free(r);
+}
+static int host_demoRespRet(Host_demoResp* r) { return r->ret; }
+static char* host_demoRespMsg(Host_demoResp* r) { return r->msg; }
+static size_t host_demoRespLen(Host_demoResp* r) { return r->len; }
+
+static void host_demoRespCb(int ret, const char* msg, size_t len, void* ud) {
+  Host_demoResp* r = (Host_demoResp*)ud;
+  pthread_mutex_lock(&r->mu);
+  r->ret = ret;
+  // Native ABI: (msg, len) is the raw result (RET_OK) or error (RET_ERR).
+  // Copy it so it survives past the callback.
+  char* e = (char*)malloc(len + 1); if (e) { memcpy(e, msg, len); e[len] = 0; }
+  r->msg = e; r->len = len;
+  r->done = 1; pthread_cond_signal(&r->cv); pthread_mutex_unlock(&r->mu);
+}
+static void host_demoRespWait(Host_demoResp* r) {
+  pthread_mutex_lock(&r->mu);
+  while (!r->done) pthread_cond_wait(&r->cv, &r->mu);
+  pthread_mutex_unlock(&r->mu);
+}
+
+static void* host_demoCall_demo_create(Host_demoResp* r) {
+  void* ctx = demo_create(host_demoRespCb, r);
+  host_demoRespWait(r);
+  return ctx;
+}
+static int host_demoCall_use_token(void* ctx, const char* key, Host_demoResp* r) {
+  int rc = use_token(ctx, host_demoRespCb, r, key);
+  if (rc == RET_OK) host_demoRespWait(r);
+  return rc;
+}
+static int host_demoCall_demo_destroy(void* ctx) { return demo_destroy(ctx); }
+static uint64_t host_demoRegisterEvents(void* ctx) { return host_demo_add_event_listener(ctx, "", (FFICallBack)host_demoGoEvent, ctx); }
+*/
+import "C"
+
+import (
+	"errors"
+	"runtime/cgo"
+	"sync"
+	"unsafe"
+)
+
+type resultSlot struct {
+	val  any
+	err  error
+	done chan struct{}
+}
+
+type Host_demoNode struct {
+	ctx unsafe.Pointer
+}
+
+// goStr extracts and frees the captured response string.
+func respStr(r *C.Host_demoResp) string {
+	return C.GoStringN(C.host_demoRespMsg(r), C.int(C.host_demoRespLen(r)))
+}
+
+var (
+	eventMu      sync.Mutex
+	eventHandler func(string)
+)
+
+// SetEventHandler installs the catch-all handler for library-initiated
+// events (delivered as raw JSON strings).
+func (n *Host_demoNode) SetEventHandler(h func(string)) {
+	eventMu.Lock()
+	eventHandler = h
+	eventMu.Unlock()
+	C.host_demoRegisterEvents(n.ctx)
+}
+
+//export host_demoGoEvent
+func host_demoGoEvent(ret C.int, msg *C.char, length C.size_t, userData unsafe.Pointer) {
+	eventMu.Lock()
+	h := eventHandler
+	eventMu.Unlock()
+	if h != nil && ret == C.RET_OK {
+		h(C.GoStringN(msg, C.int(length)))
+	}
+}
+
+type hostEntry struct {
+	ctx unsafe.Pointer
+	fn  func(string) (string, error)
+}
+
+//export host_demoHostTrampoline
+func host_demoHostTrampoline(callId C.uint64_t, req *C.char, reqLen C.size_t, userData unsafe.Pointer) {
+	e := cgo.Handle(uintptr(userData)).Value().(hostEntry)
+	reqStr := C.GoStringN(req, C.int(reqLen))
+	go func() {
+		res, err := e.fn(reqStr)
+		if err != nil {
+			msg := err.Error()
+			cmsg := C.CString(msg)
+			C.host_demo_host_complete(e.ctx, callId, C.int(C.RET_ERR), cmsg, C.size_t(len(msg)))
+			C.free(unsafe.Pointer(cmsg))
+		} else {
+			cmsg := C.CString(res)
+			C.host_demo_host_complete(e.ctx, callId, C.int(C.RET_OK), cmsg, C.size_t(len(res)))
+			C.free(unsafe.Pointer(cmsg))
+		}
+	}()
+}
+
+// SetFetchToken registers the host implementation of the 'fetch_token' {.ffiHost.} call.
+func (n *Host_demoNode) SetFetchToken(fn func(string) (string, error)) {
+	handle := cgo.NewHandle(hostEntry{ctx: n.ctx, fn: fn})
+	cname := C.CString("fetch_token")
+	C.host_demoRegisterHost(n.ctx, cname, unsafe.Pointer(handle))
+	C.free(unsafe.Pointer(cname))
+}
+
+func NewHost_demo() (*Host_demoNode, error) {
+	r := C.host_demoRespNew()
+	defer C.host_demoRespFree(r)
+	ctx := C.host_demoCall_demo_create(r)
+	if C.host_demoRespRet(r) != C.RET_OK {
+		return nil, errors.New(respStr(r))
+	}
+	return &Host_demoNode{ctx: ctx}, nil
+}
+
+func (n *Host_demoNode) UseToken(key string) (string, error) {
+	c_key := C.CString(key)
+	defer C.free(unsafe.Pointer(c_key))
+	r := C.host_demoRespNew()
+	defer C.host_demoRespFree(r)
+	C.host_demoCall_use_token(n.ctx, c_key, r)
+	if C.host_demoRespRet(r) != C.RET_OK {
+		return "", errors.New(respStr(r))
+	}
+	return respStr(r), nil
+}
+
+func (n *Host_demoNode) Destroy() error {
+	if C.host_demoCall_demo_destroy(n.ctx) != C.RET_OK {
+		return errors.New("host_demo destroy failed")
+	}
+	return nil
+}

examples/host_demo/go_bindings/host_demo.h

@@ -0,0 +1,58 @@
+// Generated by nim-ffi C codegen. Do not edit by hand.
+//
+// Native (zero-serialization) C ABI. Each call delivers its result to the
+// callback. On RET_OK:
+//   - string-returning procs: (msg, len) is the raw string bytes (not
+//     NUL-terminated; use len).
+//   - struct-returning procs: msg is a pointer to the returned C struct — cast
+//     it to `const <Type>*` (len is sizeof). It is valid ONLY for the duration
+//     of the callback; copy out anything you need before returning. The library
+//     deep-frees it right after the callback (you free nothing).
+// On RET_ERR, (msg, len) is the raw error text. A `<name>_cbor` variant of each
+// proc also exists for generic/cross-language callers that prefer CBOR.
+#ifndef NIM_FFI_GEN_HOST_DEMO_H
+#define NIM_FFI_GEN_HOST_DEMO_H
+
+#include <stddef.h>
+#include <stdint.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#ifndef NIM_FFI_RET_CODES
+#define NIM_FFI_RET_CODES
+#define RET_OK 0
+#define RET_ERR 1
+#define RET_MISSING_CALLBACK 2
+#endif
+
+#ifndef NIM_FFI_CALLBACK_T
+#define NIM_FFI_CALLBACK_T
+typedef void (*FFICallBack)(int callerRet, const char *msg, size_t len, void *userData);
+#endif
+
+
+
+void *demo_create(FFICallBack callback, void *userData);
+
+int demo_destroy(void *ctx);
+
+int use_token(void *ctx, FFICallBack callback, void *userData, const char* key);
+
+uint64_t host_demo_add_event_listener(void *ctx, const char *eventName, FFICallBack callback, void *userData);
+int host_demo_remove_event_listener(void *ctx, uint64_t listenerId);
+
+// --- host callbacks ({.ffiHost.}) — host-implemented functions --------
+#ifndef NIM_FFI_HOST_FN_T
+#define NIM_FFI_HOST_FN_T
+typedef void (*FFIHostFn)(uint64_t callId, const char *req, size_t reqLen, void *userData);
+#endif
+int host_demo_register_host_fn(void *ctx, const char *name, FFIHostFn fn, void *userData);
+int host_demo_host_complete(void *ctx, uint64_t callId, int ret, const char *msg, size_t len);
+
+#ifdef __cplusplus
+} // extern "C"
+#endif
+
+#endif /* NIM_FFI_GEN_HOST_DEMO_H */

examples/host_demo/host_demo.nim

@@ -0,0 +1,31 @@
+## Minimal example exercising a {.ffiHost.} host callback end-to-end from Go.
+##
+## `fetchToken` is implemented by the *host* (the Go app); `useToken` is a normal
+## {.ffi.} method the host calls, which in turn asks the host for a token via
+## `fetchToken` and awaits it. This proves the inverted call direction across the
+## real FFI boundary with the generated Go wrapper.
+
+import ffi, chronos, results
+
+type Demo = object
+
+declareLibrary("host_demo", Demo)
+
+# Ctor first: the {.ffiCtor.} macro declares the per-lib FFI pool that the
+# {.ffi.} method below references.
+proc demoCreate(): Future[Result[Demo, string]] {.ffiCtor.} =
+  return ok(Demo())
+
+proc demoDestroy(d: Demo) {.ffiDtor.} =
+  discard
+
+# Host-implemented: the Go app registers this with SetFetchToken.
+proc fetchToken(key: string): Future[Result[string, string]] {.ffiHost.}
+
+# A {.ffi.} method the host calls; it asks the host for a token and wraps it.
+proc useToken(d: Demo, key: string): Future[Result[string, string]] {.ffi.} =
+  let tok = (await fetchToken(key)).valueOr:
+    return err("host error: " & error)
+  return ok("token[" & tok & "]")
+
+genBindings()

examples/timer/c_bindings/my_timer.h

@@ -114,6 +114,14 @@ int my_timer_destroy(void *ctx);
 uint64_t my_timer_add_event_listener(void *ctx, const char *eventName, FFICallBack callback, void *userData);
 int my_timer_remove_event_listener(void *ctx, uint64_t listenerId);
 
+// --- host callbacks ({.ffiHost.}) — host-implemented functions --------
+#ifndef NIM_FFI_HOST_FN_T
+#define NIM_FFI_HOST_FN_T
+typedef void (*FFIHostFn)(uint64_t callId, const char *req, size_t reqLen, void *userData);
+#endif
+int my_timer_register_host_fn(void *ctx, const char *name, FFIHostFn fn, void *userData);
+int my_timer_host_complete(void *ctx, uint64_t callId, int ret, const char *msg, size_t len);
+
 #ifdef __cplusplus
 } // extern "C"
 #endif

examples/timer/go_bindings/my_timer.h

@@ -114,6 +114,14 @@ int my_timer_destroy(void *ctx);
 uint64_t my_timer_add_event_listener(void *ctx, const char *eventName, FFICallBack callback, void *userData);
 int my_timer_remove_event_listener(void *ctx, uint64_t listenerId);
 
+// --- host callbacks ({.ffiHost.}) — host-implemented functions --------
+#ifndef NIM_FFI_HOST_FN_T
+#define NIM_FFI_HOST_FN_T
+typedef void (*FFIHostFn)(uint64_t callId, const char *req, size_t reqLen, void *userData);
+#endif
+int my_timer_register_host_fn(void *ctx, const char *name, FFIHostFn fn, void *userData);
+int my_timer_host_complete(void *ctx, uint64_t callId, int ret, const char *msg, size_t len);
+
 #ifdef __cplusplus
 } // extern "C"
 #endif

ffi.nim

@@ -3,12 +3,12 @@ import chronos, chronicles
 import
   ffi/internal/[ffi_library, ffi_macro],
   ffi/[
-    alloc, ffi_types, ffi_events, ffi_context, ffi_context_pool, ffi_thread_request,
-    cbor_serial,
+    alloc, ffi_types, ffi_events, ffi_host, ffi_context, ffi_context_pool,
+    ffi_thread_request, cbor_serial,
   ]
 
 export atomics, tables
 export chronos, chronicles
 export
-  atomics, alloc, ffi_library, ffi_macro, ffi_types, ffi_events, ffi_context,
-  ffi_context_pool, ffi_thread_request, cbor_serial
+  atomics, alloc, ffi_library, ffi_macro, ffi_types, ffi_events, ffi_host,
+  ffi_context, ffi_context_pool, ffi_thread_request, cbor_serial

ffi/codegen/c.nim

@@ -175,6 +175,28 @@ proc generateCHeader*(
     "int " & libName & "_remove_event_listener(void *ctx, uint64_t listenerId);"
   )
   lines.add("")
+
+  # Host callbacks ({.ffiHost.}): the host registers an implementation, the
+  # library invokes it with a callId + raw request, and the host answers by callId
+  # (from any thread) via host_complete. Always exported, like the event ABI.
+  lines.add(
+    "// --- host callbacks ({.ffiHost.}) — host-implemented functions --------"
+  )
+  lines.add("#ifndef NIM_FFI_HOST_FN_T")
+  lines.add("#define NIM_FFI_HOST_FN_T")
+  lines.add(
+    "typedef void (*FFIHostFn)(uint64_t callId, const char *req, size_t reqLen, void *userData);"
+  )
+  lines.add("#endif")
+  lines.add(
+    "int " & libName &
+      "_register_host_fn(void *ctx, const char *name, FFIHostFn fn, void *userData);"
+  )
+  lines.add(
+    "int " & libName &
+      "_host_complete(void *ctx, uint64_t callId, int ret, const char *msg, size_t len);"
+  )
+  lines.add("")
   lines.add("#ifdef __cplusplus")
   lines.add("} // extern \"C\"")
   lines.add("#endif")

ffi/codegen/go.nim

@@ -334,6 +334,7 @@ proc generateGoFile*(
     types: seq[FFITypeMeta],
     libName: string,
     events: seq[FFIEventMeta] = @[],
+    hosts: seq[FFIHostMeta] = @[],
 ): string =
   let nodeType = capitalizeFirstLetter(libName) & "Node"
   let respT = capitalizeFirstLetter(libName) & "Resp"
@@ -370,6 +371,25 @@ proc generateGoFile*(
   L.add(
     "extern void " & libName & "GoEvent(int ret, char* msg, size_t len, void* userData);"
   )
+  # Host callbacks ({.ffiHost.}): a single exported Go trampoline backs every
+  # registered host fn; the static helper hands its address to register_host_fn
+  # (cgo drops const, so the forward decl uses char*).
+  if hosts.len > 0:
+    L.add(
+      "extern void " & libName &
+        "HostTrampoline(uint64_t callId, char* req, size_t reqLen, void* userData);"
+    )
+    L.add(
+      "static int " & libName &
+        "RegisterHost(void* ctx, const char* name, void* ud) {"
+    )
+    # cgo exports the trampoline with `char*` (it drops const); cast to FFIHostFn
+    # so the function-pointer types match.
+    L.add(
+      "  return " & libName & "_register_host_fn(ctx, name, (FFIHostFn)" & libName &
+        "HostTrampoline, ud);"
+    )
+    L.add("}")
   # One exported Go result callback per struct-returning proc (it reads the typed
   # return POD in-callback). Forward-declared here so cgo's `char*` shape matches.
   for p in procs:
@@ -559,6 +579,63 @@ proc generateGoFile*(
   L.add("}")
   L.add("")
 
+  # ---- host callbacks ({.ffiHost.}) ----------------------------------------
+  # One exported trampoline serves all host fns; the cgo.Handle in userData
+  # selects which Go closure. The closure runs on a fresh goroutine so the FFI
+  # thread is never blocked (the non-blocking contract), then answers by callId.
+  if hosts.len > 0:
+    L.add("type hostEntry struct {")
+    L.add("\tctx unsafe.Pointer")
+    L.add("\tfn  func(string) (string, error)")
+    L.add("}")
+    L.add("")
+    L.add("//export " & libName & "HostTrampoline")
+    L.add(
+      "func " & libName &
+        "HostTrampoline(callId C.uint64_t, req *C.char, reqLen C.size_t, userData unsafe.Pointer) {"
+    )
+    L.add("\te := cgo.Handle(uintptr(userData)).Value().(hostEntry)")
+    L.add("\treqStr := C.GoStringN(req, C.int(reqLen))")
+    L.add("\tgo func() {")
+    L.add("\t\tres, err := e.fn(reqStr)")
+    L.add("\t\tif err != nil {")
+    L.add("\t\t\tmsg := err.Error()")
+    L.add("\t\t\tcmsg := C.CString(msg)")
+    L.add(
+      "\t\t\tC." & libName &
+        "_host_complete(e.ctx, callId, C.int(C.RET_ERR), cmsg, C.size_t(len(msg)))"
+    )
+    L.add("\t\t\tC.free(unsafe.Pointer(cmsg))")
+    L.add("\t\t} else {")
+    L.add("\t\t\tcmsg := C.CString(res)")
+    L.add(
+      "\t\t\tC." & libName &
+        "_host_complete(e.ctx, callId, C.int(C.RET_OK), cmsg, C.size_t(len(res)))"
+    )
+    L.add("\t\t\tC.free(unsafe.Pointer(cmsg))")
+    L.add("\t\t}")
+    L.add("\t}()")
+    L.add("}")
+    L.add("")
+    for h in hosts:
+      let setName = "Set" & capitalizeFirstLetter(h.nimProcName)
+      L.add(
+        "// " & setName & " registers the host implementation of the '" & h.wireName &
+          "' {.ffiHost.} call."
+      )
+      L.add(
+        "func (n *" & nodeType & ") " & setName &
+          "(fn func(string) (string, error)) {"
+      )
+      L.add("\thandle := cgo.NewHandle(hostEntry{ctx: n.ctx, fn: fn})")
+      L.add("\tcname := C.CString(\"" & h.wireName & "\")")
+      L.add(
+        "\tC." & libName & "RegisterHost(n.ctx, cname, unsafe.Pointer(handle))"
+      )
+      L.add("\tC.free(unsafe.Pointer(cname))")
+      L.add("}")
+      L.add("")
+
   # ---- constructor ---------------------------------------------------------
   if haveCtor:
     let (goParams, conv, callArgs) = goParamConv(ctor.extraParams, types)
@@ -690,9 +767,11 @@ proc generateGoBindings*(
     outputDir: string,
     nimSrcRelPath: string,
     events: seq[FFIEventMeta] = @[],
+    hosts: seq[FFIHostMeta] = @[],
 ) =
   writeFile(
-    outputDir / (libName & ".go"), generateGoFile(procs, types, libName, events)
+    outputDir / (libName & ".go"),
+    generateGoFile(procs, types, libName, events, hosts),
   )
   # cgo `#include "<lib>.h"` resolves against this package directory, so emit the
   # native C header here too — the Go package is then self-contained (just stage

ffi/codegen/meta.nim

@@ -40,10 +40,24 @@ type
     libName*: string
     payloadTypeName*: string
 
+  FFIHostMeta* = object
+    ## Host-provided function declared with `{.ffiHost.}` — the host implements
+    ## it and a `{.ffi.}` handler awaits it. `wireName` is the snake_case name
+    ## the host registers under. First slice: one `string` arg, `string` return;
+    ## `argName`/`argTypeName`/`returnTypeName` carry the shape so generators can
+    ## emit a typed wrapper.
+    wireName*: string
+    nimProcName*: string
+    libName*: string
+    argName*: string
+    argTypeName*: string
+    returnTypeName*: string
+
 # Compile-time registries populated by the macros
 var ffiProcRegistry* {.compileTime.}: seq[FFIProcMeta]
 var ffiTypeRegistry* {.compileTime.}: seq[FFITypeMeta]
 var ffiEventRegistry* {.compileTime.}: seq[FFIEventMeta]
+var ffiHostRegistry* {.compileTime.}: seq[FFIHostMeta]
 var currentLibName* {.compileTime.}: string
 
 # Target language for binding generation; override with -d:targetLang=cpp

ffi/ffi_context.nim

@@ -3,10 +3,10 @@
 import std/[atomics, locks, json, tables]
 import chronicles, chronos, chronos/threadsync, taskpools/channels_spsc_single, results
 import
-  ./ffi_types, ./ffi_events, ./ffi_thread_request, ./internal/ffi_macro, ./logging,
-  ./cbor_serial
+  ./ffi_types, ./ffi_events, ./ffi_host, ./ffi_thread_request,
+  ./internal/ffi_macro, ./logging, ./cbor_serial
 
-export ffi_events
+export ffi_events, ffi_host
 
 type FFIContext*[T] = object
   myLib*: ptr T
@@ -28,6 +28,12 @@ type FFIContext*[T] = object
     # blocked event loop cannot hang the caller forever
   userData*: pointer
   eventRegistry*: FFIEventRegistry
+  hostRegistry*: FFIHostRegistry
+    # host-provided functions a {.ffiHost.} proc dispatches to (roadmap #1)
+  pendingTable*: FFIPendingTable
+    # in-flight {.ffiHost.} calls: callId -> the chronos Future being awaited
+  completionQueue: FFICompletionQueue
+    # host answers parked from any thread, drained + completed on the FFI thread
   running: Atomic[bool] # To control when the threads are running
   registeredRequests: ptr Table[cstring, FFIRequestProc]
     # Pointer to with the registered requests at compile time
@@ -86,6 +92,17 @@ proc sendRequestToFFIThread*(
   ## process proc.
   return ok()
 
+proc completeHostCall*[T](
+    ctx: ptr FFIContext[T], callId: uint64, ret: cint, msg: ptr cchar, len: csize_t
+) {.raises: [].} =
+  ## Backs `<lib>_host_complete`: the host delivers a `{.ffiHost.}` answer by
+  ## callId. Callable from ANY thread — it only parks the result (GC-free) and
+  ## wakes the FFI loop via the existing `reqSignal`; the future is completed on
+  ## the FFI thread when the loop drains the queue. A callId with no pending call
+  ## (late / double completion) is drained and dropped, never a crash.
+  pushCompletion(ctx[].completionQueue, callId, ret, msg, len)
+  discard ctx.reqSignal.fireSync()
+
 type Foo = object
 registerReqFFI(WatchdogReq, foo: ptr Foo):
   proc(): Future[Result[string, string]] {.async.} =
@@ -207,6 +224,8 @@ proc processRequest[T](
 proc ffiThreadBody[T](ctx: ptr FFIContext[T]) {.thread.} =
   ## FFI thread body that attends library user API requests
   ffiCurrentEventRegistry = addr ctx[].eventRegistry
+  ffiCurrentHostRegistry = addr ctx[].hostRegistry
+  ffiCurrentPendingTable = addr ctx[].pendingTable
   onFFIThread = true
 
   logging.setupLog(logging.LogLevel.DEBUG, logging.LogFormat.TEXT)
@@ -242,6 +261,12 @@ proc ffiThreadBody[T](ctx: ptr FFIContext[T]) {.thread.} =
       reapCompleted()
 
       let gotSignal = await ctx.reqSignal.wait().withTimeout(100.milliseconds)
+
+      ## Drain host-call answers every iteration (reqSignal is fired by
+      ## completeHostCall too): completing each awaited Future here, on the loop
+      ## thread, satisfies chronos's single-thread invariant.
+      drainCompletions(ctx[].completionQueue, ctx[].pendingTable)
+
       if not gotSignal:
         continue
 
@@ -264,6 +289,12 @@ proc ffiThreadBody[T](ctx: ptr FFIContext[T]) {.thread.} =
     ## before we exit. Without this, abandoning a future mid-await would
     ## leak the request allocations (visible to LSan; previously hidden
     ## because Nim's pool allocator kept the chunks alive in the process).
+    ##
+    ## Fail every outstanding {.ffiHost.} call first: a handler awaiting a host
+    ## answer that never arrives would otherwise make `allFutures(pending)` hang
+    ## forever. Then drain any answer that raced in during shutdown.
+    failAllPending(ctx[].pendingTable, "FFIContext shutting down")
+    drainCompletions(ctx[].completionQueue, ctx[].pendingTable)
     reapCompleted()
     if pending.len > 0:
       try:
@@ -280,6 +311,9 @@ proc cleanUpResources[T](ctx: ptr FFIContext[T]): Result[void, string] =
     freeShared(ctx)
   ctx.lock.deinitLock()
   deinitEventRegistry(ctx[].eventRegistry)
+  deinitHostRegistry(ctx[].hostRegistry)
+  deinitPendingTable(ctx[].pendingTable)
+  deinitCompletionQueue(ctx[].completionQueue)
   when defined(gcRefc):
     ## ThreadSignalPtr.close() is intentionally skipped under --mm:refc.
     ##
@@ -318,6 +352,9 @@ proc initContextResources*[T](ctx: ptr FFIContext[T]): Result[void, string] =
   ## is responsible for releasing the slot (freeShared or pool.releaseSlot).
   ctx.lock.initLock()
   initEventRegistry(ctx[].eventRegistry)
+  initHostRegistry(ctx[].hostRegistry)
+  initPendingTable(ctx[].pendingTable)
+  initCompletionQueue(ctx[].completionQueue)
 
   var success = false
   defer:

ffi/ffi_host.nim

@@ -0,0 +1,280 @@
+## Registry + in-flight table backing typed host callbacks (`{.ffiHost.}`).
+##
+## This is the data-structure layer of roadmap item #1 (see
+## docs/design-host-callbacks.md). It owns two per-context concerns and nothing
+## else — the FFI-thread completion bridge and the `{.ffiHost.}` macro land in
+## later increments and build on these primitives:
+##
+## 1. `FFIHostRegistry` — maps a wire name (e.g. "fetch_profile") to the host's
+##    registered function pointer + userData. A missing entry is a normal,
+##    non-fatal outcome (the imported proc resolves to an error), never a crash.
+## 2. `FFIPendingTable` — maps a monotonic `callId` to the chronos `Future` an
+##    awaiting `{.ffiHost.}` proc is blocked on. The host answers later (on any
+##    thread) by `callId`; the FFI thread drains and completes the future.
+##
+## Both structures are lock-guarded so a host thread (registering / completing)
+## and the FFI thread (looking up / completing) can touch them concurrently.
+## Futures themselves are only ever completed on the FFI thread — `complete*`
+## here is called from the loop drain, not from the host thread directly.
+
+import std/[locks, tables]
+import chronos
+import ./ffi_types, ./alloc
+
+# ---------------------------------------------------------------------------
+# Host function pointer
+# ---------------------------------------------------------------------------
+
+type FFIHostFn* = proc(
+  callId: uint64, req: ptr cchar, reqLen: csize_t, userData: pointer
+) {.cdecl, gcsafe, raises: [].}
+  ## A host-implemented function. `req`/`reqLen` carry the marshaled request
+  ## (valid only for the duration of the call — the host copies what it needs).
+  ## The host answers asynchronously via `<lib>_host_complete(ctx, callId, …)`.
+
+type HostResult* = object
+  ## The raw outcome the host delivered for one callId: a return code plus the
+  ## response bytes (native POD or CBOR — decoded by the awaiting proc).
+  ret*: cint
+  bytes*: seq[byte]
+
+proc okResult*(bytes: seq[byte]): HostResult =
+  return HostResult(ret: RET_OK, bytes: bytes)
+
+proc resultText*(res: HostResult): string =
+  ## The payload bytes as a string — used by the raw `{.ffiHost.}` path for both
+  ## the success value (string return) and the error text.
+  var s = newString(res.bytes.len)
+  if res.bytes.len > 0:
+    copyMem(addr s[0], unsafeAddr res.bytes[0], res.bytes.len)
+  return s
+
+proc errResult*(msg: string): HostResult =
+  var b = newSeq[byte](msg.len)
+  if msg.len > 0:
+    copyMem(addr b[0], unsafeAddr msg[0], msg.len)
+  return HostResult(ret: RET_ERR, bytes: b)
+
+# ---------------------------------------------------------------------------
+# Host function registry
+# ---------------------------------------------------------------------------
+
+type
+  HostFnEntry = object
+    fn: FFIHostFn
+    userData: pointer
+
+  FFIHostRegistry* = object
+    lock: Lock
+    fns: Table[string, HostFnEntry]
+
+proc initHostRegistry*(reg: var FFIHostRegistry) =
+  ## Call exactly once on the owning thread before sharing. The embedded `Lock`
+  ## wraps a platform primitive that cannot be safely double-initialised.
+  reg.lock.initLock()
+  reg.fns = initTable[string, HostFnEntry]()
+
+proc deinitHostRegistry*(reg: var FFIHostRegistry) =
+  ## Mirror of `initHostRegistry`; call once, after all other threads have
+  ## stopped using the registry. Resets the GC-managed table so pool slot reuse
+  ## on another thread doesn't run a destructor against this thread's heap.
+  reg.lock.deinitLock()
+  reg.fns = default(Table[string, HostFnEntry])
+
+proc registerHostFn*(
+    reg: var FFIHostRegistry, name: string, fn: FFIHostFn, userData: pointer
+): bool {.raises: [].} =
+  ## Registers (or replaces) the host implementation for `name`. Returns false
+  ## if `fn` is nil — the only documented failure — so a foreign caller can
+  ## treat a nil pointer as "unregister intent" without crashing.
+  if fn.isNil():
+    withLock reg.lock:
+      reg.fns.del(name)
+    return false
+  withLock reg.lock:
+    reg.fns[name] = HostFnEntry(fn: fn, userData: userData)
+  return true
+
+proc lookupHostFn*(
+    reg: var FFIHostRegistry, name: string
+): tuple[fn: FFIHostFn, userData: pointer, found: bool] {.raises: [].} =
+  ## Returns the registered `(fn, userData)` for `name`, or `found == false` when
+  ## no host implementation exists — the awaiting proc turns that into an error.
+  var entry: HostFnEntry
+  var got = false
+  withLock reg.lock:
+    if reg.fns.hasKey(name):
+      entry = reg.fns.getOrDefault(name)
+      got = true
+  return (entry.fn, entry.userData, got)
+
+proc clearHostFns*(reg: var FFIHostRegistry) {.raises: [].} =
+  withLock reg.lock:
+    reg.fns.clear()
+
+# ---------------------------------------------------------------------------
+# In-flight completion table
+# ---------------------------------------------------------------------------
+
+type FFIPendingTable* = object
+  lock: Lock
+  nextCallId: uint64 ## Monotonic; 0 is reserved as "invalid", callIds start at 1.
+  pending: Table[uint64, Future[HostResult]]
+
+# Set by the FFI thread at startup (see ffi_context.ffiThreadBody) so the body a
+# `{.ffiHost.}` macro generates can reach its context's host registry + pending
+# table without threading a ctx pointer through the user's signature.
+var ffiCurrentHostRegistry* {.threadvar.}: ptr FFIHostRegistry
+var ffiCurrentPendingTable* {.threadvar.}: ptr FFIPendingTable
+
+proc initPendingTable*(tbl: var FFIPendingTable) =
+  tbl.lock.initLock()
+  tbl.nextCallId = 0'u64
+  tbl.pending = initTable[uint64, Future[HostResult]]()
+
+proc deinitPendingTable*(tbl: var FFIPendingTable) =
+  tbl.lock.deinitLock()
+  tbl.pending = default(Table[uint64, Future[HostResult]])
+  tbl.nextCallId = 0'u64
+
+proc newPending*(
+    tbl: var FFIPendingTable
+): tuple[callId: uint64, fut: Future[HostResult]] =
+  ## Allocates a callId and registers a fresh, uncompleted future under it. The
+  ## `{.ffiHost.}` proc awaits the returned future; the host answers by callId.
+  let fut = newFuture[HostResult]("ffiHostCall")
+  var assigned: uint64 = 0
+  withLock tbl.lock:
+    tbl.nextCallId.inc()
+    assigned = tbl.nextCallId
+    tbl.pending[assigned] = fut
+  return (assigned, fut)
+
+proc completePending*(
+    tbl: var FFIPendingTable, callId: uint64, res: HostResult
+): bool =
+  ## Completes and removes the future for `callId`. Returns false for an unknown
+  ## or already-completed callId — a late / double completion is dropped, not a
+  ## crash. MUST be called on the FFI (event-loop) thread: it touches the
+  ## chronos future.
+  var fut: Future[HostResult] = nil
+  withLock tbl.lock:
+    if tbl.pending.hasKey(callId):
+      fut = tbl.pending.getOrDefault(callId)
+      tbl.pending.del(callId)
+  if fut.isNil() or fut.finished():
+    return false
+  fut.complete(res)
+  return true
+
+proc failAllPending*(tbl: var FFIPendingTable, msg: string) =
+  ## Completes every outstanding future with an error and clears the table —
+  ## used on context teardown so no awaiting handler is abandoned. FFI thread
+  ## only.
+  var futs: seq[Future[HostResult]] = @[]
+  withLock tbl.lock:
+    for _, fut in tbl.pending:
+      futs.add(fut)
+    tbl.pending.clear()
+  for fut in futs:
+    if not fut.isNil() and not fut.finished():
+      fut.complete(errResult(msg))
+
+proc pendingCount*(tbl: var FFIPendingTable): int {.raises: [].} =
+  var n = 0
+  withLock tbl.lock:
+    n = tbl.pending.len
+  return n
+
+# ---------------------------------------------------------------------------
+# Cross-thread completion queue
+# ---------------------------------------------------------------------------
+#
+# `<lib>_host_complete` runs on the host's thread, but a chronos `Future` can
+# only be completed on the FFI (event-loop) thread. So the host's answer is
+# parked here and drained on the FFI thread. The producer side is **GC-free** —
+# node and payload are `c_malloc`'d (ffiCMalloc / ffiCAllocArray) so no Nim GC
+# runs on the foreign thread — mirroring how the rest of the boundary allocates.
+
+type
+  CompletionNode = object
+    callId: uint64
+    ret: cint
+    buf: ptr UncheckedArray[byte] ## c_malloc'd copy of the host payload (or nil)
+    bufLen: int
+    next: ptr CompletionNode
+
+  FFICompletionQueue* = object
+    lock: Lock
+    head: ptr CompletionNode
+    tail: ptr CompletionNode
+
+proc initCompletionQueue*(q: var FFICompletionQueue) =
+  q.lock.initLock()
+  q.head = nil
+  q.tail = nil
+
+proc pushCompletion*(
+    q: var FFICompletionQueue, callId: uint64, ret: cint, msg: ptr cchar, len: csize_t
+) {.raises: [].} =
+  ## Enqueue one host answer. Safe to call from **any** thread; allocates only
+  ## via c_malloc so it never touches the Nim GC on a foreign thread. The FFI
+  ## thread copies the payload into a `seq[byte]` and frees the node on drain.
+  let node = ffiCMalloc(CompletionNode)
+  node.callId = callId
+  node.ret = ret
+  node.bufLen = int(len)
+  node.next = nil
+  if len > 0'u and not msg.isNil():
+    node.buf = ffiCAllocArray(byte, int(len))
+    copyMem(node.buf, msg, int(len))
+  else:
+    node.buf = nil
+  withLock q.lock:
+    if q.tail.isNil():
+      q.head = node
+    else:
+      q.tail.next = node
+    q.tail = node
+
+proc drainCompletions*(
+    q: var FFICompletionQueue, tbl: var FFIPendingTable
+): int {.discardable.} =
+  ## FFI-thread only. Detaches the whole queue, then for each entry resolves the
+  ## pending future by callId (copying the payload into GC memory here, on the FFI
+  ## thread) and frees the c_malloc'd node. Returns the number drained.
+  var head: ptr CompletionNode = nil
+  withLock q.lock:
+    head = q.head
+    q.head = nil
+    q.tail = nil
+
+  var n = 0
+  while not head.isNil():
+    let node = head
+    head = node.next
+    var b = newSeq[byte](node.bufLen)
+    if node.bufLen > 0:
+      copyMem(addr b[0], node.buf, node.bufLen)
+    discard completePending(tbl, node.callId, HostResult(ret: node.ret, bytes: b))
+    if not node.buf.isNil():
+      ffiCFree(node.buf)
+    ffiCFree(node)
+    inc n
+  return n
+
+proc deinitCompletionQueue*(q: var FFICompletionQueue) =
+  ## Frees any still-queued nodes (their futures are handled separately by
+  ## `failAllPending` on teardown) and releases the lock.
+  var head: ptr CompletionNode = nil
+  withLock q.lock:
+    head = q.head
+    q.head = nil
+    q.tail = nil
+  while not head.isNil():
+    let node = head
+    head = node.next
+    if not node.buf.isNil():
+      ffiCFree(node.buf)
+    ffiCFree(node)
+  q.lock.deinitLock()

ffi/internal/ffi_library.nim

@@ -189,4 +189,64 @@ macro declareLibrary*(libraryName: static[string], libType: untyped): untyped =
     )
   )
 
+  # --- {libraryName}_register_host_fn -------------------------------------
+  # Registers the host's implementation of a {.ffiHost.} proc, keyed by its
+  # snake_case wire name. Returns 0 on success, non-zero on a nil ctx / nil fn.
+  let registerName = libraryName & "_register_host_fn"
+  let registerErr = "error: invalid context in " & registerName
+  let registerBody = quote:
+    var ret: cint = 1
+    if isNil(ctx):
+      echo `registerErr`
+      return ret
+    let resolvedName = if hostFnName.isNil(): "" else: $hostFnName
+    if registerHostFn(ctx[].hostRegistry, resolvedName, fn, userData):
+      ret = 0
+    return ret
+
+  stmts.add(
+    newProc(
+      name = ident(registerName),
+      params = @[
+        ident("cint"),
+        newIdentDefs(ident("ctx"), ctxType),
+        newIdentDefs(ident("hostFnName"), ident("cstring")),
+        newIdentDefs(ident("fn"), ident("FFIHostFn")),
+        newIdentDefs(ident("userData"), ident("pointer")),
+      ],
+      body = registerBody,
+      pragmas = cdeclExportPragma,
+    )
+  )
+
+  # --- {libraryName}_host_complete ----------------------------------------
+  # The host delivers a {.ffiHost.} answer by callId. Callable from ANY thread —
+  # it parks the result and wakes the FFI loop, which completes the awaited
+  # future. `retCode` (not `ret`) avoids colliding with chronos templates under
+  # quote injection, like `listenerId` above.
+  let completeName = libraryName & "_host_complete"
+  let completeErr = "error: invalid context in " & completeName
+  let completeBody = quote:
+    if isNil(ctx):
+      echo `completeErr`
+      return cint(1)
+    completeHostCall(ctx, callId, retCode, msg, msgLen)
+    return cint(0)
+
+  stmts.add(
+    newProc(
+      name = ident(completeName),
+      params = @[
+        ident("cint"),
+        newIdentDefs(ident("ctx"), ctxType),
+        newIdentDefs(ident("callId"), ident("uint64")),
+        newIdentDefs(ident("retCode"), ident("cint")),
+        newIdentDefs(ident("msg"), nnkPtrTy.newTree(ident("cchar"))),
+        newIdentDefs(ident("msgLen"), ident("csize_t")),
+      ],
+      body = completeBody,
+      pragmas = cdeclExportPragma,
+    )
+  )
+
   return stmts

ffi/internal/ffi_macro.nim

@@ -1924,6 +1924,122 @@ macro ffiEvent*(wireName: static[string], prc: untyped): untyped =
     echo withPods.repr
   return withPods
 
+# ---------------------------------------------------------------------------
+# ffiHost — host-provided functions the Nim side can await (roadmap #1)
+# ---------------------------------------------------------------------------
+
+macro ffiHost*(prc: untyped): untyped =
+  ## Declares a function the *host* implements, which a `{.ffi.}` handler can
+  ## call and `await` (the inverse of `{.ffi.}`). The annotated proc has an empty
+  ## body; the macro fills it with the dispatch: look up the host's registered
+  ## implementation, hand it the marshaled request + a callId, and await the
+  ## answer the host delivers (via `<lib>_host_complete`) on the FFI thread.
+  ##
+  ## First slice — raw (zero-serialization) ABI, exactly one `string` parameter,
+  ## returning `Future[Result[string, string]]`:
+  ##
+  ##   proc fetchToken(key: string): Future[Result[string, string]] {.ffiHost.}
+  ##
+  ##   # ...then from inside any {.ffi.} handler:
+  ##   let tok = (await fetchToken("session")).valueOr:
+  ##     return err("host lookup failed: " & error)
+  ##
+  ## Struct params/returns and the `{.ffiHost: cbor.}` format arg are follow-ups
+  ## (see docs/design-host-callbacks.md and docs/design-abi-format.md).
+
+  if prc.kind notin {nnkProcDef, nnkFuncDef}:
+    error("ffiHost must be applied to a proc declaration")
+
+  let procName = prc[0]
+  let formalParams = prc[3]
+
+  if formalParams.len != 2:
+    error(
+      "ffiHost (first pass) supports exactly one `string` parameter; got " &
+        $(formalParams.len - 1)
+    )
+
+  let paramDef = formalParams[1]
+  let argName = paramDef[0]
+  if paramDef[1].kind != nnkIdent or $paramDef[1] != "string":
+    error("ffiHost (first pass) parameter must be `string`, got: " & paramDef[1].repr)
+
+  let retTypeNode = formalParams[0]
+  if retTypeNode.kind != nnkBracketExpr or $retTypeNode[0] != "Future":
+    error(
+      "ffiHost return type must be Future[Result[string, string]], got: " &
+        retTypeNode.repr
+    )
+  let resultInner = retTypeNode[1]
+  if resultInner.kind != nnkBracketExpr or $resultInner[0] != "Result" or
+      $resultInner[1] != "string":
+    error(
+      "ffiHost (first pass) return type must be Future[Result[string, string]], got: " &
+        retTypeNode.repr
+    )
+
+  let procNameStr =
+    block:
+      let raw = $procName
+      if raw.endsWith("*"): raw[0 ..^ 2] else: raw
+  let wireName = camelToSnakeCase(procNameStr)
+  let wireNameLit = newStrLitNode(wireName)
+
+  # Record metadata so the per-language generators can emit an idiomatic wrapper
+  # (register a closure + a trampoline that answers via <lib>_host_complete).
+  ffiHostRegistry.add(
+    FFIHostMeta(
+      wireName: wireName,
+      nimProcName: procNameStr,
+      libName: currentLibName,
+      argName: $argName,
+      argTypeName: "string",
+      returnTypeName: "string",
+    )
+  )
+
+  # The generated async body: resolve the thread-local host context, look up the
+  # registered fn, allocate a pending callId, invoke the host with the raw request
+  # bytes, and await the answer. The host fn is called synchronously here (before
+  # the await) while `argName` is still alive, honouring the "req valid only for
+  # the call" contract.
+  let body = quote do:
+    let ffiReg = ffiCurrentHostRegistry
+    let ffiTbl = ffiCurrentPendingTable
+    if ffiReg.isNil() or ffiTbl.isNil():
+      return err("ffiHost " & `wireNameLit` & ": no host context on this thread")
+    let ffiHit = lookupHostFn(ffiReg[], `wireNameLit`)
+    if not ffiHit.found:
+      return err("ffiHost: host fn '" & `wireNameLit` & "' not registered")
+    let (ffiCallId, ffiFut) = newPending(ffiTbl[])
+    if `argName`.len > 0:
+      ffiHit.fn(
+        ffiCallId, cast[ptr cchar](unsafeAddr `argName`[0]), csize_t(`argName`.len),
+        ffiHit.userData,
+      )
+    else:
+      ffiHit.fn(ffiCallId, nil, 0, ffiHit.userData)
+    let ffiRes = await ffiFut
+    if ffiRes.ret != RET_OK:
+      return err(resultText(ffiRes))
+    return ok(resultText(ffiRes))
+
+  var newParams = newSeq[NimNode]()
+  newParams.add(formalParams[0])
+  newParams.add(paramDef)
+
+  let generated = newProc(
+    name = procName,
+    params = newParams,
+    body = body,
+    procType = prc.kind,
+    pragmas = newTree(nnkPragma, ident("async")),
+  )
+
+  when defined(ffiDumpMacros):
+    echo generated.repr
+  return generated
+
 # ---------------------------------------------------------------------------
 # genBindings — codegen entry point
 # ---------------------------------------------------------------------------
@@ -1986,7 +2102,7 @@ macro genBindings*(
     of "go":
       generateGoBindings(
         ffiProcRegistry, ffiTypeRegistry, libName, outputDir, nimSrcRelPath,
-        ffiEventRegistry,
+        ffiEventRegistry, ffiHostRegistry,
       )
     else:
       error(

tests/unit/test_ffi_host_e2e.nim

@@ -0,0 +1,137 @@
+## End-to-end cross-thread test for {.ffiHost.} (roadmap #1, increment 4).
+##
+## Proves the full bridge under the real FFI thread + the *exported* C ABI:
+##   request -> {.ffi.} handler awaits a {.ffiHost.} call -> library invokes the
+##   host fn ON the FFI thread -> host hands the work to a SEPARATE worker thread
+##   (non-blocking) -> worker answers via the exported <lib>_host_complete ->
+##   reqSignal wakes the loop -> drain completes the future on the loop thread ->
+##   handler resumes -> callback fires.
+##
+## The host answering from a different thread than the FFI loop is the property
+## the in-thread macro test can't cover.
+
+import std/[locks, atomics]
+import unittest2
+import results
+import ffi
+
+type TestLib = object
+
+# NB: this drives the runtime bridge directly (registerHostFn / completeHostCall),
+# not the exported C shims `<lib>_register_host_fn` / `<lib>_host_complete` — those
+# need an --app:lib build (declareLibrary emits an importc NimMain) and are verified
+# separately by the symbol check on the timer library. The shims are thin wrappers
+# over exactly the two procs used here.
+
+# The host implements this; a {.ffi.} handler awaits it.
+proc lookupHost(key: string): Future[Result[string, string]] {.ffiHost.}
+
+# A {.ffi.}-style request whose handler depends on the host's answer.
+registerReqFFI(HostCallRequest, lib: ptr TestLib):
+  proc(key: cstring): Future[Result[string, string]] {.async.} =
+    let v = (await lookupHost($key)).valueOr:
+      return err("host failed: " & error)
+    return ok("got:" & v)
+
+# --- the host, answering on a worker thread --------------------------------
+# The host fn runs on the FFI thread, so it must NOT block: it copies the
+# request and hands (callId, key) to a worker via a channel, then returns. The
+# worker answers later through the exported <lib>_host_complete.
+var gHostJobs: Channel[tuple[callId: uint64, key: string]]
+var gCtx: Atomic[pointer]
+
+proc lookupHostFnImpl(
+    callId: uint64, req: ptr cchar, reqLen: csize_t, userData: pointer
+) {.cdecl, gcsafe, raises: [].} =
+  var key = newString(int(reqLen))
+  if reqLen > 0'u:
+    copyMem(addr key[0], req, int(reqLen))
+  try:
+    gHostJobs.send((callId: callId, key: key))
+  except Exception:
+    discard
+
+proc hostWorker(_: pointer) {.thread.} =
+  while true:
+    let job = gHostJobs.recv()
+    if job.callId == 0'u64: # sentinel: shut down
+      break
+    let answer = "reply:" & job.key
+    completeHostCall(
+      cast[ptr FFIContext[TestLib]](gCtx.load()),
+      job.callId,
+      RET_OK,
+      cast[ptr cchar](unsafeAddr answer[0]),
+      csize_t(answer.len),
+    )
+
+# --- blocking callback capture (same shape as test_ffi_context) -------------
+type CallbackData = object
+  lock: Lock
+  cond: Cond
+  called: bool
+  retCode: cint
+  msg: array[1024, byte]
+  msgLen: int
+
+proc testCallback(
+    retCode: cint, msg: ptr cchar, len: csize_t, userData: pointer
+) {.cdecl, gcsafe, raises: [].} =
+  let d = cast[ptr CallbackData](userData)
+  acquire(d[].lock)
+  d[].retCode = retCode
+  let n = min(int(len), d[].msg.len)
+  if n > 0 and not msg.isNil:
+    copyMem(addr d[].msg[0], msg, n)
+  d[].msgLen = n
+  d[].called = true
+  signal(d[].cond)
+  release(d[].lock)
+
+proc waitCallback(d: var CallbackData) =
+  acquire(d.lock)
+  while not d.called:
+    wait(d.cond, d.lock)
+  release(d.lock)
+
+proc callbackBytes(d: var CallbackData): seq[byte] =
+  var b = newSeq[byte](d.msgLen)
+  if d.msgLen > 0:
+    copyMem(addr b[0], addr d.msg[0], d.msgLen)
+  return b
+
+suite "ffiHost end-to-end (cross-thread)":
+  test "handler awaits a host fn answered from another thread":
+    gHostJobs.open()
+    var pool: FFIContextPool[TestLib]
+    let ctx = pool.createFFIContext().valueOr:
+      assert false, "createFFIContext failed: " & $error
+      return
+    gCtx.store(ctx)
+
+    check registerHostFn(ctx[].hostRegistry, "lookup_host", lookupHostFnImpl, nil)
+
+    var worker: Thread[pointer]
+    createThread(worker, hostWorker, nil)
+
+    var d = CallbackData()
+    d.lock.initLock()
+    d.cond.initCond()
+
+    check sendRequestToFFIThread(
+      ctx, HostCallRequest.ffiNewReq(testCallback, addr d, "session".cstring)
+    )
+      .isOk()
+    waitCallback(d)
+
+    check d.retCode == RET_OK
+    # The {.ffi.} OK payload is CBOR-encoded (registerReqFFI returns seq[byte]).
+    check cborDecode(callbackBytes(d), string).value == "got:reply:session"
+
+    # Shut the worker down, then tear the context down.
+    gHostJobs.send((callId: 0'u64, key: ""))
+    joinThread(worker)
+    d.cond.deinitCond()
+    d.lock.deinitLock()
+    gHostJobs.close()
+    check pool.destroyFFIContext(ctx).isOk()

tests/unit/test_ffi_host_macro.nim

@@ -0,0 +1,101 @@
+## Unit tests for the `{.ffiHost.}` macro (roadmap #1, increment 3) — the
+## generated proc that dispatches to a host-registered function and awaits the
+## answer over the raw (zero-serialization) ABI.
+##
+## These drive the generated proc directly with a synchronous "host": the
+## registered fn completes the pending future inline (its userData carries the
+## pending table), so the await resolves without the full FFI thread. The
+## cross-thread bridge is covered separately by the FFIContext wiring.
+
+import std/strutils
+import unittest2
+import chronos
+import ffi
+
+# A {.ffiHost.} declaration: the host implements `echoHost`, Nim awaits it.
+proc echoHost(s: string): Future[Result[string, string]] {.ffiHost.}
+
+# Synchronous host impls. `userData` carries the pending table so the fn can
+# resolve the token inline (a real host answers later via <lib>_host_complete).
+proc echoFn(
+    token: uint64, req: ptr cchar, reqLen: csize_t, userData: pointer
+) {.cdecl, gcsafe, raises: [].} =
+  let tbl = cast[ptr FFIPendingTable](userData)
+  var b = newSeq[byte](int(reqLen))
+  if reqLen > 0'u:
+    copyMem(addr b[0], req, int(reqLen))
+  discard completePending(tbl[], token, okResult(b))
+
+proc failFn(
+    token: uint64, req: ptr cchar, reqLen: csize_t, userData: pointer
+) {.cdecl, gcsafe, raises: [].} =
+  let tbl = cast[ptr FFIPendingTable](userData)
+  discard completePending(tbl[], token, errResult("host said no"))
+
+suite "ffiHost macro":
+  test "round-trips the value through the registered host fn":
+    var reg: FFIHostRegistry
+    var tbl: FFIPendingTable
+    initHostRegistry(reg)
+    initPendingTable(tbl)
+    defer:
+      deinitHostRegistry(reg)
+      deinitPendingTable(tbl)
+    ffiCurrentHostRegistry = addr reg
+    ffiCurrentPendingTable = addr tbl
+    check registerHostFn(reg, "echo_host", echoFn, addr tbl)
+
+    let r = waitFor echoHost("hello host")
+    check r.isOk
+    check r.get == "hello host"
+
+  test "empty argument is handled":
+    var reg: FFIHostRegistry
+    var tbl: FFIPendingTable
+    initHostRegistry(reg)
+    initPendingTable(tbl)
+    defer:
+      deinitHostRegistry(reg)
+      deinitPendingTable(tbl)
+    ffiCurrentHostRegistry = addr reg
+    ffiCurrentPendingTable = addr tbl
+    discard registerHostFn(reg, "echo_host", echoFn, addr tbl)
+    let r = waitFor echoHost("")
+    check r.isOk
+    check r.get == ""
+
+  test "unregistered host fn yields an error":
+    var reg: FFIHostRegistry
+    var tbl: FFIPendingTable
+    initHostRegistry(reg)
+    initPendingTable(tbl)
+    defer:
+      deinitHostRegistry(reg)
+      deinitPendingTable(tbl)
+    ffiCurrentHostRegistry = addr reg
+    ffiCurrentPendingTable = addr tbl
+    let r = waitFor echoHost("x")
+    check r.isErr
+    check "not registered" in r.error
+
+  test "host-reported error propagates as the Result error":
+    var reg: FFIHostRegistry
+    var tbl: FFIPendingTable
+    initHostRegistry(reg)
+    initPendingTable(tbl)
+    defer:
+      deinitHostRegistry(reg)
+      deinitPendingTable(tbl)
+    ffiCurrentHostRegistry = addr reg
+    ffiCurrentPendingTable = addr tbl
+    discard registerHostFn(reg, "echo_host", failFn, addr tbl)
+    let r = waitFor echoHost("x")
+    check r.isErr
+    check r.error == "host said no"
+
+  test "no host context on the thread yields an error":
+    ffiCurrentHostRegistry = nil
+    ffiCurrentPendingTable = nil
+    let r = waitFor echoHost("x")
+    check r.isErr
+    check "no host context" in r.error

tests/unit/test_host_callbacks.nim

@@ -0,0 +1,173 @@
+## Unit tests for the host-callback primitives (`FFIHostRegistry` /
+## `FFIPendingTable`) that back `{.ffiHost.}` — roadmap item #1, increment 1
+## (see docs/design-host-callbacks.md).
+##
+## These exercise the data structures directly: no FFI thread, no macro, no
+## completion bridge. They pin down registration, lookup, callId allocation, and
+## future completion semantics in isolation.
+
+import std/locks
+import unittest2
+import chronos
+import ffi
+
+# A host fn does nothing here — we only assert it round-trips through the
+# registry. `userData` carries a tag we read back to prove identity.
+proc noopHostFn(
+    callId: uint64, req: ptr cchar, reqLen: csize_t, userData: pointer
+) {.cdecl, gcsafe, raises: [].} =
+  discard
+
+proc bytesToStr(b: seq[byte]): string =
+  var s = newString(b.len)
+  if b.len > 0:
+    copyMem(addr s[0], unsafeAddr b[0], b.len)
+  return s
+
+suite "FFIHostRegistry":
+  test "register, lookup, replace, and remove":
+    var reg: FFIHostRegistry
+    initHostRegistry(reg)
+    defer:
+      deinitHostRegistry(reg)
+
+    var tag = 42
+    check registerHostFn(reg, "fetch_profile", noopHostFn, addr tag)
+
+    let hit = lookupHostFn(reg, "fetch_profile")
+    check hit.found
+    check hit.userData == addr tag
+    check not hit.fn.isNil()
+
+    # missing name -> found == false (never a crash)
+    check not lookupHostFn(reg, "does_not_exist").found
+
+    # nil fn unregisters and reports false
+    check not registerHostFn(reg, "fetch_profile", nil, nil)
+    check not lookupHostFn(reg, "fetch_profile").found
+
+  test "clear drops every registration":
+    var reg: FFIHostRegistry
+    initHostRegistry(reg)
+    defer:
+      deinitHostRegistry(reg)
+    check registerHostFn(reg, "a", noopHostFn, nil)
+    check registerHostFn(reg, "b", noopHostFn, nil)
+    clearHostFns(reg)
+    check not lookupHostFn(reg, "a").found
+    check not lookupHostFn(reg, "b").found
+
+suite "FFIPendingTable":
+  test "callIds are monotonic and start at 1":
+    var tbl: FFIPendingTable
+    initPendingTable(tbl)
+    defer:
+      deinitPendingTable(tbl)
+    let a = newPending(tbl)
+    let b = newPending(tbl)
+    check a.callId == 1'u64
+    check b.callId == 2'u64
+    check tbl.pendingCount == 2
+
+  test "completePending resolves the awaiting future and removes it":
+    var tbl: FFIPendingTable
+    initPendingTable(tbl)
+    defer:
+      deinitPendingTable(tbl)
+    let p = newPending(tbl)
+    check completePending(tbl, p.callId, okResult(@[byte 1, 2, 3]))
+    check p.fut.finished()
+    check waitFor(p.fut).ret == RET_OK
+    check waitFor(p.fut).bytes == @[byte 1, 2, 3]
+    check tbl.pendingCount == 0
+
+  test "unknown or double completion is dropped, not fatal":
+    var tbl: FFIPendingTable
+    initPendingTable(tbl)
+    defer:
+      deinitPendingTable(tbl)
+    check not completePending(tbl, 999'u64, okResult(@[]))
+    let p = newPending(tbl)
+    check completePending(tbl, p.callId, okResult(@[]))
+    check not completePending(tbl, p.callId, okResult(@[])) # second time: dropped
+
+  test "failAllPending errors every outstanding future":
+    var tbl: FFIPendingTable
+    initPendingTable(tbl)
+    defer:
+      deinitPendingTable(tbl)
+    let p1 = newPending(tbl)
+    let p2 = newPending(tbl)
+    failAllPending(tbl, "context shutting down")
+    check p1.fut.finished()
+    check p2.fut.finished()
+    let r = waitFor(p1.fut)
+    check r.ret == RET_ERR
+    check bytesToStr(r.bytes) == "context shutting down"
+    check tbl.pendingCount == 0
+
+# `pushCompletion` takes the raw (msg, len) a host hands across the C ABI.
+proc pushStr(q: var FFICompletionQueue, callId: uint64, ret: cint, s: string) =
+  if s.len == 0:
+    pushCompletion(q, callId, ret, nil, 0)
+  else:
+    pushCompletion(q, callId, ret, cast[ptr cchar](unsafeAddr s[0]), csize_t(s.len))
+
+suite "FFICompletionQueue":
+  test "drain resolves pending futures by callId, in FIFO order":
+    var tbl: FFIPendingTable
+    var q: FFICompletionQueue
+    initPendingTable(tbl)
+    initCompletionQueue(q)
+    defer:
+      deinitPendingTable(tbl)
+      deinitCompletionQueue(q)
+
+    let a = newPending(tbl) # callId 1
+    let b = newPending(tbl) # callId 2
+    pushStr(q, a.callId, RET_OK, "alpha")
+    pushStr(q, b.callId, RET_ERR, "boom")
+
+    check drainCompletions(q, tbl) == 2
+    check bytesToStr(waitFor(a.fut).bytes) == "alpha"
+    check waitFor(a.fut).ret == RET_OK
+    check bytesToStr(waitFor(b.fut).bytes) == "boom"
+    check waitFor(b.fut).ret == RET_ERR
+    check tbl.pendingCount == 0
+
+  test "empty payload and empty queue drain cleanly":
+    var tbl: FFIPendingTable
+    var q: FFICompletionQueue
+    initPendingTable(tbl)
+    initCompletionQueue(q)
+    defer:
+      deinitPendingTable(tbl)
+      deinitCompletionQueue(q)
+    check drainCompletions(q, tbl) == 0 # nothing queued
+    let p = newPending(tbl)
+    pushStr(q, p.callId, RET_OK, "") # empty (nil buf) payload
+    check drainCompletions(q, tbl) == 1
+    check waitFor(p.fut).bytes.len == 0
+
+  test "completion for an unknown callId is drained and dropped":
+    var tbl: FFIPendingTable
+    var q: FFICompletionQueue
+    initPendingTable(tbl)
+    initCompletionQueue(q)
+    defer:
+      deinitPendingTable(tbl)
+      deinitCompletionQueue(q)
+    pushStr(q, 999'u64, RET_OK, "orphan") # no pending future for this callId
+    check drainCompletions(q, tbl) == 1 # drained (and its buffer freed)
+
+  test "deinit frees still-queued nodes without draining":
+    var tbl: FFIPendingTable
+    var q: FFICompletionQueue
+    initPendingTable(tbl)
+    initCompletionQueue(q)
+    let p = newPending(tbl)
+    pushStr(q, p.callId, RET_OK, "leftover")
+    failAllPending(tbl, "shutdown") # the future is settled separately
+    deinitPendingTable(tbl)
+    deinitCompletionQueue(q) # must free the queued node, no leak/crash
+    check waitFor(p.fut).ret == RET_ERR