`nimble setup -l` (used by status-go's from-source builds: the functional-test
Docker image with nim 2.2.4 and the Windows job) walked each locked package's
`dependencies` array, looked every name up in the lock's package table, and
crashed with "key not found: nim" because the `nim` package entry was removed
to keep the offline Nix build working.
Strip the implicit `nim` compiler from the dependency arrays instead of
re-adding it as a package: re-adding it makes `nimble setup -l` clone and
rebuild the Nim compiler from source (slow, pointless — the toolchain is
already installed) and its checksum is nimble-version-specific. With no `nim`
references, `setup -l` resolves cleanly against the system compiler, and the
Nix build (which never choked on the missing key) is unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Nix build pre-populates deps into pkgs2 and can't clone in the sandbox.
nim-ffi isn't in the nimble registry, so requiring it by URL made nimble
re-clone it during resolution (offline -> fail). Resolve ffi by name from the
installed pkgs2 in the Nix build (gated by SDS_NIX_DEPS), keeping the URL fetch
for plain `nimble` builds; the URL now points at the published v0.2.0-rc.1 tag.
Also drop the `nim` entry from the lock (the sandbox can't fetch Nim by SHA),
record numeric lock versions for ffi (0.2.0) and cbor_serialization (0.3.0) so
they resolve cleanly, and regenerate nix/deps.nix to include cbor_serialization.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rebuild the C wrapper on nim-ffi 0.2's high-level macros (declareLibrary +
{.ffiCtor.}/{.ffi.}/{.ffiEvent.}), which marshal parameters and results as
CBOR and expose snake_case sds_* entry points. Replaces the previous
hand-written positional/JSON ABI.
- request/response objects are {.ffi.} types (CBOR); the unwrap response is
a proper nested object (message, channelId, missingDeps) instead of
hand-built JSON, and retrievalHint travels as raw bytes (no base64).
- events become {.ffiEvent.} procs (message_ready, message_sent,
missing_dependencies, periodic_sync, repair_ready), delivered to the host
via sds_add_event_listener.
- the retrieval-hint provider stays hand-written (a C function pointer has
no CBOR form); its pointers travel as uint64 through the request channel,
and the provided buffer is freed with libc free to match the host's
malloc (Go's C.CBytes).
- pin nim-ffi at the fix/foreign-host-concurrency-v0.2 branch (recycle pool
+ foreign-thread GC fixes) and add the cbor_serialization dependency.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nim-ffi 0.1.6 fails the build if -d:noSignalHandler is missing, so the
libsds build can no longer silently drop the flag that lets the Go host
keep ownership of OS signal handlers (the regression that crashed
status-go functional tests). sds.nimble already passes the flag; this
wires the compile-time guard in.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Regenerating the lock with newer nimble (0.22.3) rewrote entries to the
"#<rev>" special-version form, added a `nim` compiler entry, and listed `nim`
as a dependency everywhere. The nimble bundled with the nim 2.2.4 toolchain
(used by status-go CI) can't consume that: it fails on the nim checksum, then
"key not found: nim", then `git init` on a "#"-versioned bearssl dir.
Take the pre-existing lock format (proper tagged versions, no `nim` entry,
bearssl_pkey_decoder pinned as 0.1.0) and only bump the `ffi` entry to 0.1.5.
Verified building libsds in a fresh clean-room under both nimble 0.18.2 and
0.22.3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nim-ffi 0.1.5 reworked the context pool to recycle contexts (reuse the
worker thread + its fds) instead of tearing them down per cycle, which is
what fixes the per-create/destroy fd leak. Two consumer-side changes are
required for libsds to work with it:
- SdsCreateRmReq: nim-ffi no longer points `myLib` at a thread-stack var;
it owns it as a createShared'd object and frees it on recycle. So the
handler must (re)allocate `ctx.myLib` before assigning, otherwise the
first create dereferences a nil `myLib` and segfaults.
- SdsCleanupReliabilityManager: use `releaseFFIContext` (recycle) instead
of `destroyFFIContext` (full teardown). Recycle keeps the worker and its
fds alive for the next manager; destroy would reintroduce the leak. It is
non-blocking and fires `callback` from the FFI thread once drained, so the
synchronous RET_OK fire is dropped.
Lock nim-ffi to 0.1.5 (f6a3a33). Verified: libsds builds (--mm:refc) and the
status-go reliability suite (incl. the multi-manager fd test) is green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Changes include:
- Removing all submodules from vendor folder.
- Updating sds.nimble with required depndencies.
- Generating a nimble.lock file using Nimble.
- Updated Nim code to reference depndencies correctly.
- Added nix/deps.nix fixed output derivation that calls Nimble.
- Updated nixpkgs to use 25.11 commit which provides Nimbe 0.20.1.
- Disabled Nix Android builds on MacOS due to Nimble segfault.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
