nim-ffi was required by bare git URL with no version, so a fresh nimble
resolution (as triggered by the bumped nimbledeps cache key) ignored the
nimble.lock pin and floated it to HEAD (0.1.4). nim-ffi 0.1.4 changed
declareLibrary to take a second `libType` argument, so library/declare_lib.nim
fails to compile with a type mismatch and libwaku/liblogosdelivery break.
Pinning the require to the same commit nimble.lock records keeps the
version-less git dependency deterministic across clean resolves, matching
the API the current code targets. Mirrors the existing nim-sds pin style.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Windows build hung for 6h (until Actions cancelled it) at
`nimble wakunode2`, on "Downloading nim-lang/Nim.git using git".
`nimble setup` runs with --localdeps, so the locked nim is never placed
in nimbledeps/. Every later `nimble <task>` re-resolves deps, finds nim
missing, and clones the full Nim repo from git — a fetch that completes
on Linux/macOS but blocks forever on the MSYS2 runner.
--useSystemNim cannot be set from inside waku.nimble: it is a nimble CLI
flag governing the dependency resolution that wraps the task, and the
.nimble file is evaluated as part of that step. So fix it at the single
place every task is invoked — the Makefile NIMBLE variable.
NIMBLE resolves nimble via PATH rather than $(HOME)/.nimble/bin: on the
MSYS2 Windows runner that absolute path does not hold the binary (it is
on PATH via the workflow's GITHUB_PATH), so a hardcoded path fails with
"No such file or directory". Bare `nimble` works on every platform.
clean and the global nph install stay bare; they resolve no project deps.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a portable (macOS bash 3.2 / Linux) helper that detects git-URL pinned
`requires` in waku.nimble which changed vs a git base ref (default HEAD) and
updates ONLY those nimble.lock entries — version, vcsRevision and the sha1
checksum — leaving every other entry byte-for-byte untouched.
It does not run `nimble lock` (which rewrites the whole file). The sha1 is
computed directly, reproducing nimble's algorithm from
src/nimblepkg/checksums.nim (git ls-files -> sort -> SHA1 over path +
symlink-target/file-bytes). Resolves tags to commits via git rev-parse and
guards against invalid commit hashes (e.g. a stray leading character).
Dry-run by default (exit 1 on drift); --apply writes; --base REF to compare
against another ref. Requires git + python3; nimble not required.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The docker image build compiled secp256k1 with -march=native and failed.
waku.nimble's getNimParams reads the NIM_PARAMS env var, not NIMFLAGS, so
the -d:disableMarchNative passed via NIMFLAGS on the make line never
reached `nim c` and config.nims applied -march=native. #3916 added
NIM_PARAMS to ci.yml but missed container-image.yml; set it here too.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Stripping nim from the lock breaks nimble 0.22.3: 41 packages list nim in
their dependencies, so removing the nim entry makes resolution fail with
"key not found: nim". nimble already has the right knob — --useSystemNim
("Use system nim and ignore nim from the lock") — which skips the locked
nim (no download, no checksum) while still verifying every other locked
dependency, and uses the nim the CI already installed.
This is the proper fix for the Windows checksum mismatch: nim's tree
checks out differently per platform, so its locked checksum is unreliable
there. Verified locally with nimble 0.22.3 (exit 0, system nim used, lock
left untouched).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Restores a fix that was dropped in e5b46e5e and re-exposed when cda01971
reintroduced nim/nimble into nimble.lock: the nim package never matches
its Linux-computed checksum on Windows because nim's own .gitattributes
forces line endings on parts of its tree, so the SHA1 differs regardless
of core.autocrlf. autocrlf alone therefore cannot fix the nim mismatch.
Remove nim/nimble from the lock before `nimble setup` (they are provided
by the toolchain installed in CI, not real project deps). Added to the
Windows workflow, which never had it, and restored in ci.yml's setup
steps since the cache-key bump now forces those to run on a fresh tree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Linux nimbledeps cache held a corrupt lsquic/nimblemeta.json (invalid
JSON), so restoring it made `make wakunode2` fail parsing it while a fresh
lsquic download in the same run was fine. The key is derived from
nimble.lock, which is unchanged, so the poison persisted across runs.
Add a v2 salt to the key to force a clean rebuild of the cache.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The earlier autocrlf fix ran in a pwsh step, which only configures Git
for Windows. nimble clones its dependency packages from the MSYS2 shell,
whose git reads a separate global config, so autocrlf stayed enabled
there. That kept altering dependency source trees, breaking their SHA1
match against nimble.lock (e.g. the nim package), so nimble re-downloaded
and hung. Repeat the setting in an MSYS2-shell step so nimble's own git
clones produce Linux-identical trees.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Windows Git defaults to core.autocrlf=true. When nimble clones dependency
packages into nimbledeps/, the LF→CRLF conversion changes the SHA1 of the
source tree relative to the Linux-computed checksums in nimble.lock.
nimble then treats every cached package as invalid and re-downloads on each
invocation; in PR #3919 the re-download of bearssl_pkey_decoder during
`nimble wakunode2` hung until the 6h job timeout.
Set core.autocrlf=false / core.eol=lf globally before checkout so every
package nimble clones keeps LF endings and matches the Linux SHA1s. This
also obsoletes the targeted sed patch for the nim checksum, which is
removed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Remove makefile target update
* fix: set execute permission on install_nimble.sh
* improve install_nim script
* skip second nim install on Windows
* fix path check in install-nim
* Makefile workfile reordering
* chore: pin confutils to merged upstream commit
status-im/nim-confutils#146 is merged; move the confutils pin from the
PR fork back to status-im/nim-confutils master (36f3115). Content is
identical to the fork commit, so nimble sha1 and nix sha256 are unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #3899 fixes zerokit v2.0.2's stale cargoHash, but only via an internal
`let` binding consumed by liblogosdelivery. Downstream consumers (e.g.
logos-delivery-module) that need librln still pull zerokit's rln package
directly and hit the stale hash.
Expose that corrected derivation as `packages.<system>.rln` so consumers can
bundle the exact same librln this build links, instead of overriding the
cargoHash themselves.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(node-info): real Version + new Commit in Nix/lgpm builds
getNodeInfo Version returned "n/a" on Nix-built libs (and lgpm
releases built from the flake) because nix/default.nix never passed
-d:git_version. A flake sandbox has no .git, so git describe is
impossible; derive the semver from waku.nimble's version field plus
the flake short commit, and expose the full commit SHA via a new
Commit node info id.
- waku_state_info: add Commit to NodeInfoId + dispatch git_commit
- waku_node: add git_commit {.strdefine.} (default "n/a")
- node start logs ("Starting Waku node" / "Running nwaku node") now
print commit = git_commit alongside version
- Makefile: inject -d:git_commit (full SHA), mirrors docker label
- nix/default.nix: accept gitVersion/gitCommit, pass as nim defines
- flake.nix: gitVersion = <nimble version>-g<shortRev>, gitCommit = rev
- CI version-check (PR only): ancestor-aware `git describe --tags
--abbrev=0` vs PR HEAD, base-version compare, so waku.nimble is kept
current early and a new tag never breaks in-flight PRs
- release-assets.yml: gate build/upload on a verify-version job
asserting tag base == waku.nimble (RC tags allowed), so a mismatched
tag publishes no artifacts
- docs: prepare_release.md explains the bump-before-tag requirement
Refs: status-im/infra-logos#4Closes: logos-messaging/logos-delivery#3884
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: simplify
* chore: update version in waku.nimble
* fix(node-info): remove Commit node info field
Drop the newly added Commit (full SHA) node info id and its
git_commit compile-time define plumbing across Makefile, flake.nix
and nix/default.nix; revert the start/run log lines to version only.
The PR now solely fixes the getNodeInfo Version regression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(nix): align git_version format closer to Makefile
Adds the `v` prefix and uses a 6-char SHA so Nix-built nodes report
e.g. `v0.38.1-g52e980`, matching the shape of `git describe --abbrev=6
--always --tags` aside from the unreachable commit-count segment (tag
metadata isn't exposed through the flake input protocol).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds NodeInfoId.MyMixPubKey, returning the node's mix public key as
0x-prefixed hex via the existing debug API. Returns an empty string
when the mix protocol is not mounted.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* persistency: per-job SQLite-backed storage layer (singleton, brokered)
Adds a backend-neutral CRUD library at waku/persistency/, plus the
nim-brokers dependency swap that enables it.
Architecture (ports-and-adapters):
* Persistency: process-wide singleton, one root directory.
* Job: one tenant, one DB file, one worker thread, one BrokerContext.
* Backend: SQLite via waku/common/databases/db_sqlite. Uniform schema
kv(category BLOB, key BLOB, payload BLOB) PRIMARY KEY (category, key)
WITHOUT ROWID, WAL mode.
* Writes are fire-and-forget via EventBroker(mt) PersistEvent.
* Reads are async via five RequestBroker(mt) shapes (KvGet, KvExists,
KvScan, KvCount, KvDelete). Reads return Result[T, PersistencyError].
* One storage thread per job; tenants isolated by BrokerContext.
Public surface (waku/persistency/persistency.nim):
Persistency.instance(rootDir) / Persistency.instance() / Persistency.reset()
p.openJob(id) / p.closeJob(id) / p.dropJob(id) / p.close()
p.job(id) / p[id] / p.hasJob(id)
Writes (Job form & string-id form, fire-and-forget):
persist / persistPut / persistDelete / persistEncoded
Reads (Job form & string-id form, async Result):
get / exists / scan / scanPrefix / count / deleteAcked
Key & payload encoding (keys.nim, payload.nim):
* encodePart family + variadic key(...) / payload(...) macros +
single-value toKey / toPayload.
* Primitives: string and openArray[byte] are 2-byte BE length + bytes;
int{8..64} are sign-flipped 8-byte BE; uint{16..64} are 8-byte BE;
bool/byte/char are 1 byte; enums are int64(ord(v)).
* Generic encodePart[T: tuple | object] recurses through fields() so
any composite Nim type is encodable without ceremony.
* Stable across Nim/C compiler upgrades: no sizeof, no memcpy, no
cast on pointers, no host-endianness dependency.
* `rawKey(bytes)` + `persistPut(..., openArray[byte])` let callers
bypass the built-in encoder with their own format (CBOR, protobuf...).
Lifecycle:
* Persistency.new is private; Persistency.instance is the only public
constructor. Same rootDir is idempotent; conflicting rootDir is
peInvalidArgument. Persistency.reset for test/restart paths.
* openJob opens-or-creates the per-job SQLite file; an existing file
is reused with its data preserved.
* Teardown integration: Persistency.instance registers a Teardown
MultiRequestBroker provider that closes all jobs and clears the
singleton slot when Waku.stop() issues Teardown.request.
Internal layering:
types.nim pure value types (Key, KeyRange, KvRow, TxOp,
PersistencyError)
keys.nim encodePart primitives + key(...) macro
payload.nim toPayload + payload(...) macro
schema.nim CREATE TABLE + connection pragmas + user_version
backend_sqlite.nim KvBackend, applyOps (single source of write SQL),
getOne/existsOne/deleteOne, scanRange (asc/desc,
half-open ranges, open-ended stop), countRange
backend_comm.nim EventBroker(mt) PersistEvent + 5 RequestBroker(mt)
declarations; encodeErr/decodeErr boundary helpers
backend_thread.nim startStorageThread / stopStorageThread (shared
allocShared0 arg, cstring dbPath, atomic
ready/shutdown flags); per-thread provider
registration
persistency.nim Persistency + Job types, singleton state, public
facade
../requests/lifecycle_requests.nim
Teardown MultiRequestBroker
Tests (69 cases, all passing):
test_keys.nim sort-order invariants (length-prefix strings,
sign-flipped ints, composite tuples, prefix
range)
test_backend.nim round-trip / replace / delete-return-value /
batched atomicity / asc-desc-half-open-open-
ended scans / category isolation / batch
txDelete
test_lifecycle.nim open-or-create rootDir / non-dir collision /
reopen across sessions / idempotent openJob /
two-tenant parallel isolation / closeJob joins
worker / dropJob removes file / acked delete
test_facade.nim put-then-get / atomic batch / scanPrefix
asc/desc / deleteAcked hit-miss /
fire-and-forget delete / two-tenant facade
isolation
test_encoding.nim tuple/named-tuple/object keys, embedded Key,
enum encoding, field-major composite sort,
payload struct encoding, end-to-end struct
round-trip through SQLite
test_string_lookup.nim peJobNotFound semantics / hasJob / subscript /
persistPut+get via id / reads short-circuit /
writes drop+warn / persistEncoded via id /
scan parity Job-ref vs id
test_singleton.nim idempotent same-rootDir / different-rootDir
rejection / no-arg instance lifecycle / reset
retargets / reset idempotence / Teardown.request
end-to-end
Prerequisite delivered in the same series: replace the in-tree broker
implementation with the external nim-brokers package; update all
broker call-sites (waku_filter_v2, waku_relay, waku_rln_relay,
delivery_service, peer_manager, requests/*, factory/*, api tests, etc.)
to the new package API; chat2 made to compile again.
Note: SDS adapter (Phase 5 of the design) is deferred -- nim-sds is
still developed side-by-side and the persistency layer is intentionally
SDS-agnostic.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* persistency: pin nim-brokers by URL+commit (workaround for stale registry)
The bare `brokers >= 2.0.1` form cannot resolve on machines where the
local nimble SAT solver enumerates only the registry-recorded 0.1.0 for
brokers. The nim-lang/packages entry for `brokers` carries no per-tag
metadata (only the URL), so until that registry entry is refreshed the
SAT solver clamps the available-versions list to 0.1.0 and rejects the
>= 2.0.1 constraint -- even though pkgs2 and pkgcache both have v2.0.1
cloned locally.
Pinning by URL+commit bypasses the registry path entirely. Inline
comment in waku.nimble documents the situation and the path back to
the bare form once nim-lang/packages is updated.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* persistency: nph format pass
Run `nph` on all 57 Nim files touched by this PR. Pure formatting:
17 files re-styled, no semantic change. Suite still 69/69.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Fix build, add local-storage-path config, lazy init of Persistency from Waku start
* fix: fix nix deps
* fixes for nix build, regenerate deps
* reverting accidental dependency changes
* Fixing deps
* Apply suggestions from code review
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* persistency tests: migrate to suite / asyncTest / await
Match the in-tree test convention (procSuite -> suite, sync test +
waitFor -> asyncTest + await):
- procSuite "X": -> suite "X":
- For tests doing async work: test -> asyncTest, waitFor -> await.
- Poll helpers (proc waitFor(t: Job, ...) in test_lifecycle.nim,
proc waitUntilExists(...) in test_facade.nim and
test_string_lookup.nim) -> Future[bool] {.async.}, internal
`waitFor X` -> `await X`, internal `sleep(N)` ->
`await sleepAsync(chronos.milliseconds(N))`.
- Renamed test_lifecycle.nim's helper proc from `waitFor(t: Job, ...)`
-> `pollExists(t: Job, ...)`; the previous name shadowed
chronos.waitFor in the chronos macro expansion.
- `chronos.milliseconds(N)` explicitly qualified because `std/times`
also exports `milliseconds` (returning TimeInterval, not Duration).
- `check await x` -> `let okN = await x; check okN` to dodge chronos's
"yield in expr not lowered" with await-as-macro-argument.
- `(await x).foo()` -> `let awN = await x; ... awN.foo() ...` for the
same reason.
waku/persistency/persistency.nim: nph also pulled the proc signatures
across multiple lines; restored explicit `Future[void] {.async.}`
return types after the colon (an intermediate nph pass had elided them).
Suite: 71 / 71 OK against the new async write surface.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* use idiomatic valueOr instead of ifs
* Reworked persistency shutdown, remove not necessary teardown mechanism
* Use const for DefaultStoragePath
* format to follow coding guidelines - no use of result and explicit returns - no functional change
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* any port set to 0 on conf results in a random port bound
* Debug API MyBoundPorts reports actually bound ports for all services, reports 0 if disabled
* write back bound values to both WakuConf and WakuNode.ports
* setupDiscoveryV5 returns Result and errors out on port 0
* rename setupAndStartDiscv5WithAutoPort to setupAndStartDiscv5
* updateWaku ENR rebuild now runs after discv5 startup
* Add DefaultP2pTcpPort, DefaultDiscv5UdpPort, DefaultWebSocketPort, DefaultRestPort, DefaultMetricsHttpPort
* add tests
* nix: parameterize build flags with named args
Expose `enablePostgres`, `enableNimDebugDlOpen`, and `chroniclesLogLevel`
as arguments on `nix/default.nix`. Defaults preserve today's hardcoded
behavior, so `nix build .#liblogosdelivery` with no overrides is a
no-op change.
Consume the package via `callPackage` in `flake.nix` so consumers can
use `.override { ... }` without extra wrapping.
* nix: link libstdc++ on Linux so consumers don't need patchelf
Append `stdenv.cc.cc.lib` to `buildInputs` on Linux and add `-lstdc++`
to the Nim `--passL` flags. Nix stdenv's fixupPhase will auto-inject
`${stdenv.cc.cc.lib}/lib` into the output's RUNPATH, so downstream
consumers can drop their patchelf step.
macOS resolves the C++ stdlib via dyld/libc++ and is unaffected.
* nix: bundle librln into the output for a self-contained package
Copy the librln shared library (`librln.so` / `librln.dylib`) from the
zerokit input into `$out/lib` and rewrite the internal reference in
`liblogosdelivery`:
- Darwin: set librln's install name to `@rpath/librln.dylib`, change the
consumer's reference to match, and add `@loader_path` as an rpath.
- Linux: add `$ORIGIN` to the rpath so `librln.so` resolves from the
sibling directory, preserving the gcc-lib entry injected by the stdenv
fixupPhase for libstdc++.
The installed `liblogosdelivery` no longer carries a `/nix/store/...`
absolute path to zerokit, so downstream consumers can ship the bundle
as-is.
Rebuild nat libs (miniupnpc, libnatpmp) for the host architecture during
nimble deps setup. The prebuilt libs from the nimble cache are x86_64 and
fail to link on arm64 (Apple Silicon).
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
