Cascades the mix-rln plugin's libp2p_mix pin bump (PR #14 50c4ab4f ->
master e314cdd5) and our own plugin bump (ba32e9f -> 8f4db89) through
nwaku's lockfile.
The checksums.sha1 values are LEFT AS-IS for now; nimble will recompute
both on first build and bail with the expected values, which we'll write
back. Same workaround pattern as the existing 'nim' and 'bearssl_pkey_
decoder' patches in setup_from_scratch.sh:patch_delivery_nimble_lock.
PR #3807 still uses std/Option.valueOr in ~30 places across logos_delivery
but does not consistently import waku/common/option_shims (the local
template that restores libp2p-1.15.2's valueOr/withValue for Option[T];
libp2p 1.15.3 dropped those overloads). Without the shim, nim picks up
only Result[T,E].valueOr from nim-results and fails with 'type mismatch'.
Surfaced when building liblogoschat which pulls a much wider slice of
nwaku than liblogosdelivery does. Each touched file now imports
option_shims either inline alongside other ../common/ entries or as a
standalone top-level import where the file uses absolute logos_delivery/
paths.
No semantic change — the shim restores the historical Option valueOr
behavior. Same files compiling against the older libp2p (where valueOr
on Option came from libp2p/utility) would behave identically.
Combines the post-feature cleanup pass:
- Extract bytesToHexUpper helper for idCommitment hex (was duplicated
across logos_core_client.nim + node_factory.nim).
- Drop dead helper functions, unused imports, and duplicate kademlia
wiring left over from the feature commit.
- Narrow waitForChainCommit's broad except, add CancelledError
carve-outs to the self-registration watcher to avoid swallowing
shutdown signals.
Ports commit 14562878 onto PR #3807's logos_delivery/ layout.
Changes vs original 14562878:
- Plugin consumed via nimble dep (no submodule add); the PR #3807 nimble
resolution already provides it.
- mountMix signature carries both PR #3807's disableSpamProtection AND
the patch's useOnchainLEZ parameters (callsite passes both).
- WakuMix.new branches on disableSpamProtection while also setting
useOnchainLEZ on the spam-protection config. Cover-traffic params come
from the plugin config when RLN is enabled, fall back to waku defaults
when disabled.
- start() retains HEAD's split between local-only init and the separate
registerDoSProtectionWithNetwork retry loop. publishGossipsubTrigger is
promoted to a standalone proc instead of inline in an else branch.
- MixProtocol.init uses libp2p_mix's new Opt-wrapped coverTraffic and
the MixRlnSpamProtection.new constructor (not the renamed newMixRlnSpamProtection).
- Makefile reverted to HEAD: PR #3807 dropped the separate mix-librln
archive in favor of a single stateless zerokit, so the patch's
MIX_LIBRLN_* additions are obsolete.
- README/sim helper config files: HEAD's values retained (env-specific
paths from patch author dropped).
Adds:
- logos_delivery/waku/waku_mix/logos_core_client.nim (RLN client layer)
- logos_delivery/waku/waku_mix/coordination handler (via waku_node)
- logos_delivery/waku/waku_rln_relay/rln_gifter/{client,protocol,rpc,rpc_codec}.nim
- 5 new C FFI exports (logosdelivery_set_rln_fetcher, _set_rln_config,
_set_rln_identity, _push_roots, _push_proof)
- --mix-gifter-* CLI flags + MixConf fields + MixConfBuilder methods
- tests/waku_rln_relay/test_rln_gifter.nim
Bumps the nimble pin from upstream PR #9 tip (logos-co@61ee3e5) to our
combined branch adklempner/feat/lez-rln-stateless@ba32e9f, which is
PR #9 stateless RLN + our 4 LEZ-RLN commits cherry-picked on top:
- LEZ-backed RLN spam protection (OnchainLEZGroupManager)
- try/except cleanup around await in pollLoop
- info-level proof markers + atomic root+proof refresh
- defensive add of witness-implied root at proof-gen time
The nimble.lock checksum will be re-validated on first build; if it
mismatches we'll update with the value nimble reports.
Rename the call site to the typedesc constructor
MixRlnSpamProtection.new (was newMixRlnSpamProtection) to match
nim-libp2p conventions, and bump the mix-rln-spam-protection-plugin
pin to 61ee3e5 which provides the renamed constructor.
- waku/waku_mix/protocol.nim: drop the magic-2 cover-traffic fallback
and the hardcoded 10s epoch. Source cover-traffic totalSlots and
epochDuration from spamProtectionConfig when RLN is on (so cover
emission can't outpace proof minting), and from named waku
constants (WakuCoverTrafficTotalSlots=40, WakuCoverTrafficEpochDuration=60s,
~10 emissions/min/node) when RLN is disabled. Single ConstantRate
CoverTraffic.new call site at the end of the block; the if/else
only sets up spam protection. Addresses PR review comment on
protocol.nim line 102.
- waku.nimble: bump mix-rln plugin to 8ec5dc24 (latest on
feat/cover-traffic-epoch-support: messageId guard + drift-corrected
epoch timer) and pin nim-lsquic to #6d2bc489 (v0.2.0) so libp2p
1.15.3's certificate_ffi keeps finding EVP_PKEY in lsquic_ffi.nim.
- waku/common/option_shims: explain in a header comment that the file
exists because libp2p 1.15.3 dropped Option[T] overloads of
valueOr/withValue from libp2p/utility; can be removed once those are
restored upstream. Addresses PR review comment.
- simulations/mixnet/setup_credentials: drop the unused
SpammerUserMessageLimit constant (the "Higher" comment was wrong
since 3 < DefaultUserMessageLimit=4, and zerokit blocks proof-gen
past the per-user limit anyway, so it could never simulate a
spammer). Addresses PR review comment.
- simulations/mixnet/run_chat_mix{,1}.sh: pass --rln-user-message-limit=4
so the chat client's RLN budget matches the keystores baked at
limit=4, otherwise cover-traffic totalSlots vs RLN-budget mismatch
jams the sim.
Sim verified end-to-end:
- RLN-on (default sim config): PASS, ≥2 proof-verified per node,
cover-traffic metrics non-zero.
- RLN-off (mix nodes only, ad-hoc config): cover-traffic emits at
~13/min/node from the new waku defaults (target ~10/min); the
no-RLN code path is exercised correctly.
- Integrate ConstantRateCoverTraffic from libp2p mix module with default
totalSlots = userMessageLimit (or 2) and 10s epoch
- Add --mix-user-message-limit and --mix-disable-spam-protection CLI flags
with corresponding MixConfBuilder accessors and MixConf fields
- Wrap mixRlnSpamProtection construction so it is skipped when spam
protection is disabled, with a nil guard in setupSpamProtectionCallbacks
- Add waku/common/option_shims.nim restoring valueOr/withValue templates
for std/options (removed upstream by results), and import it across
modules that relied on the old behavior
- Sink chat2mix logs to textlines (stdout) instead of textlines[file] to
work around a chronicles compile-time macro-eval bug under Nim 2.2.4
- Rename ExtendedKademliaDiscoveryParams -> ExtendedServiceDiscoveryParams
to match the kad_disco -> service_discovery rename in nim-libp2p
- Bump nim-libp2p to e1bbda4f6 (PR #2243 "cover traffic with constant
rate") and mix-rln-spam-protection-plugin to 153d0c0 (PR #5 cover
traffic epoch change support); both pre-libp2p_mix-extraction
- Add simulations/mixnet/check_cover_traffic.sh for monitoring
mix_cover_* / mix_slot_* metrics, plus per-node cover-traffic configs
Rename the call site to the typedesc constructor
MixRlnSpamProtection.new (was newMixRlnSpamProtection) to match
nim-libp2p conventions, and bump the mix-rln-spam-protection-plugin
pin to 61ee3e5 which provides the renamed constructor.
* Only add new roots, not all received
* Fix error in removing recent roots not checking AcceptableWindowSize
* fix merging
* more merging fixes
* merge fixes
* add test for updated merkle roots window
* add pr re-add gauge for proof-generation-duration-seconds
* Decrease AcceptableRootWindowSize for testing
* debug spam log
* linting
* start trackRootChanges call loop immediately
* Fix 5s delay trackRootChanges
* set rpcDelay for root tracking to 10s
* add default params to sendEthCallWithParams
* improve recents roots retrieval and logs
* Use updateRecentRoots to track root changes
* simplify updateRecentRoots
* set root polling to 15s
* set rpc poll delay to 30s
* set acceptablerootwindowsize and root poll delay
* Improve test 'should fetch history correctly' for root cache
* Make root cache handling more efficient
* add contract root cache size as constant and function use fix
* updateRecentRoots comments update
* Update group_manager and tests
* fix linting
* persistency: follow nim-sds 0.3.0 snapshot persistence contract
nim-sds 0.3.0 replaced the ~14 fine-grained per-row Persistence callbacks
with a 5-proc snapshot model (saveChannelMeta / updateHistory / loadChannel
/ dropChannel / setRetrievalHint), all returning Future[Result[...]].
Rewrite waku/persistency/sds_persistency.nim accordingly:
- ChannelMeta is stored as one blob per channel; the message log as
append/evict rows. Categories collapse from 7 to 2 (sds.meta, sds.log).
- Blob transform uses nim-sds' own codecs: snapshot_codec (schema-versioned
protobuf) for ChannelMeta, the SDS wire codec for SdsMessage log rows. The
generic payload_codec/BlobCodec path is retired (removed payload_codec.nim
and test_blob_codec.nim).
- setRetrievalHint is a deliberate no-op: persisted hints are never read back
(loadChannel/ChannelMeta carry none; hints are supplied live via the
onRetrievalHint provider). The closure stays because the field is required.
- Fix the module import spelling (srcDir="sds" => bare module paths), which
the previous adapter got wrong and never compiled against the locked deps.
Add tests/persistency/test_sds_persistency.nim (round-trip, empty-load,
evict, drop) replacing test_blob_codec in test_all. Full persistency suite
passes 74/74 under both refc and ORC.
* Bump to latest nim-sds and nim-brokers 3.1.1
* Update with latest nim-sds changes - removal of setRetrievalHints - not needed
Adds a portable (macOS bash 3.2 / Linux) helper that detects git-URL pinned
`requires` in waku.nimble which changed vs a git base ref (default HEAD) and
updates ONLY those nimble.lock entries — version, vcsRevision and the sha1
checksum — leaving every other entry byte-for-byte untouched.
It does not run `nimble lock` (which rewrites the whole file). The sha1 is
computed directly, reproducing nimble's algorithm from
src/nimblepkg/checksums.nim (git ls-files -> sort -> SHA1 over path +
symlink-target/file-bytes). Resolves tags to commits via git rev-parse and
guards against invalid commit hashes (e.g. a stray leading character).
Dry-run by default (exit 1 on drift); --apply writes; --base REF to compare
against another ref. Requires git + python3; nimble not required.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Remove makefile target update
* fix: set execute permission on install_nimble.sh
* improve install_nim script
* skip second nim install on Windows
* fix path check in install-nim
* Makefile workfile reordering
* chore: pin confutils to merged upstream commit
status-im/nim-confutils#146 is merged; move the confutils pin from the
PR fork back to status-im/nim-confutils master (36f3115). Content is
identical to the fork commit, so nimble sha1 and nix sha256 are unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #3899 fixes zerokit v2.0.2's stale cargoHash, but only via an internal
`let` binding consumed by liblogosdelivery. Downstream consumers (e.g.
logos-delivery-module) that need librln still pull zerokit's rln package
directly and hit the stale hash.
Expose that corrected derivation as `packages.<system>.rln` so consumers can
bundle the exact same librln this build links, instead of overriding the
cargoHash themselves.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(node-info): real Version + new Commit in Nix/lgpm builds
getNodeInfo Version returned "n/a" on Nix-built libs (and lgpm
releases built from the flake) because nix/default.nix never passed
-d:git_version. A flake sandbox has no .git, so git describe is
impossible; derive the semver from waku.nimble's version field plus
the flake short commit, and expose the full commit SHA via a new
Commit node info id.
- waku_state_info: add Commit to NodeInfoId + dispatch git_commit
- waku_node: add git_commit {.strdefine.} (default "n/a")
- node start logs ("Starting Waku node" / "Running nwaku node") now
print commit = git_commit alongside version
- Makefile: inject -d:git_commit (full SHA), mirrors docker label
- nix/default.nix: accept gitVersion/gitCommit, pass as nim defines
- flake.nix: gitVersion = <nimble version>-g<shortRev>, gitCommit = rev
- CI version-check (PR only): ancestor-aware `git describe --tags
--abbrev=0` vs PR HEAD, base-version compare, so waku.nimble is kept
current early and a new tag never breaks in-flight PRs
- release-assets.yml: gate build/upload on a verify-version job
asserting tag base == waku.nimble (RC tags allowed), so a mismatched
tag publishes no artifacts
- docs: prepare_release.md explains the bump-before-tag requirement
Refs: status-im/infra-logos#4Closes: logos-messaging/logos-delivery#3884
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: simplify
* chore: update version in waku.nimble
* fix(node-info): remove Commit node info field
Drop the newly added Commit (full SHA) node info id and its
git_commit compile-time define plumbing across Makefile, flake.nix
and nix/default.nix; revert the start/run log lines to version only.
The PR now solely fixes the getNodeInfo Version regression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(nix): align git_version format closer to Makefile
Adds the `v` prefix and uses a 6-char SHA so Nix-built nodes report
e.g. `v0.38.1-g52e980`, matching the shape of `git describe --abbrev=6
--always --tags` aside from the unreachable commit-count segment (tag
metadata isn't exposed through the flake input protocol).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds NodeInfoId.MyMixPubKey, returning the node's mix public key as
0x-prefixed hex via the existing debug API. Returns an empty string
when the mix protocol is not mounted.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* persistency: per-job SQLite-backed storage layer (singleton, brokered)
Adds a backend-neutral CRUD library at waku/persistency/, plus the
nim-brokers dependency swap that enables it.
Architecture (ports-and-adapters):
* Persistency: process-wide singleton, one root directory.
* Job: one tenant, one DB file, one worker thread, one BrokerContext.
* Backend: SQLite via waku/common/databases/db_sqlite. Uniform schema
kv(category BLOB, key BLOB, payload BLOB) PRIMARY KEY (category, key)
WITHOUT ROWID, WAL mode.
* Writes are fire-and-forget via EventBroker(mt) PersistEvent.
* Reads are async via five RequestBroker(mt) shapes (KvGet, KvExists,
KvScan, KvCount, KvDelete). Reads return Result[T, PersistencyError].
* One storage thread per job; tenants isolated by BrokerContext.
Public surface (waku/persistency/persistency.nim):
Persistency.instance(rootDir) / Persistency.instance() / Persistency.reset()
p.openJob(id) / p.closeJob(id) / p.dropJob(id) / p.close()
p.job(id) / p[id] / p.hasJob(id)
Writes (Job form & string-id form, fire-and-forget):
persist / persistPut / persistDelete / persistEncoded
Reads (Job form & string-id form, async Result):
get / exists / scan / scanPrefix / count / deleteAcked
Key & payload encoding (keys.nim, payload.nim):
* encodePart family + variadic key(...) / payload(...) macros +
single-value toKey / toPayload.
* Primitives: string and openArray[byte] are 2-byte BE length + bytes;
int{8..64} are sign-flipped 8-byte BE; uint{16..64} are 8-byte BE;
bool/byte/char are 1 byte; enums are int64(ord(v)).
* Generic encodePart[T: tuple | object] recurses through fields() so
any composite Nim type is encodable without ceremony.
* Stable across Nim/C compiler upgrades: no sizeof, no memcpy, no
cast on pointers, no host-endianness dependency.
* `rawKey(bytes)` + `persistPut(..., openArray[byte])` let callers
bypass the built-in encoder with their own format (CBOR, protobuf...).
Lifecycle:
* Persistency.new is private; Persistency.instance is the only public
constructor. Same rootDir is idempotent; conflicting rootDir is
peInvalidArgument. Persistency.reset for test/restart paths.
* openJob opens-or-creates the per-job SQLite file; an existing file
is reused with its data preserved.
* Teardown integration: Persistency.instance registers a Teardown
MultiRequestBroker provider that closes all jobs and clears the
singleton slot when Waku.stop() issues Teardown.request.
Internal layering:
types.nim pure value types (Key, KeyRange, KvRow, TxOp,
PersistencyError)
keys.nim encodePart primitives + key(...) macro
payload.nim toPayload + payload(...) macro
schema.nim CREATE TABLE + connection pragmas + user_version
backend_sqlite.nim KvBackend, applyOps (single source of write SQL),
getOne/existsOne/deleteOne, scanRange (asc/desc,
half-open ranges, open-ended stop), countRange
backend_comm.nim EventBroker(mt) PersistEvent + 5 RequestBroker(mt)
declarations; encodeErr/decodeErr boundary helpers
backend_thread.nim startStorageThread / stopStorageThread (shared
allocShared0 arg, cstring dbPath, atomic
ready/shutdown flags); per-thread provider
registration
persistency.nim Persistency + Job types, singleton state, public
facade
../requests/lifecycle_requests.nim
Teardown MultiRequestBroker
Tests (69 cases, all passing):
test_keys.nim sort-order invariants (length-prefix strings,
sign-flipped ints, composite tuples, prefix
range)
test_backend.nim round-trip / replace / delete-return-value /
batched atomicity / asc-desc-half-open-open-
ended scans / category isolation / batch
txDelete
test_lifecycle.nim open-or-create rootDir / non-dir collision /
reopen across sessions / idempotent openJob /
two-tenant parallel isolation / closeJob joins
worker / dropJob removes file / acked delete
test_facade.nim put-then-get / atomic batch / scanPrefix
asc/desc / deleteAcked hit-miss /
fire-and-forget delete / two-tenant facade
isolation
test_encoding.nim tuple/named-tuple/object keys, embedded Key,
enum encoding, field-major composite sort,
payload struct encoding, end-to-end struct
round-trip through SQLite
test_string_lookup.nim peJobNotFound semantics / hasJob / subscript /
persistPut+get via id / reads short-circuit /
writes drop+warn / persistEncoded via id /
scan parity Job-ref vs id
test_singleton.nim idempotent same-rootDir / different-rootDir
rejection / no-arg instance lifecycle / reset
retargets / reset idempotence / Teardown.request
end-to-end
Prerequisite delivered in the same series: replace the in-tree broker
implementation with the external nim-brokers package; update all
broker call-sites (waku_filter_v2, waku_relay, waku_rln_relay,
delivery_service, peer_manager, requests/*, factory/*, api tests, etc.)
to the new package API; chat2 made to compile again.
Note: SDS adapter (Phase 5 of the design) is deferred -- nim-sds is
still developed side-by-side and the persistency layer is intentionally
SDS-agnostic.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* persistency: pin nim-brokers by URL+commit (workaround for stale registry)
The bare `brokers >= 2.0.1` form cannot resolve on machines where the
local nimble SAT solver enumerates only the registry-recorded 0.1.0 for
brokers. The nim-lang/packages entry for `brokers` carries no per-tag
metadata (only the URL), so until that registry entry is refreshed the
SAT solver clamps the available-versions list to 0.1.0 and rejects the
>= 2.0.1 constraint -- even though pkgs2 and pkgcache both have v2.0.1
cloned locally.
Pinning by URL+commit bypasses the registry path entirely. Inline
comment in waku.nimble documents the situation and the path back to
the bare form once nim-lang/packages is updated.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* persistency: nph format pass
Run `nph` on all 57 Nim files touched by this PR. Pure formatting:
17 files re-styled, no semantic change. Suite still 69/69.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Fix build, add local-storage-path config, lazy init of Persistency from Waku start
* fix: fix nix deps
* fixes for nix build, regenerate deps
* reverting accidental dependency changes
* Fixing deps
* Apply suggestions from code review
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* persistency tests: migrate to suite / asyncTest / await
Match the in-tree test convention (procSuite -> suite, sync test +
waitFor -> asyncTest + await):
- procSuite "X": -> suite "X":
- For tests doing async work: test -> asyncTest, waitFor -> await.
- Poll helpers (proc waitFor(t: Job, ...) in test_lifecycle.nim,
proc waitUntilExists(...) in test_facade.nim and
test_string_lookup.nim) -> Future[bool] {.async.}, internal
`waitFor X` -> `await X`, internal `sleep(N)` ->
`await sleepAsync(chronos.milliseconds(N))`.
- Renamed test_lifecycle.nim's helper proc from `waitFor(t: Job, ...)`
-> `pollExists(t: Job, ...)`; the previous name shadowed
chronos.waitFor in the chronos macro expansion.
- `chronos.milliseconds(N)` explicitly qualified because `std/times`
also exports `milliseconds` (returning TimeInterval, not Duration).
- `check await x` -> `let okN = await x; check okN` to dodge chronos's
"yield in expr not lowered" with await-as-macro-argument.
- `(await x).foo()` -> `let awN = await x; ... awN.foo() ...` for the
same reason.
waku/persistency/persistency.nim: nph also pulled the proc signatures
across multiple lines; restored explicit `Future[void] {.async.}`
return types after the colon (an intermediate nph pass had elided them).
Suite: 71 / 71 OK against the new async write surface.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* use idiomatic valueOr instead of ifs
* Reworked persistency shutdown, remove not necessary teardown mechanism
* Use const for DefaultStoragePath
* format to follow coding guidelines - no use of result and explicit returns - no functional change
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
