* Only add new roots, not all received
* Fix error in removing recent roots not checking AcceptableWindowSize
* fix merging
* more merging fixes
* merge fixes
* add test for updated merkle roots window
* add pr re-add gauge for proof-generation-duration-seconds
* Decrease AcceptableRootWindowSize for testing
* debug spam log
* linting
* start trackRootChanges call loop immediately
* Fix 5s delay trackRootChanges
* set rpcDelay for root tracking to 10s
* add default params to sendEthCallWithParams
* improve recents roots retrieval and logs
* Use updateRecentRoots to track root changes
* simplify updateRecentRoots
* set root polling to 15s
* set rpc poll delay to 30s
* set acceptablerootwindowsize and root poll delay
* Improve test 'should fetch history correctly' for root cache
* Make root cache handling more efficient
* add contract root cache size as constant and function use fix
* updateRecentRoots comments update
* Update group_manager and tests
* fix linting
* persistency: follow nim-sds 0.3.0 snapshot persistence contract
nim-sds 0.3.0 replaced the ~14 fine-grained per-row Persistence callbacks
with a 5-proc snapshot model (saveChannelMeta / updateHistory / loadChannel
/ dropChannel / setRetrievalHint), all returning Future[Result[...]].
Rewrite waku/persistency/sds_persistency.nim accordingly:
- ChannelMeta is stored as one blob per channel; the message log as
append/evict rows. Categories collapse from 7 to 2 (sds.meta, sds.log).
- Blob transform uses nim-sds' own codecs: snapshot_codec (schema-versioned
protobuf) for ChannelMeta, the SDS wire codec for SdsMessage log rows. The
generic payload_codec/BlobCodec path is retired (removed payload_codec.nim
and test_blob_codec.nim).
- setRetrievalHint is a deliberate no-op: persisted hints are never read back
(loadChannel/ChannelMeta carry none; hints are supplied live via the
onRetrievalHint provider). The closure stays because the field is required.
- Fix the module import spelling (srcDir="sds" => bare module paths), which
the previous adapter got wrong and never compiled against the locked deps.
Add tests/persistency/test_sds_persistency.nim (round-trip, empty-load,
evict, drop) replacing test_blob_codec in test_all. Full persistency suite
passes 74/74 under both refc and ORC.
* Bump to latest nim-sds and nim-brokers 3.1.1
* Update with latest nim-sds changes - removal of setRetrievalHints - not needed
* Make Waku.send a shim of MessagingClient.send
* At ReliableChannelManager mount time, set its default sendHandler
* Remove Waku ref from ReliableChannelManager
* Refactor channels to fit layering with explicit mount
* Fix channels test to perform the new mount sequence
* Add Waku.reliableChannelManager
* liblogosdelivery_start_node mounts ReliableChannelManager after MessagingClient
* Fix test_wakunode_peer_exchange missing lockNewGlobalBrokerContext
* Fix test_protocol missing lockNewGlobalBrokerContext
* Harden provider setup vs. misconfiguration (improper shared broker context situations)
* persistency: per-job SQLite-backed storage layer (singleton, brokered)
Adds a backend-neutral CRUD library at waku/persistency/, plus the
nim-brokers dependency swap that enables it.
Architecture (ports-and-adapters):
* Persistency: process-wide singleton, one root directory.
* Job: one tenant, one DB file, one worker thread, one BrokerContext.
* Backend: SQLite via waku/common/databases/db_sqlite. Uniform schema
kv(category BLOB, key BLOB, payload BLOB) PRIMARY KEY (category, key)
WITHOUT ROWID, WAL mode.
* Writes are fire-and-forget via EventBroker(mt) PersistEvent.
* Reads are async via five RequestBroker(mt) shapes (KvGet, KvExists,
KvScan, KvCount, KvDelete). Reads return Result[T, PersistencyError].
* One storage thread per job; tenants isolated by BrokerContext.
Public surface (waku/persistency/persistency.nim):
Persistency.instance(rootDir) / Persistency.instance() / Persistency.reset()
p.openJob(id) / p.closeJob(id) / p.dropJob(id) / p.close()
p.job(id) / p[id] / p.hasJob(id)
Writes (Job form & string-id form, fire-and-forget):
persist / persistPut / persistDelete / persistEncoded
Reads (Job form & string-id form, async Result):
get / exists / scan / scanPrefix / count / deleteAcked
Key & payload encoding (keys.nim, payload.nim):
* encodePart family + variadic key(...) / payload(...) macros +
single-value toKey / toPayload.
* Primitives: string and openArray[byte] are 2-byte BE length + bytes;
int{8..64} are sign-flipped 8-byte BE; uint{16..64} are 8-byte BE;
bool/byte/char are 1 byte; enums are int64(ord(v)).
* Generic encodePart[T: tuple | object] recurses through fields() so
any composite Nim type is encodable without ceremony.
* Stable across Nim/C compiler upgrades: no sizeof, no memcpy, no
cast on pointers, no host-endianness dependency.
* `rawKey(bytes)` + `persistPut(..., openArray[byte])` let callers
bypass the built-in encoder with their own format (CBOR, protobuf...).
Lifecycle:
* Persistency.new is private; Persistency.instance is the only public
constructor. Same rootDir is idempotent; conflicting rootDir is
peInvalidArgument. Persistency.reset for test/restart paths.
* openJob opens-or-creates the per-job SQLite file; an existing file
is reused with its data preserved.
* Teardown integration: Persistency.instance registers a Teardown
MultiRequestBroker provider that closes all jobs and clears the
singleton slot when Waku.stop() issues Teardown.request.
Internal layering:
types.nim pure value types (Key, KeyRange, KvRow, TxOp,
PersistencyError)
keys.nim encodePart primitives + key(...) macro
payload.nim toPayload + payload(...) macro
schema.nim CREATE TABLE + connection pragmas + user_version
backend_sqlite.nim KvBackend, applyOps (single source of write SQL),
getOne/existsOne/deleteOne, scanRange (asc/desc,
half-open ranges, open-ended stop), countRange
backend_comm.nim EventBroker(mt) PersistEvent + 5 RequestBroker(mt)
declarations; encodeErr/decodeErr boundary helpers
backend_thread.nim startStorageThread / stopStorageThread (shared
allocShared0 arg, cstring dbPath, atomic
ready/shutdown flags); per-thread provider
registration
persistency.nim Persistency + Job types, singleton state, public
facade
../requests/lifecycle_requests.nim
Teardown MultiRequestBroker
Tests (69 cases, all passing):
test_keys.nim sort-order invariants (length-prefix strings,
sign-flipped ints, composite tuples, prefix
range)
test_backend.nim round-trip / replace / delete-return-value /
batched atomicity / asc-desc-half-open-open-
ended scans / category isolation / batch
txDelete
test_lifecycle.nim open-or-create rootDir / non-dir collision /
reopen across sessions / idempotent openJob /
two-tenant parallel isolation / closeJob joins
worker / dropJob removes file / acked delete
test_facade.nim put-then-get / atomic batch / scanPrefix
asc/desc / deleteAcked hit-miss /
fire-and-forget delete / two-tenant facade
isolation
test_encoding.nim tuple/named-tuple/object keys, embedded Key,
enum encoding, field-major composite sort,
payload struct encoding, end-to-end struct
round-trip through SQLite
test_string_lookup.nim peJobNotFound semantics / hasJob / subscript /
persistPut+get via id / reads short-circuit /
writes drop+warn / persistEncoded via id /
scan parity Job-ref vs id
test_singleton.nim idempotent same-rootDir / different-rootDir
rejection / no-arg instance lifecycle / reset
retargets / reset idempotence / Teardown.request
end-to-end
Prerequisite delivered in the same series: replace the in-tree broker
implementation with the external nim-brokers package; update all
broker call-sites (waku_filter_v2, waku_relay, waku_rln_relay,
delivery_service, peer_manager, requests/*, factory/*, api tests, etc.)
to the new package API; chat2 made to compile again.
Note: SDS adapter (Phase 5 of the design) is deferred -- nim-sds is
still developed side-by-side and the persistency layer is intentionally
SDS-agnostic.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* persistency: pin nim-brokers by URL+commit (workaround for stale registry)
The bare `brokers >= 2.0.1` form cannot resolve on machines where the
local nimble SAT solver enumerates only the registry-recorded 0.1.0 for
brokers. The nim-lang/packages entry for `brokers` carries no per-tag
metadata (only the URL), so until that registry entry is refreshed the
SAT solver clamps the available-versions list to 0.1.0 and rejects the
>= 2.0.1 constraint -- even though pkgs2 and pkgcache both have v2.0.1
cloned locally.
Pinning by URL+commit bypasses the registry path entirely. Inline
comment in waku.nimble documents the situation and the path back to
the bare form once nim-lang/packages is updated.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* persistency: nph format pass
Run `nph` on all 57 Nim files touched by this PR. Pure formatting:
17 files re-styled, no semantic change. Suite still 69/69.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Fix build, add local-storage-path config, lazy init of Persistency from Waku start
* fix: fix nix deps
* fixes for nix build, regenerate deps
* reverting accidental dependency changes
* Fixing deps
* Apply suggestions from code review
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* persistency tests: migrate to suite / asyncTest / await
Match the in-tree test convention (procSuite -> suite, sync test +
waitFor -> asyncTest + await):
- procSuite "X": -> suite "X":
- For tests doing async work: test -> asyncTest, waitFor -> await.
- Poll helpers (proc waitFor(t: Job, ...) in test_lifecycle.nim,
proc waitUntilExists(...) in test_facade.nim and
test_string_lookup.nim) -> Future[bool] {.async.}, internal
`waitFor X` -> `await X`, internal `sleep(N)` ->
`await sleepAsync(chronos.milliseconds(N))`.
- Renamed test_lifecycle.nim's helper proc from `waitFor(t: Job, ...)`
-> `pollExists(t: Job, ...)`; the previous name shadowed
chronos.waitFor in the chronos macro expansion.
- `chronos.milliseconds(N)` explicitly qualified because `std/times`
also exports `milliseconds` (returning TimeInterval, not Duration).
- `check await x` -> `let okN = await x; check okN` to dodge chronos's
"yield in expr not lowered" with await-as-macro-argument.
- `(await x).foo()` -> `let awN = await x; ... awN.foo() ...` for the
same reason.
waku/persistency/persistency.nim: nph also pulled the proc signatures
across multiple lines; restored explicit `Future[void] {.async.}`
return types after the colon (an intermediate nph pass had elided them).
Suite: 71 / 71 OK against the new async write surface.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* use idiomatic valueOr instead of ifs
* Reworked persistency shutdown, remove not necessary teardown mechanism
* Use const for DefaultStoragePath
* format to follow coding guidelines - no use of result and explicit returns - no functional change
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* Add createNode(preset, mode, overrides, additions) nim api
* Set p2pTcp/discv5Udp/websocket ports to 0 (auto-bind) in new createNode()
* Soft-deprecate --cluster-id=N triggering the associated preset selection
* Rewrite applyNetworkConf to apply user-set fields over preset fields
* Generate WakuNodeConfOverlay (all Option fields) from WakuNodeConf
* New parser for configJson handles new messaging shape and full conf shape
* Change all confbuilder defaults from literal values to DefaultXXX consts
* Change int/bool WakuNodeConf fields to Option to get user intent w/o sentinels
* Make Option CLI default-value help mention defaults now owned by confbuilder
* Misc refactors, fixes
* Add tests
* any port set to 0 on conf results in a random port bound
* Debug API MyBoundPorts reports actually bound ports for all services, reports 0 if disabled
* write back bound values to both WakuConf and WakuNode.ports
* setupDiscoveryV5 returns Result and errors out on port 0
* rename setupAndStartDiscv5WithAutoPort to setupAndStartDiscv5
* updateWaku ENR rebuild now runs after discv5 startup
* Add DefaultP2pTcpPort, DefaultDiscv5UdpPort, DefaultWebSocketPort, DefaultRestPort, DefaultMetricsHttpPort
* add tests
feat: active filter subscription management for edge nodes
## Subscription Manager
* edgeFilterSubLoop reconciles desired vs actual filter subscriptions
* edgeFilterHealthLoop pings filter peers, evicts stale ones
* EdgeFilterSubState per-shard tracking of confirmed peers and health
* best-effort unsubscribe on peer removal
* RequestEdgeShardHealth and RequestEdgeFilterPeerCount broker providers
## WakuNode
* Remove old edge health loop (loopEdgeHealth, edgeHealthEvent, calculateEdgeTopicHealth)
* Register MessageSeenEvent push handler on filter client during start
* startDeliveryService now returns `Result[void, string]` and propagates errors
## Health Monitor
* getFilterClientHealth queries RequestEdgeFilterPeerCount via broker
* Shard/content health providers fall back to RequestEdgeShardHealth when relay inactive
* Listen to EventShardTopicHealthChange for health recalculation
* Add missing return p.notReady() on failed edge filter peer count request
* HealthyThreshold constant moved to `connection_status.nim`
## Broker types
* RequestEdgeShardHealth, RequestEdgeFilterPeerCount request types
* EventShardTopicHealthChange event type
## Filter Client
* Add timeout parameter to ping proc
## Tests
* Health monitor event tests with per-node lockNewGlobalBrokerContext
* Edge (light client) health update test
* Edge health driven by confirmed filter subscriptions test
* API subscription tests: sub/receive, failover, peer replacement
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
Co-authored by Zoltan Nagy
* refactor retention policy to allow union of several retention policies
* bug fix time retention policy
* add removal of orphan partitions if any
* use nim-http-utils 0.4.1
* LogosDeliveryAPI: NodeConfig -> WakluNodeConf + mode selector and logos.dev preset
* Adjustment made on test, logos.dev preset
* change default agentString from nwaku to logos-delivery
* Add p2pReliability switch to presets and make it default to true.
* Borrow entryNode idea from NodeConfig to WakuNodeConf to easy shortcut among diffrent bootstrap node list all needs different formats
* Fix rateLimit assignment for builder
* Remove Core mode default as we already have a defaul, user must override
* Removed obsolate API createNode with NodeConfig - tests are refactored for WakuNodeConf usage
* Fix failing test due to twn-clusterid(1) default has overwrite for maxMessagSize. Fix readme.
* SubscriptionManager tracks shard and content topic interest
* RecvService emits MessageReceivedEvent on subscribed content topics
* Route MAPI through old Kernel API relay unique-handler infra to avoid code duplication
* Encode current gen-zero network policy: on Core node boot, subscribe to all pubsub topics (all shards)
* Add test_api_subscriptions.nim (basic relay/core testing only)
* Removed any MAPI Edge sub/unsub/receive support code that was there (will add in next PR)
* Hook MessageSeenEvent to Kernel API bus
* Fix MAPI vs Kernel API unique relay handler support
* RecvService delegating topic subs to SubscriptionManager
* RecvService emits MessageReceivedEvent (fully filtered)
* Rename old SubscriptionManager to LegacySubscriptionManager
* Initial for liblogosdelivery library (static & dynamic) based on current state of API.
* nix build support added.
* logosdelivery_example
* Added support for missing logLevel/logFormat in new API create_node
* Added full JSON to NodeConfig support
* Added ctx and ctx.myLib check to avoid uninitialzed calls and crash. Adjusted logosdelivery_example with proper error handling and JSON config format
* target aware install phase
* Fix base64 decode of payload
