* persistency: per-job SQLite-backed storage layer (singleton, brokered)
Adds a backend-neutral CRUD library at waku/persistency/, plus the
nim-brokers dependency swap that enables it.
Architecture (ports-and-adapters):
* Persistency: process-wide singleton, one root directory.
* Job: one tenant, one DB file, one worker thread, one BrokerContext.
* Backend: SQLite via waku/common/databases/db_sqlite. Uniform schema
kv(category BLOB, key BLOB, payload BLOB) PRIMARY KEY (category, key)
WITHOUT ROWID, WAL mode.
* Writes are fire-and-forget via EventBroker(mt) PersistEvent.
* Reads are async via five RequestBroker(mt) shapes (KvGet, KvExists,
KvScan, KvCount, KvDelete). Reads return Result[T, PersistencyError].
* One storage thread per job; tenants isolated by BrokerContext.
Public surface (waku/persistency/persistency.nim):
Persistency.instance(rootDir) / Persistency.instance() / Persistency.reset()
p.openJob(id) / p.closeJob(id) / p.dropJob(id) / p.close()
p.job(id) / p[id] / p.hasJob(id)
Writes (Job form & string-id form, fire-and-forget):
persist / persistPut / persistDelete / persistEncoded
Reads (Job form & string-id form, async Result):
get / exists / scan / scanPrefix / count / deleteAcked
Key & payload encoding (keys.nim, payload.nim):
* encodePart family + variadic key(...) / payload(...) macros +
single-value toKey / toPayload.
* Primitives: string and openArray[byte] are 2-byte BE length + bytes;
int{8..64} are sign-flipped 8-byte BE; uint{16..64} are 8-byte BE;
bool/byte/char are 1 byte; enums are int64(ord(v)).
* Generic encodePart[T: tuple | object] recurses through fields() so
any composite Nim type is encodable without ceremony.
* Stable across Nim/C compiler upgrades: no sizeof, no memcpy, no
cast on pointers, no host-endianness dependency.
* `rawKey(bytes)` + `persistPut(..., openArray[byte])` let callers
bypass the built-in encoder with their own format (CBOR, protobuf...).
Lifecycle:
* Persistency.new is private; Persistency.instance is the only public
constructor. Same rootDir is idempotent; conflicting rootDir is
peInvalidArgument. Persistency.reset for test/restart paths.
* openJob opens-or-creates the per-job SQLite file; an existing file
is reused with its data preserved.
* Teardown integration: Persistency.instance registers a Teardown
MultiRequestBroker provider that closes all jobs and clears the
singleton slot when Waku.stop() issues Teardown.request.
Internal layering:
types.nim pure value types (Key, KeyRange, KvRow, TxOp,
PersistencyError)
keys.nim encodePart primitives + key(...) macro
payload.nim toPayload + payload(...) macro
schema.nim CREATE TABLE + connection pragmas + user_version
backend_sqlite.nim KvBackend, applyOps (single source of write SQL),
getOne/existsOne/deleteOne, scanRange (asc/desc,
half-open ranges, open-ended stop), countRange
backend_comm.nim EventBroker(mt) PersistEvent + 5 RequestBroker(mt)
declarations; encodeErr/decodeErr boundary helpers
backend_thread.nim startStorageThread / stopStorageThread (shared
allocShared0 arg, cstring dbPath, atomic
ready/shutdown flags); per-thread provider
registration
persistency.nim Persistency + Job types, singleton state, public
facade
../requests/lifecycle_requests.nim
Teardown MultiRequestBroker
Tests (69 cases, all passing):
test_keys.nim sort-order invariants (length-prefix strings,
sign-flipped ints, composite tuples, prefix
range)
test_backend.nim round-trip / replace / delete-return-value /
batched atomicity / asc-desc-half-open-open-
ended scans / category isolation / batch
txDelete
test_lifecycle.nim open-or-create rootDir / non-dir collision /
reopen across sessions / idempotent openJob /
two-tenant parallel isolation / closeJob joins
worker / dropJob removes file / acked delete
test_facade.nim put-then-get / atomic batch / scanPrefix
asc/desc / deleteAcked hit-miss /
fire-and-forget delete / two-tenant facade
isolation
test_encoding.nim tuple/named-tuple/object keys, embedded Key,
enum encoding, field-major composite sort,
payload struct encoding, end-to-end struct
round-trip through SQLite
test_string_lookup.nim peJobNotFound semantics / hasJob / subscript /
persistPut+get via id / reads short-circuit /
writes drop+warn / persistEncoded via id /
scan parity Job-ref vs id
test_singleton.nim idempotent same-rootDir / different-rootDir
rejection / no-arg instance lifecycle / reset
retargets / reset idempotence / Teardown.request
end-to-end
Prerequisite delivered in the same series: replace the in-tree broker
implementation with the external nim-brokers package; update all
broker call-sites (waku_filter_v2, waku_relay, waku_rln_relay,
delivery_service, peer_manager, requests/*, factory/*, api tests, etc.)
to the new package API; chat2 made to compile again.
Note: SDS adapter (Phase 5 of the design) is deferred -- nim-sds is
still developed side-by-side and the persistency layer is intentionally
SDS-agnostic.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* persistency: pin nim-brokers by URL+commit (workaround for stale registry)
The bare `brokers >= 2.0.1` form cannot resolve on machines where the
local nimble SAT solver enumerates only the registry-recorded 0.1.0 for
brokers. The nim-lang/packages entry for `brokers` carries no per-tag
metadata (only the URL), so until that registry entry is refreshed the
SAT solver clamps the available-versions list to 0.1.0 and rejects the
>= 2.0.1 constraint -- even though pkgs2 and pkgcache both have v2.0.1
cloned locally.
Pinning by URL+commit bypasses the registry path entirely. Inline
comment in waku.nimble documents the situation and the path back to
the bare form once nim-lang/packages is updated.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* persistency: nph format pass
Run `nph` on all 57 Nim files touched by this PR. Pure formatting:
17 files re-styled, no semantic change. Suite still 69/69.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Fix build, add local-storage-path config, lazy init of Persistency from Waku start
* fix: fix nix deps
* fixes for nix build, regenerate deps
* reverting accidental dependency changes
* Fixing deps
* Apply suggestions from code review
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* persistency tests: migrate to suite / asyncTest / await
Match the in-tree test convention (procSuite -> suite, sync test +
waitFor -> asyncTest + await):
- procSuite "X": -> suite "X":
- For tests doing async work: test -> asyncTest, waitFor -> await.
- Poll helpers (proc waitFor(t: Job, ...) in test_lifecycle.nim,
proc waitUntilExists(...) in test_facade.nim and
test_string_lookup.nim) -> Future[bool] {.async.}, internal
`waitFor X` -> `await X`, internal `sleep(N)` ->
`await sleepAsync(chronos.milliseconds(N))`.
- Renamed test_lifecycle.nim's helper proc from `waitFor(t: Job, ...)`
-> `pollExists(t: Job, ...)`; the previous name shadowed
chronos.waitFor in the chronos macro expansion.
- `chronos.milliseconds(N)` explicitly qualified because `std/times`
also exports `milliseconds` (returning TimeInterval, not Duration).
- `check await x` -> `let okN = await x; check okN` to dodge chronos's
"yield in expr not lowered" with await-as-macro-argument.
- `(await x).foo()` -> `let awN = await x; ... awN.foo() ...` for the
same reason.
waku/persistency/persistency.nim: nph also pulled the proc signatures
across multiple lines; restored explicit `Future[void] {.async.}`
return types after the colon (an intermediate nph pass had elided them).
Suite: 71 / 71 OK against the new async write surface.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* use idiomatic valueOr instead of ifs
* Reworked persistency shutdown, remove not necessary teardown mechanism
* Use const for DefaultStoragePath
* format to follow coding guidelines - no use of result and explicit returns - no functional change
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* any port set to 0 on conf results in a random port bound
* Debug API MyBoundPorts reports actually bound ports for all services, reports 0 if disabled
* write back bound values to both WakuConf and WakuNode.ports
* setupDiscoveryV5 returns Result and errors out on port 0
* rename setupAndStartDiscv5WithAutoPort to setupAndStartDiscv5
* updateWaku ENR rebuild now runs after discv5 startup
* Add DefaultP2pTcpPort, DefaultDiscv5UdpPort, DefaultWebSocketPort, DefaultRestPort, DefaultMetricsHttpPort
* add tests
Simplify NodeHealthMonitor creation
* Force NodeHealthMonitor.new() to set up a WakuNode
* Remove all checks for isNil(node) in NodeHealthMonitor
* Fix tests to use the new NodeHealthMonitor.new()
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* Introduce api/send
Added events and requests for support.
Reworked delivery_monitor into a featured devlivery_service, that
- supports relay publish and lightpush depending on configuration but with fallback options
- if available and configured it utilizes store api to confirm message delivery
- emits message delivery events accordingly
prepare for use in api_example
* Fix edge mode config and test added
* Fix some import issues, start and stop waku shall not throw exception but return with result properly
* Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover
* add api_example app to examples2
* Adapt after merge from master
* Adapt code for using broker context
* Fix brokerCtx settings for all usedbrokers, cover locked node init
* Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api
* More test added
* Fix multi propagate event emit, fix fail send test case
* Fix rebase
* Fix PushMessageHandlers in tests
* adapt libwaku to api changes
* Fix relay test by adapting publish return error in case NoPeersToPublish
* Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos
* Fix rln relay broker, missed brokerCtx
* Fix rest relay test failed, due to publish will fail if no peer avail
* ignore anvil test state file
* Make terst_wakunode_rln_relay broker context aware to fix
* Fix waku rln tests by having them broker context aware
* fix typo in test_app.nim
* chore: do not mount lightpush without relay (fixes#2808)
- Change mountLightPush signature to return Result[void, string]
- Return error when relay is not mounted
- Update all call sites to handle Result return type
- Add test verifying mounting fails without relay
- Only advertise lightpush capability when relay is enabled
* chore: don't mount legacy lightpush without relay
* Bump commit for vendor wakurlnv2contract
* Update RLN registration proc for contract updates
* add option to runAnvil for state dump or load with optional contract deployment on setup
* Code clean up
* Upodate rln relay tests to use cached anvil state
* Minor updates to utils and new test for anvil state dump
* stopAnvil needs to wait for graceful shutdown
* configure runAnvil to use load state in other tests
* reduce ci timeout
* Allow for RunAnvil load state file to be compressed
* Fix linting
* Change return type of sendMintCall to Futre[void]
* Update naming of ci path for interop tests
* fix: admin API peer shards field from metadata protocol
Store and return peer shard info from metadata protocol exchange instead of only checking ENR records.
* peer_manager set shard info and extend rest test to validate it
Co-authored-by: MorganaFuture <andrewmochalskyi@gmail.com>
* Rename waku_api to rest_api and underlying rest to endpoint for clearity
* Rename node/api to node/kernel_api to suggest that it is an internal accessor to node interface + make everything compile after renaming
* make waku api a top level import
* fix use of relative path imports and use default to root rather in case of waku and tools modules
* Reserve `networkconfig` name to waku network related settings
* Rename cluster conf to network conf
A `NetworkConf` is a Waku network configuration.
# Conflicts:
# tests/factory/test_waku_conf.nim
# Conflicts:
# tests/factory/test_waku_conf.nim
* Improve sharding configuration
A smarter data types simplifies the logic.
* Fixing tests
* fixup! rename to endpointConf
* wip: autosharding is a specific configuration state and treat it like
it
# Conflicts:
# waku/factory/external_config.nim
* refactor lightpush handler
some metrics error reporting were missing
# Conflicts:
# waku/waku_lightpush/protocol.nim
* test_node_factory tests pass
* remove warnings
* fix tests
* Revert eager previous replace-all command
* fix up build tools compilation
* metadata is used to store cluster id
* Mount relay routes in static sharding
* Rename activeRelayShards to subscribeShards
To make it clearer that these are the shards the node will subscribe to.
* Remove unused msg var
* Improve error handling
* Set autosharding as default, with 1 shard in network
Also makes shards to subscribe to all shards in auto sharding, none in
static sharding.
* fix: deprecate `--dns-discovery`
Properly deprecates `--dns-discovery` CLI arg.
DNS Discovery is enabled if a non-empty DNS Discovery URL is passed.
* test: add test_all for factory
add and use test_all for some tests.
* waku_relay protocol fix unsubscribe and remove topic validator
* simplify subscription and avoid unneeded code
* tests adaptations
* call wakuRelay.subscribe only in one place within waku_node
* Extend ndoe /health REST endpoint with all protocol's state
* Added check for Rendezvous peers availability
* Fine tune filter, added client protocols to health report
* Fix /health endpoint test
* Add explanatory description for state NOT_READY
* Fix formattings
* Apply suggestions from code review
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* Apply code style changes and extended test
* Fix formatting
---------
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* properly pass userMessageLimit to OnchainGroupManager
* waku.nimble 2.2.4 Nim compiler
* rm stew/shims/net import
* change ValidIpAddress.init with parseIpAddress
* fix serialize for zerokit
* group_manager: separate if statements
* protocol_types: add encode UInt32 with zeros up to 32 bytes
* windows build: skip libunwind build and rm libunwind.a inlcusion step
* bump nph to overcome the compilation issues with 2.2.x
* bump nim-libp2p to v1.10.1
Split `WakuNodeConfig` object for better separation of concerns and to introduce a tree-like structure to configuration.
* fix: ensure twn cluster conf is still applied when clusterId=1
* test: remove usage of `WakuNodeConf`
* Remove macro, split builder files, remove wakunodeconf from tests
* rm network_conf_builder module as it is not used
---------
Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
Co-authored-by: Ivan Folgueira Bande <ivansete@status.im>
* Extended /admin/v1 RESP API with different option to look at current connected/relay/mesh state of the node
* Added score information for peer info retrievals
* Separate new lightpush protocol
New RPC defined
Rename al occurence of old lightpush to legacy lightpush, fix rest tests of lightpush
New lightpush protocol added back
Setup new lightpush protocol, mounting and rest api for it
modified: apps/chat2/chat2.nim
modified: tests/node/test_wakunode_lightpush.nim
modified: tests/node/test_wakunode_sharding.nim
modified: tests/test_peer_manager.nim
modified: tests/test_wakunode_lightpush.nim
renamed: tests/waku_lightpush/lightpush_utils.nim -> tests/waku_lightpush_legacy/lightpush_utils.nim
renamed: tests/waku_lightpush/test_all.nim -> tests/waku_lightpush_legacy/test_all.nim
renamed: tests/waku_lightpush/test_client.nim -> tests/waku_lightpush_legacy/test_client.nim
renamed: tests/waku_lightpush/test_ratelimit.nim -> tests/waku_lightpush_legacy/test_ratelimit.nim
modified: tests/wakunode_rest/test_all.nim
renamed: tests/wakunode_rest/test_rest_lightpush.nim -> tests/wakunode_rest/test_rest_lightpush_legacy.nim
modified: waku/factory/node_factory.nim
modified: waku/node/waku_node.nim
modified: waku/waku_api/rest/admin/handlers.nim
modified: waku/waku_api/rest/builder.nim
new file: waku/waku_api/rest/legacy_lightpush/client.nim
new file: waku/waku_api/rest/legacy_lightpush/handlers.nim
new file: waku/waku_api/rest/legacy_lightpush/types.nim
modified: waku/waku_api/rest/lightpush/client.nim
modified: waku/waku_api/rest/lightpush/handlers.nim
modified: waku/waku_api/rest/lightpush/types.nim
modified: waku/waku_core/codecs.nim
modified: waku/waku_lightpush.nim
modified: waku/waku_lightpush/callbacks.nim
modified: waku/waku_lightpush/client.nim
modified: waku/waku_lightpush/common.nim
modified: waku/waku_lightpush/protocol.nim
modified: waku/waku_lightpush/rpc.nim
modified: waku/waku_lightpush/rpc_codec.nim
modified: waku/waku_lightpush/self_req_handler.nim
new file: waku/waku_lightpush_legacy.nim
renamed: waku/waku_lightpush/README.md -> waku/waku_lightpush_legacy/README.md
new file: waku/waku_lightpush_legacy/callbacks.nim
new file: waku/waku_lightpush_legacy/client.nim
new file: waku/waku_lightpush_legacy/common.nim
new file: waku/waku_lightpush_legacy/protocol.nim
new file: waku/waku_lightpush_legacy/protocol_metrics.nim
new file: waku/waku_lightpush_legacy/rpc.nim
new file: waku/waku_lightpush_legacy/rpc_codec.nim
new file: waku/waku_lightpush_legacy/self_req_handler.nim
Adapt to non-invasive libp2p observers
cherry pick latest lightpush (v1) changes into legacy lightpush code after rebase to latest master
Fix vendor dependencies from origin/master after failed rebase of them
Adjust examples, test to new lightpush - keep using of legacy
Fixup error code mappings
Fix REST admin interface with distinct legacy and new lightpush
Fix lightpush v2 tests
* Utilize new publishEx interface of pubsub libp2p
* Adapt to latest libp2p pubslih design changes. publish returns an outcome as Result error.
* Fix review findings
* Fix tests, re-added lost one
* Fix rebase
* Apply suggestions from code review
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* Addressing review comments
* Fix incentivization tests
* Fix build failed on libwaku
* Change new lightpush endpoint version to 3 instead of 2. Noticed that old and new lightpush metrics can cause trouble in monitoring dashboards so decided to give new name as v3 for the new lightpush metrics and change legacy ones back - temporarly till old lightpush will be decommissioned
* Fixing flaky test with rate limit timing
* Fixing logscope of lightpush and legacy lightpush
---------
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* undo apt install libpcre (not circuit-relay related.)
* nat.nim: protect against possible exceptions when calling getExternalIP
* new external CLI argument, isRelayClient
* waku factory change to mount circuit hop proto by default
* waku_node: move autonat_service to a separate module
* Remove the condition of gathering connected peers with relay and user req/resp protocols.
* Return PeerExchange protocol support of connected nodes with /admin/peers
* Added test for checking return of PeerExchange mounted protocol of connected peer by GET /admin/peers
---------
Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
* DOS protection of non relay protocols - rate limit phase3:
- Enhanced TokenBucket to be able to add compensation tokens based on previous usage percentage,
- per peer rate limiter 'PeerRateLimier' applied on waku_filter_v2 with opinionated default of acceptable request rate
- Add traffic metrics to filter message push
- RequestRateLimiter added to combine simple token bucket limiting of request numbers but consider per peer usage over time and prevent some peers to over use the service
(although currently rule violating peers will not be disconnected by this time only their requests will get not served)
- TimedMap utility created (inspired and taken from libp2p TimedCache) which serves as forgiving feature for peers had been overusing the service.
- Added more tests
- Fix rebase issues
- Applied new RequestRateLimiter for store and legacy_store and lightpush
* Incorporate review comments, typos, file/class naming and placement changes.
* Add issue link reference of the original issue with nim-chronos TokenBucket
* Make TimedEntry of TimedMap private and not mixable with similar named in libp2p
* Fix review comments, renamings, const instead of values and more comments.
* queue driver refactor (#2753)
* chore(archive): archive refactor (#2752)
* chore(archive): sqlite driver refactor (#2754)
* chore(archive): postgres driver refactor (#2755)
* chore(archive): renaming & copies (#2751)
* posgres legacy: stop using the storedAt field
* migration script 6: we still need the id column
The id column is needed because it contains the message digest
which is used in store v2, and we need to keep support to
store v2 for a while
* legacy archive: set target migration version to 6
* waku_node: try to use wakuLegacyArchive if wakuArchive is nil
* node_factory, waku_node: mount legacy and future store simultaneously
We want the nwaku node to simultaneously support store-v2
requests and store-v3 requests.
Only the legacy archive is in charge of archiving messages, and the
archived information is suitable to fulfill both store-v2 and
store-v3 needs.
* postgres_driver: adding temporary code until store-v2 is removed
---------
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Ivan Folgueira Bande <ivansete@status.im>
* chore: add unit test for testing duplicate message push with timedcache
* chore: update according to better naming convention
---------
Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
* chore: update logs with topic discription & debug msg
* chore: update unit according to error msg
* chore: update rest unit according to error msg
* chore: add content-topic with debug msg
