* persistency: per-job SQLite-backed storage layer (singleton, brokered)
Adds a backend-neutral CRUD library at waku/persistency/, plus the
nim-brokers dependency swap that enables it.
Architecture (ports-and-adapters):
* Persistency: process-wide singleton, one root directory.
* Job: one tenant, one DB file, one worker thread, one BrokerContext.
* Backend: SQLite via waku/common/databases/db_sqlite. Uniform schema
kv(category BLOB, key BLOB, payload BLOB) PRIMARY KEY (category, key)
WITHOUT ROWID, WAL mode.
* Writes are fire-and-forget via EventBroker(mt) PersistEvent.
* Reads are async via five RequestBroker(mt) shapes (KvGet, KvExists,
KvScan, KvCount, KvDelete). Reads return Result[T, PersistencyError].
* One storage thread per job; tenants isolated by BrokerContext.
Public surface (waku/persistency/persistency.nim):
Persistency.instance(rootDir) / Persistency.instance() / Persistency.reset()
p.openJob(id) / p.closeJob(id) / p.dropJob(id) / p.close()
p.job(id) / p[id] / p.hasJob(id)
Writes (Job form & string-id form, fire-and-forget):
persist / persistPut / persistDelete / persistEncoded
Reads (Job form & string-id form, async Result):
get / exists / scan / scanPrefix / count / deleteAcked
Key & payload encoding (keys.nim, payload.nim):
* encodePart family + variadic key(...) / payload(...) macros +
single-value toKey / toPayload.
* Primitives: string and openArray[byte] are 2-byte BE length + bytes;
int{8..64} are sign-flipped 8-byte BE; uint{16..64} are 8-byte BE;
bool/byte/char are 1 byte; enums are int64(ord(v)).
* Generic encodePart[T: tuple | object] recurses through fields() so
any composite Nim type is encodable without ceremony.
* Stable across Nim/C compiler upgrades: no sizeof, no memcpy, no
cast on pointers, no host-endianness dependency.
* `rawKey(bytes)` + `persistPut(..., openArray[byte])` let callers
bypass the built-in encoder with their own format (CBOR, protobuf...).
Lifecycle:
* Persistency.new is private; Persistency.instance is the only public
constructor. Same rootDir is idempotent; conflicting rootDir is
peInvalidArgument. Persistency.reset for test/restart paths.
* openJob opens-or-creates the per-job SQLite file; an existing file
is reused with its data preserved.
* Teardown integration: Persistency.instance registers a Teardown
MultiRequestBroker provider that closes all jobs and clears the
singleton slot when Waku.stop() issues Teardown.request.
Internal layering:
types.nim pure value types (Key, KeyRange, KvRow, TxOp,
PersistencyError)
keys.nim encodePart primitives + key(...) macro
payload.nim toPayload + payload(...) macro
schema.nim CREATE TABLE + connection pragmas + user_version
backend_sqlite.nim KvBackend, applyOps (single source of write SQL),
getOne/existsOne/deleteOne, scanRange (asc/desc,
half-open ranges, open-ended stop), countRange
backend_comm.nim EventBroker(mt) PersistEvent + 5 RequestBroker(mt)
declarations; encodeErr/decodeErr boundary helpers
backend_thread.nim startStorageThread / stopStorageThread (shared
allocShared0 arg, cstring dbPath, atomic
ready/shutdown flags); per-thread provider
registration
persistency.nim Persistency + Job types, singleton state, public
facade
../requests/lifecycle_requests.nim
Teardown MultiRequestBroker
Tests (69 cases, all passing):
test_keys.nim sort-order invariants (length-prefix strings,
sign-flipped ints, composite tuples, prefix
range)
test_backend.nim round-trip / replace / delete-return-value /
batched atomicity / asc-desc-half-open-open-
ended scans / category isolation / batch
txDelete
test_lifecycle.nim open-or-create rootDir / non-dir collision /
reopen across sessions / idempotent openJob /
two-tenant parallel isolation / closeJob joins
worker / dropJob removes file / acked delete
test_facade.nim put-then-get / atomic batch / scanPrefix
asc/desc / deleteAcked hit-miss /
fire-and-forget delete / two-tenant facade
isolation
test_encoding.nim tuple/named-tuple/object keys, embedded Key,
enum encoding, field-major composite sort,
payload struct encoding, end-to-end struct
round-trip through SQLite
test_string_lookup.nim peJobNotFound semantics / hasJob / subscript /
persistPut+get via id / reads short-circuit /
writes drop+warn / persistEncoded via id /
scan parity Job-ref vs id
test_singleton.nim idempotent same-rootDir / different-rootDir
rejection / no-arg instance lifecycle / reset
retargets / reset idempotence / Teardown.request
end-to-end
Prerequisite delivered in the same series: replace the in-tree broker
implementation with the external nim-brokers package; update all
broker call-sites (waku_filter_v2, waku_relay, waku_rln_relay,
delivery_service, peer_manager, requests/*, factory/*, api tests, etc.)
to the new package API; chat2 made to compile again.
Note: SDS adapter (Phase 5 of the design) is deferred -- nim-sds is
still developed side-by-side and the persistency layer is intentionally
SDS-agnostic.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* persistency: pin nim-brokers by URL+commit (workaround for stale registry)
The bare `brokers >= 2.0.1` form cannot resolve on machines where the
local nimble SAT solver enumerates only the registry-recorded 0.1.0 for
brokers. The nim-lang/packages entry for `brokers` carries no per-tag
metadata (only the URL), so until that registry entry is refreshed the
SAT solver clamps the available-versions list to 0.1.0 and rejects the
>= 2.0.1 constraint -- even though pkgs2 and pkgcache both have v2.0.1
cloned locally.
Pinning by URL+commit bypasses the registry path entirely. Inline
comment in waku.nimble documents the situation and the path back to
the bare form once nim-lang/packages is updated.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* persistency: nph format pass
Run `nph` on all 57 Nim files touched by this PR. Pure formatting:
17 files re-styled, no semantic change. Suite still 69/69.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Fix build, add local-storage-path config, lazy init of Persistency from Waku start
* fix: fix nix deps
* fixes for nix build, regenerate deps
* reverting accidental dependency changes
* Fixing deps
* Apply suggestions from code review
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* persistency tests: migrate to suite / asyncTest / await
Match the in-tree test convention (procSuite -> suite, sync test +
waitFor -> asyncTest + await):
- procSuite "X": -> suite "X":
- For tests doing async work: test -> asyncTest, waitFor -> await.
- Poll helpers (proc waitFor(t: Job, ...) in test_lifecycle.nim,
proc waitUntilExists(...) in test_facade.nim and
test_string_lookup.nim) -> Future[bool] {.async.}, internal
`waitFor X` -> `await X`, internal `sleep(N)` ->
`await sleepAsync(chronos.milliseconds(N))`.
- Renamed test_lifecycle.nim's helper proc from `waitFor(t: Job, ...)`
-> `pollExists(t: Job, ...)`; the previous name shadowed
chronos.waitFor in the chronos macro expansion.
- `chronos.milliseconds(N)` explicitly qualified because `std/times`
also exports `milliseconds` (returning TimeInterval, not Duration).
- `check await x` -> `let okN = await x; check okN` to dodge chronos's
"yield in expr not lowered" with await-as-macro-argument.
- `(await x).foo()` -> `let awN = await x; ... awN.foo() ...` for the
same reason.
waku/persistency/persistency.nim: nph also pulled the proc signatures
across multiple lines; restored explicit `Future[void] {.async.}`
return types after the colon (an intermediate nph pass had elided them).
Suite: 71 / 71 OK against the new async write surface.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* use idiomatic valueOr instead of ifs
* Reworked persistency shutdown, remove not necessary teardown mechanism
* Use const for DefaultStoragePath
* format to follow coding guidelines - no use of result and explicit returns - no functional change
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* Introduce api/send
Added events and requests for support.
Reworked delivery_monitor into a featured devlivery_service, that
- supports relay publish and lightpush depending on configuration but with fallback options
- if available and configured it utilizes store api to confirm message delivery
- emits message delivery events accordingly
prepare for use in api_example
* Fix edge mode config and test added
* Fix some import issues, start and stop waku shall not throw exception but return with result properly
* Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover
* add api_example app to examples2
* Adapt after merge from master
* Adapt code for using broker context
* Fix brokerCtx settings for all usedbrokers, cover locked node init
* Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api
* More test added
* Fix multi propagate event emit, fix fail send test case
* Fix rebase
* Fix PushMessageHandlers in tests
* adapt libwaku to api changes
* Fix relay test by adapting publish return error in case NoPeersToPublish
* Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos
* Fix rln relay broker, missed brokerCtx
* Fix rest relay test failed, due to publish will fail if no peer avail
* ignore anvil test state file
* Make terst_wakunode_rln_relay broker context aware to fix
* Fix waku rln tests by having them broker context aware
* fix typo in test_app.nim
* fix: hash inputs for external nullifier, remove length prefix for sha256
* feat: use nimcrypto keccak instead of sha256 ffi
* feat: wrapper function to generate external nullifier
* Bump commit for vendor wakurlnv2contract
* Update RLN registration proc for contract updates
* add option to runAnvil for state dump or load with optional contract deployment on setup
* Code clean up
* Upodate rln relay tests to use cached anvil state
* Minor updates to utils and new test for anvil state dump
* stopAnvil needs to wait for graceful shutdown
* configure runAnvil to use load state in other tests
* reduce ci timeout
* Allow for RunAnvil load state file to be compressed
* Fix linting
* Change return type of sendMintCall to Futre[void]
* Update naming of ci path for interop tests
* update RLN contract abi functions and procs
* Clean up debugging lines
* Use more descriptive object field names for MembershipInfo
* Fix formatting
* fix group_manager after rebase to use new contract method sig
* Fix linting for group_manager.nim
* Test idcommitment to BE and debug logs
* Improve IdCommitment logging
* Update all keystore credentials to use BE format
* Add workaround for groupmanager web3 eth_call
* Add await to sendEthCallWithChainID
* Add error handling for failed eth_call
* Improve error handling for eth_call workaround
* Revert keystore credentials back to using LE
* Update toRateCommitment proc to use LE instead of BE
* Add IdCommitment to calldata as BE
* feat: Update rln contract deployment and tests (#3408)
* update RLN contract abi functions and procs
* update waku-rlnv2-contract submodule commit to latest
* Add RlnV2 contract deployment using forge scripts
* Clean up output of forge script command, debug logs to trace, warn to error
* Move TestToken deployment to own proc
* first implementation of token minting and approval
* Update rln tests with usermessagelimit new minimum
* Clean up code and error handling
* Rework RLN tests WIP
* Fix RLN test for new contract
* RLN Tests updated
* Fix formatting
* Improve error logs
* Fix error message formatting
* Fix linting
* Add pnpm dependency installation for rln tests
* Update test dependencies in makefile
* Minor updates, error messages etc
* Code cleanup and change some debug logging to trace
* Improve handling of Result return value
* Use absolute path for waku-rlnv2-contract
* Simplify token approval and balance check
* Remove unused Anvil options
* Add additional checks for stopAnvil process
* Fix anvil process call to null
* Add lock to tests for rln_group_manager_onchain
* Debug for forge command
* Verify paths
* Install pnpm as global
* Cleanup anvil running procs
* Add check before installing anvil
* CLean up onchain group_manager
* Add proc to setup environment for contract deployer
* Refactoring and improved error handling
* Fix anvil install directory string
* Fix linting in test_range_split
* Add const for the contract address length
* Add separate checks for why Approval transaction fails
* Update RLN contract address and chainID for TWN
* waku_relay protocol fix unsubscribe and remove topic validator
* simplify subscription and avoid unneeded code
* tests adaptations
* call wakuRelay.subscribe only in one place within waku_node
* properly pass userMessageLimit to OnchainGroupManager
* waku.nimble 2.2.4 Nim compiler
* rm stew/shims/net import
* change ValidIpAddress.init with parseIpAddress
* fix serialize for zerokit
* group_manager: separate if statements
* protocol_types: add encode UInt32 with zeros up to 32 bytes
* windows build: skip libunwind build and rm libunwind.a inlcusion step
* bump nph to overcome the compilation issues with 2.2.x
* bump nim-libp2p to v1.10.1
* use of multiple Eth clients instead of just one
* config_chat2 enhance param comment
* group_manager: raise exception if could not connect to any of the eth clients
Split `WakuNodeConfig` object for better separation of concerns and to introduce a tree-like structure to configuration.
* fix: ensure twn cluster conf is still applied when clusterId=1
* test: remove usage of `WakuNodeConf`
* Remove macro, split builder files, remove wakunodeconf from tests
* rm network_conf_builder module as it is not used
---------
Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
Co-authored-by: Ivan Folgueira Bande <ivansete@status.im>
* store test adaptations because the tests were using future times
* waku_store_sync.nim, test_waku_archive, test_rln_group_manager_onchain.nim nph (unrelated change)
Implement data structures and tests for checking transaction eligibility based on tx hash. This work will be continues in future PRs. All code added in this PR is only used in tests.
* feat: add simple txid-based eligibility check with hard-coded params (#3166)
* use new proc to generate eligibility status
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
* minor fixes
* add comments to clarify eligibility definition
* use Address.fromHex conversion from eth-web3
* move isEligible to common
* refactor: avoid result and unnecesary branching
* define const for simple transfer gas usage
* avoid unnecessary parentheses
* chore: run nph linter manually
* refactor, move all hard-coded constants to tests
* use Result type in eligibility tests
* use standard method of error handling
* make try-block smaller
* add a try-block in case of connection failure to web3 provider
* make queries to web3 provider in parallel
* move Web3 provider RPC URL into env variable
* remove unused import
* rename functions
* use await in async proc
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
* add timeout to tx receipt query
* parallelize queries for tx and txreceipt
* make test txids non public
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* use assert in txid i13n test
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* use parentheses when calling verb-methods without arguments
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* remove unused import
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* use init for stack-allocated objects
* add txReceipt error message to error
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* introduce eligibility manager
* [WIP] use Anvil for eligibility testing
* add eligibility test with contract deployment tx
* add eligibility test with contract call
* add asyncSetup and asyncTeardown for eligibility tests
* minor refactor
* refactor tests for onchain group manager with asyncSetup and asyncTeardown
* minor refactor
* remove unnecessary defer in asyncTeardown
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* remove unnecessary call in test (moved to asyncTeardown)
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* add comment justidying the use of discard
* rename file txid_proof to eligibility_manager
---------
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
