logos-messaging/logos-messaging-nim
1
0
Fork 0
mirror of https://github.com/logos-messaging/logos-messaging-nim.git synced 2026-05-18 00:09:52 +00:00
Code Issues Packages Projects Releases Wiki Activity
logos-messaging-nim/tests/api/test_api_send.nim

438 lines
15 KiB
Nim
Raw Normal View History

feat: waku api send (#3669) * Introduce api/send Added events and requests for support. Reworked delivery_monitor into a featured devlivery_service, that - supports relay publish and lightpush depending on configuration but with fallback options - if available and configured it utilizes store api to confirm message delivery - emits message delivery events accordingly prepare for use in api_example * Fix edge mode config and test added * Fix some import issues, start and stop waku shall not throw exception but return with result properly * Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover * add api_example app to examples2 * Adapt after merge from master * Adapt code for using broker context * Fix brokerCtx settings for all usedbrokers, cover locked node init * Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api * More test added * Fix multi propagate event emit, fix fail send test case * Fix rebase * Fix PushMessageHandlers in tests * adapt libwaku to api changes * Fix relay test by adapting publish return error in case NoPeersToPublish * Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos * Fix rln relay broker, missed brokerCtx * Fix rest relay test failed, due to publish will fail if no peer avail * ignore anvil test state file * Make terst_wakunode_rln_relay broker context aware to fix * Fix waku rln tests by having them broker context aware * fix typo in test_app.nim
2026-01-30 01:06:00 +01:00
{.used.}
import std/strutils
import chronos, testutils/unittests, stew/byteutils, libp2p/[switch, peerinfo]
feat: persistency (#3880) * persistency: per-job SQLite-backed storage layer (singleton, brokered) Adds a backend-neutral CRUD library at waku/persistency/, plus the nim-brokers dependency swap that enables it. Architecture (ports-and-adapters): * Persistency: process-wide singleton, one root directory. * Job: one tenant, one DB file, one worker thread, one BrokerContext. * Backend: SQLite via waku/common/databases/db_sqlite. Uniform schema kv(category BLOB, key BLOB, payload BLOB) PRIMARY KEY (category, key) WITHOUT ROWID, WAL mode. * Writes are fire-and-forget via EventBroker(mt) PersistEvent. * Reads are async via five RequestBroker(mt) shapes (KvGet, KvExists, KvScan, KvCount, KvDelete). Reads return Result[T, PersistencyError]. * One storage thread per job; tenants isolated by BrokerContext. Public surface (waku/persistency/persistency.nim): Persistency.instance(rootDir) / Persistency.instance() / Persistency.reset() p.openJob(id) / p.closeJob(id) / p.dropJob(id) / p.close() p.job(id) / p[id] / p.hasJob(id) Writes (Job form & string-id form, fire-and-forget): persist / persistPut / persistDelete / persistEncoded Reads (Job form & string-id form, async Result): get / exists / scan / scanPrefix / count / deleteAcked Key & payload encoding (keys.nim, payload.nim): * encodePart family + variadic key(...) / payload(...) macros + single-value toKey / toPayload. * Primitives: string and openArray[byte] are 2-byte BE length + bytes; int{8..64} are sign-flipped 8-byte BE; uint{16..64} are 8-byte BE; bool/byte/char are 1 byte; enums are int64(ord(v)). * Generic encodePart[T: tuple | object] recurses through fields() so any composite Nim type is encodable without ceremony. * Stable across Nim/C compiler upgrades: no sizeof, no memcpy, no cast on pointers, no host-endianness dependency. * `rawKey(bytes)` + `persistPut(..., openArray[byte])` let callers bypass the built-in encoder with their own format (CBOR, protobuf...). Lifecycle: * Persistency.new is private; Persistency.instance is the only public constructor. Same rootDir is idempotent; conflicting rootDir is peInvalidArgument. Persistency.reset for test/restart paths. * openJob opens-or-creates the per-job SQLite file; an existing file is reused with its data preserved. * Teardown integration: Persistency.instance registers a Teardown MultiRequestBroker provider that closes all jobs and clears the singleton slot when Waku.stop() issues Teardown.request. Internal layering: types.nim pure value types (Key, KeyRange, KvRow, TxOp, PersistencyError) keys.nim encodePart primitives + key(...) macro payload.nim toPayload + payload(...) macro schema.nim CREATE TABLE + connection pragmas + user_version backend_sqlite.nim KvBackend, applyOps (single source of write SQL), getOne/existsOne/deleteOne, scanRange (asc/desc, half-open ranges, open-ended stop), countRange backend_comm.nim EventBroker(mt) PersistEvent + 5 RequestBroker(mt) declarations; encodeErr/decodeErr boundary helpers backend_thread.nim startStorageThread / stopStorageThread (shared allocShared0 arg, cstring dbPath, atomic ready/shutdown flags); per-thread provider registration persistency.nim Persistency + Job types, singleton state, public facade ../requests/lifecycle_requests.nim Teardown MultiRequestBroker Tests (69 cases, all passing): test_keys.nim sort-order invariants (length-prefix strings, sign-flipped ints, composite tuples, prefix range) test_backend.nim round-trip / replace / delete-return-value / batched atomicity / asc-desc-half-open-open- ended scans / category isolation / batch txDelete test_lifecycle.nim open-or-create rootDir / non-dir collision / reopen across sessions / idempotent openJob / two-tenant parallel isolation / closeJob joins worker / dropJob removes file / acked delete test_facade.nim put-then-get / atomic batch / scanPrefix asc/desc / deleteAcked hit-miss / fire-and-forget delete / two-tenant facade isolation test_encoding.nim tuple/named-tuple/object keys, embedded Key, enum encoding, field-major composite sort, payload struct encoding, end-to-end struct round-trip through SQLite test_string_lookup.nim peJobNotFound semantics / hasJob / subscript / persistPut+get via id / reads short-circuit / writes drop+warn / persistEncoded via id / scan parity Job-ref vs id test_singleton.nim idempotent same-rootDir / different-rootDir rejection / no-arg instance lifecycle / reset retargets / reset idempotence / Teardown.request end-to-end Prerequisite delivered in the same series: replace the in-tree broker implementation with the external nim-brokers package; update all broker call-sites (waku_filter_v2, waku_relay, waku_rln_relay, delivery_service, peer_manager, requests/*, factory/*, api tests, etc.) to the new package API; chat2 made to compile again. Note: SDS adapter (Phase 5 of the design) is deferred -- nim-sds is still developed side-by-side and the persistency layer is intentionally SDS-agnostic. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * persistency: pin nim-brokers by URL+commit (workaround for stale registry) The bare `brokers >= 2.0.1` form cannot resolve on machines where the local nimble SAT solver enumerates only the registry-recorded 0.1.0 for brokers. The nim-lang/packages entry for `brokers` carries no per-tag metadata (only the URL), so until that registry entry is refreshed the SAT solver clamps the available-versions list to 0.1.0 and rejects the >= 2.0.1 constraint -- even though pkgs2 and pkgcache both have v2.0.1 cloned locally. Pinning by URL+commit bypasses the registry path entirely. Inline comment in waku.nimble documents the situation and the path back to the bare form once nim-lang/packages is updated. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * persistency: nph format pass Run `nph` on all 57 Nim files touched by this PR. Pure formatting: 17 files re-styled, no semantic change. Suite still 69/69. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Fix build, add local-storage-path config, lazy init of Persistency from Waku start * fix: fix nix deps * fixes for nix build, regenerate deps * reverting accidental dependency changes * Fixing deps * Apply suggestions from code review Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com> * persistency tests: migrate to suite / asyncTest / await Match the in-tree test convention (procSuite -> suite, sync test + waitFor -> asyncTest + await): - procSuite "X": -> suite "X": - For tests doing async work: test -> asyncTest, waitFor -> await. - Poll helpers (proc waitFor(t: Job, ...) in test_lifecycle.nim, proc waitUntilExists(...) in test_facade.nim and test_string_lookup.nim) -> Future[bool] {.async.}, internal `waitFor X` -> `await X`, internal `sleep(N)` -> `await sleepAsync(chronos.milliseconds(N))`. - Renamed test_lifecycle.nim's helper proc from `waitFor(t: Job, ...)` -> `pollExists(t: Job, ...)`; the previous name shadowed chronos.waitFor in the chronos macro expansion. - `chronos.milliseconds(N)` explicitly qualified because `std/times` also exports `milliseconds` (returning TimeInterval, not Duration). - `check await x` -> `let okN = await x; check okN` to dodge chronos's "yield in expr not lowered" with await-as-macro-argument. - `(await x).foo()` -> `let awN = await x; ... awN.foo() ...` for the same reason. waku/persistency/persistency.nim: nph also pulled the proc signatures across multiple lines; restored explicit `Future[void] {.async.}` return types after the colon (an intermediate nph pass had elided them). Suite: 71 / 71 OK against the new async write surface. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * use idiomatic valueOr instead of ifs * Reworked persistency shutdown, remove not necessary teardown mechanism * Use const for DefaultStoragePath * format to follow coding guidelines - no use of result and explicit returns - no functional change --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2026-05-16 00:09:07 +02:00
import brokers/broker_context
feat: waku api send (#3669) * Introduce api/send Added events and requests for support. Reworked delivery_monitor into a featured devlivery_service, that - supports relay publish and lightpush depending on configuration but with fallback options - if available and configured it utilizes store api to confirm message delivery - emits message delivery events accordingly prepare for use in api_example * Fix edge mode config and test added * Fix some import issues, start and stop waku shall not throw exception but return with result properly * Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover * add api_example app to examples2 * Adapt after merge from master * Adapt code for using broker context * Fix brokerCtx settings for all usedbrokers, cover locked node init * Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api * More test added * Fix multi propagate event emit, fix fail send test case * Fix rebase * Fix PushMessageHandlers in tests * adapt libwaku to api changes * Fix relay test by adapting publish return error in case NoPeersToPublish * Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos * Fix rln relay broker, missed brokerCtx * Fix rest relay test failed, due to publish will fail if no peer avail * ignore anvil test state file * Make terst_wakunode_rln_relay broker context aware to fix * Fix waku rln tests by having them broker context aware * fix typo in test_app.nim
2026-01-30 01:06:00 +01:00
import ../testlib/[common, wakucore, wakunode, testasync]
import ../waku_archive/archive_utils
feat: persistency (#3880) * persistency: per-job SQLite-backed storage layer (singleton, brokered) Adds a backend-neutral CRUD library at waku/persistency/, plus the nim-brokers dependency swap that enables it. Architecture (ports-and-adapters): * Persistency: process-wide singleton, one root directory. * Job: one tenant, one DB file, one worker thread, one BrokerContext. * Backend: SQLite via waku/common/databases/db_sqlite. Uniform schema kv(category BLOB, key BLOB, payload BLOB) PRIMARY KEY (category, key) WITHOUT ROWID, WAL mode. * Writes are fire-and-forget via EventBroker(mt) PersistEvent. * Reads are async via five RequestBroker(mt) shapes (KvGet, KvExists, KvScan, KvCount, KvDelete). Reads return Result[T, PersistencyError]. * One storage thread per job; tenants isolated by BrokerContext. Public surface (waku/persistency/persistency.nim): Persistency.instance(rootDir) / Persistency.instance() / Persistency.reset() p.openJob(id) / p.closeJob(id) / p.dropJob(id) / p.close() p.job(id) / p[id] / p.hasJob(id) Writes (Job form & string-id form, fire-and-forget): persist / persistPut / persistDelete / persistEncoded Reads (Job form & string-id form, async Result): get / exists / scan / scanPrefix / count / deleteAcked Key & payload encoding (keys.nim, payload.nim): * encodePart family + variadic key(...) / payload(...) macros + single-value toKey / toPayload. * Primitives: string and openArray[byte] are 2-byte BE length + bytes; int{8..64} are sign-flipped 8-byte BE; uint{16..64} are 8-byte BE; bool/byte/char are 1 byte; enums are int64(ord(v)). * Generic encodePart[T: tuple | object] recurses through fields() so any composite Nim type is encodable without ceremony. * Stable across Nim/C compiler upgrades: no sizeof, no memcpy, no cast on pointers, no host-endianness dependency. * `rawKey(bytes)` + `persistPut(..., openArray[byte])` let callers bypass the built-in encoder with their own format (CBOR, protobuf...). Lifecycle: * Persistency.new is private; Persistency.instance is the only public constructor. Same rootDir is idempotent; conflicting rootDir is peInvalidArgument. Persistency.reset for test/restart paths. * openJob opens-or-creates the per-job SQLite file; an existing file is reused with its data preserved. * Teardown integration: Persistency.instance registers a Teardown MultiRequestBroker provider that closes all jobs and clears the singleton slot when Waku.stop() issues Teardown.request. Internal layering: types.nim pure value types (Key, KeyRange, KvRow, TxOp, PersistencyError) keys.nim encodePart primitives + key(...) macro payload.nim toPayload + payload(...) macro schema.nim CREATE TABLE + connection pragmas + user_version backend_sqlite.nim KvBackend, applyOps (single source of write SQL), getOne/existsOne/deleteOne, scanRange (asc/desc, half-open ranges, open-ended stop), countRange backend_comm.nim EventBroker(mt) PersistEvent + 5 RequestBroker(mt) declarations; encodeErr/decodeErr boundary helpers backend_thread.nim startStorageThread / stopStorageThread (shared allocShared0 arg, cstring dbPath, atomic ready/shutdown flags); per-thread provider registration persistency.nim Persistency + Job types, singleton state, public facade ../requests/lifecycle_requests.nim Teardown MultiRequestBroker Tests (69 cases, all passing): test_keys.nim sort-order invariants (length-prefix strings, sign-flipped ints, composite tuples, prefix range) test_backend.nim round-trip / replace / delete-return-value / batched atomicity / asc-desc-half-open-open- ended scans / category isolation / batch txDelete test_lifecycle.nim open-or-create rootDir / non-dir collision / reopen across sessions / idempotent openJob / two-tenant parallel isolation / closeJob joins worker / dropJob removes file / acked delete test_facade.nim put-then-get / atomic batch / scanPrefix asc/desc / deleteAcked hit-miss / fire-and-forget delete / two-tenant facade isolation test_encoding.nim tuple/named-tuple/object keys, embedded Key, enum encoding, field-major composite sort, payload struct encoding, end-to-end struct round-trip through SQLite test_string_lookup.nim peJobNotFound semantics / hasJob / subscript / persistPut+get via id / reads short-circuit / writes drop+warn / persistEncoded via id / scan parity Job-ref vs id test_singleton.nim idempotent same-rootDir / different-rootDir rejection / no-arg instance lifecycle / reset retargets / reset idempotence / Teardown.request end-to-end Prerequisite delivered in the same series: replace the in-tree broker implementation with the external nim-brokers package; update all broker call-sites (waku_filter_v2, waku_relay, waku_rln_relay, delivery_service, peer_manager, requests/*, factory/*, api tests, etc.) to the new package API; chat2 made to compile again. Note: SDS adapter (Phase 5 of the design) is deferred -- nim-sds is still developed side-by-side and the persistency layer is intentionally SDS-agnostic. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * persistency: pin nim-brokers by URL+commit (workaround for stale registry) The bare `brokers >= 2.0.1` form cannot resolve on machines where the local nimble SAT solver enumerates only the registry-recorded 0.1.0 for brokers. The nim-lang/packages entry for `brokers` carries no per-tag metadata (only the URL), so until that registry entry is refreshed the SAT solver clamps the available-versions list to 0.1.0 and rejects the >= 2.0.1 constraint -- even though pkgs2 and pkgcache both have v2.0.1 cloned locally. Pinning by URL+commit bypasses the registry path entirely. Inline comment in waku.nimble documents the situation and the path back to the bare form once nim-lang/packages is updated. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * persistency: nph format pass Run `nph` on all 57 Nim files touched by this PR. Pure formatting: 17 files re-styled, no semantic change. Suite still 69/69. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Fix build, add local-storage-path config, lazy init of Persistency from Waku start * fix: fix nix deps * fixes for nix build, regenerate deps * reverting accidental dependency changes * Fixing deps * Apply suggestions from code review Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com> * persistency tests: migrate to suite / asyncTest / await Match the in-tree test convention (procSuite -> suite, sync test + waitFor -> asyncTest + await): - procSuite "X": -> suite "X": - For tests doing async work: test -> asyncTest, waitFor -> await. - Poll helpers (proc waitFor(t: Job, ...) in test_lifecycle.nim, proc waitUntilExists(...) in test_facade.nim and test_string_lookup.nim) -> Future[bool] {.async.}, internal `waitFor X` -> `await X`, internal `sleep(N)` -> `await sleepAsync(chronos.milliseconds(N))`. - Renamed test_lifecycle.nim's helper proc from `waitFor(t: Job, ...)` -> `pollExists(t: Job, ...)`; the previous name shadowed chronos.waitFor in the chronos macro expansion. - `chronos.milliseconds(N)` explicitly qualified because `std/times` also exports `milliseconds` (returning TimeInterval, not Duration). - `check await x` -> `let okN = await x; check okN` to dodge chronos's "yield in expr not lowered" with await-as-macro-argument. - `(await x).foo()` -> `let awN = await x; ... awN.foo() ...` for the same reason. waku/persistency/persistency.nim: nph also pulled the proc signatures across multiple lines; restored explicit `Future[void] {.async.}` return types after the colon (an intermediate nph pass had elided them). Suite: 71 / 71 OK against the new async write surface. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * use idiomatic valueOr instead of ifs * Reworked persistency shutdown, remove not necessary teardown mechanism * Use const for DefaultStoragePath * format to follow coding guidelines - no use of result and explicit returns - no functional change --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2026-05-16 00:09:07 +02:00
import waku, waku/[waku_node, waku_core, waku_relay/protocol]
Chore: adapt cli args for delivery api (#3744) * LogosDeliveryAPI: NodeConfig -> WakluNodeConf + mode selector and logos.dev preset * Adjustment made on test, logos.dev preset * change default agentString from nwaku to logos-delivery * Add p2pReliability switch to presets and make it default to true. * Borrow entryNode idea from NodeConfig to WakuNodeConf to easy shortcut among diffrent bootstrap node list all needs different formats * Fix rateLimit assignment for builder * Remove Core mode default as we already have a defaul, user must override * Removed obsolate API createNode with NodeConfig - tests are refactored for WakuNodeConf usage * Fix failing test due to twn-clusterid(1) default has overwrite for maxMessagSize. Fix readme.
2026-03-03 19:17:54 +01:00
import waku/factory/waku_conf
import tools/confutils/cli_args
feat: waku api send (#3669) * Introduce api/send Added events and requests for support. Reworked delivery_monitor into a featured devlivery_service, that - supports relay publish and lightpush depending on configuration but with fallback options - if available and configured it utilizes store api to confirm message delivery - emits message delivery events accordingly prepare for use in api_example * Fix edge mode config and test added * Fix some import issues, start and stop waku shall not throw exception but return with result properly * Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover * add api_example app to examples2 * Adapt after merge from master * Adapt code for using broker context * Fix brokerCtx settings for all usedbrokers, cover locked node init * Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api * More test added * Fix multi propagate event emit, fix fail send test case * Fix rebase * Fix PushMessageHandlers in tests * adapt libwaku to api changes * Fix relay test by adapting publish return error in case NoPeersToPublish * Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos * Fix rln relay broker, missed brokerCtx * Fix rest relay test failed, due to publish will fail if no peer avail * ignore anvil test state file * Make terst_wakunode_rln_relay broker context aware to fix * Fix waku rln tests by having them broker context aware * fix typo in test_app.nim
2026-01-30 01:06:00 +01:00
type SendEventOutcome {.pure.} = enum
Sent
Propagated
Error
type SendEventListenerManager = ref object
brokerCtx: BrokerContext
sentListener: MessageSentEventListener
errorListener: MessageErrorEventListener
propagatedListener: MessagePropagatedEventListener
sentFuture: Future[void]
errorFuture: Future[void]
propagatedFuture: Future[void]
sentCount: int
errorCount: int
propagatedCount: int
sentRequestIds: seq[RequestId]
errorRequestIds: seq[RequestId]
propagatedRequestIds: seq[RequestId]
proc newSendEventListenerManager(brokerCtx: BrokerContext): SendEventListenerManager =
let manager = SendEventListenerManager(brokerCtx: brokerCtx)
manager.sentFuture = newFuture[void]("sentEvent")
manager.errorFuture = newFuture[void]("errorEvent")
manager.propagatedFuture = newFuture[void]("propagatedEvent")
manager.sentListener = MessageSentEvent.listen(
brokerCtx,
proc(event: MessageSentEvent) {.async: (raises: []).} =
inc manager.sentCount
manager.sentRequestIds.add(event.requestId)
echo "SENT EVENT TRIGGERED (#",
manager.sentCount, "): requestId=", event.requestId
if not manager.sentFuture.finished():
manager.sentFuture.complete()
,
).valueOr:
raiseAssert error
manager.errorListener = MessageErrorEvent.listen(
brokerCtx,
proc(event: MessageErrorEvent) {.async: (raises: []).} =
inc manager.errorCount
manager.errorRequestIds.add(event.requestId)
echo "ERROR EVENT TRIGGERED (#", manager.errorCount, "): ", event.error
if not manager.errorFuture.finished():
manager.errorFuture.fail(
newException(CatchableError, "Error event triggered: " & event.error)
)
,
).valueOr:
raiseAssert error
manager.propagatedListener = MessagePropagatedEvent.listen(
brokerCtx,
proc(event: MessagePropagatedEvent) {.async: (raises: []).} =
inc manager.propagatedCount
manager.propagatedRequestIds.add(event.requestId)
echo "PROPAGATED EVENT TRIGGERED (#",
manager.propagatedCount, "): requestId=", event.requestId
if not manager.propagatedFuture.finished():
manager.propagatedFuture.complete()
,
).valueOr:
raiseAssert error
return manager
feat: persistency (#3880) * persistency: per-job SQLite-backed storage layer (singleton, brokered) Adds a backend-neutral CRUD library at waku/persistency/, plus the nim-brokers dependency swap that enables it. Architecture (ports-and-adapters): * Persistency: process-wide singleton, one root directory. * Job: one tenant, one DB file, one worker thread, one BrokerContext. * Backend: SQLite via waku/common/databases/db_sqlite. Uniform schema kv(category BLOB, key BLOB, payload BLOB) PRIMARY KEY (category, key) WITHOUT ROWID, WAL mode. * Writes are fire-and-forget via EventBroker(mt) PersistEvent. * Reads are async via five RequestBroker(mt) shapes (KvGet, KvExists, KvScan, KvCount, KvDelete). Reads return Result[T, PersistencyError]. * One storage thread per job; tenants isolated by BrokerContext. Public surface (waku/persistency/persistency.nim): Persistency.instance(rootDir) / Persistency.instance() / Persistency.reset() p.openJob(id) / p.closeJob(id) / p.dropJob(id) / p.close() p.job(id) / p[id] / p.hasJob(id) Writes (Job form & string-id form, fire-and-forget): persist / persistPut / persistDelete / persistEncoded Reads (Job form & string-id form, async Result): get / exists / scan / scanPrefix / count / deleteAcked Key & payload encoding (keys.nim, payload.nim): * encodePart family + variadic key(...) / payload(...) macros + single-value toKey / toPayload. * Primitives: string and openArray[byte] are 2-byte BE length + bytes; int{8..64} are sign-flipped 8-byte BE; uint{16..64} are 8-byte BE; bool/byte/char are 1 byte; enums are int64(ord(v)). * Generic encodePart[T: tuple | object] recurses through fields() so any composite Nim type is encodable without ceremony. * Stable across Nim/C compiler upgrades: no sizeof, no memcpy, no cast on pointers, no host-endianness dependency. * `rawKey(bytes)` + `persistPut(..., openArray[byte])` let callers bypass the built-in encoder with their own format (CBOR, protobuf...). Lifecycle: * Persistency.new is private; Persistency.instance is the only public constructor. Same rootDir is idempotent; conflicting rootDir is peInvalidArgument. Persistency.reset for test/restart paths. * openJob opens-or-creates the per-job SQLite file; an existing file is reused with its data preserved. * Teardown integration: Persistency.instance registers a Teardown MultiRequestBroker provider that closes all jobs and clears the singleton slot when Waku.stop() issues Teardown.request. Internal layering: types.nim pure value types (Key, KeyRange, KvRow, TxOp, PersistencyError) keys.nim encodePart primitives + key(...) macro payload.nim toPayload + payload(...) macro schema.nim CREATE TABLE + connection pragmas + user_version backend_sqlite.nim KvBackend, applyOps (single source of write SQL), getOne/existsOne/deleteOne, scanRange (asc/desc, half-open ranges, open-ended stop), countRange backend_comm.nim EventBroker(mt) PersistEvent + 5 RequestBroker(mt) declarations; encodeErr/decodeErr boundary helpers backend_thread.nim startStorageThread / stopStorageThread (shared allocShared0 arg, cstring dbPath, atomic ready/shutdown flags); per-thread provider registration persistency.nim Persistency + Job types, singleton state, public facade ../requests/lifecycle_requests.nim Teardown MultiRequestBroker Tests (69 cases, all passing): test_keys.nim sort-order invariants (length-prefix strings, sign-flipped ints, composite tuples, prefix range) test_backend.nim round-trip / replace / delete-return-value / batched atomicity / asc-desc-half-open-open- ended scans / category isolation / batch txDelete test_lifecycle.nim open-or-create rootDir / non-dir collision / reopen across sessions / idempotent openJob / two-tenant parallel isolation / closeJob joins worker / dropJob removes file / acked delete test_facade.nim put-then-get / atomic batch / scanPrefix asc/desc / deleteAcked hit-miss / fire-and-forget delete / two-tenant facade isolation test_encoding.nim tuple/named-tuple/object keys, embedded Key, enum encoding, field-major composite sort, payload struct encoding, end-to-end struct round-trip through SQLite test_string_lookup.nim peJobNotFound semantics / hasJob / subscript / persistPut+get via id / reads short-circuit / writes drop+warn / persistEncoded via id / scan parity Job-ref vs id test_singleton.nim idempotent same-rootDir / different-rootDir rejection / no-arg instance lifecycle / reset retargets / reset idempotence / Teardown.request end-to-end Prerequisite delivered in the same series: replace the in-tree broker implementation with the external nim-brokers package; update all broker call-sites (waku_filter_v2, waku_relay, waku_rln_relay, delivery_service, peer_manager, requests/*, factory/*, api tests, etc.) to the new package API; chat2 made to compile again. Note: SDS adapter (Phase 5 of the design) is deferred -- nim-sds is still developed side-by-side and the persistency layer is intentionally SDS-agnostic. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * persistency: pin nim-brokers by URL+commit (workaround for stale registry) The bare `brokers >= 2.0.1` form cannot resolve on machines where the local nimble SAT solver enumerates only the registry-recorded 0.1.0 for brokers. The nim-lang/packages entry for `brokers` carries no per-tag metadata (only the URL), so until that registry entry is refreshed the SAT solver clamps the available-versions list to 0.1.0 and rejects the >= 2.0.1 constraint -- even though pkgs2 and pkgcache both have v2.0.1 cloned locally. Pinning by URL+commit bypasses the registry path entirely. Inline comment in waku.nimble documents the situation and the path back to the bare form once nim-lang/packages is updated. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * persistency: nph format pass Run `nph` on all 57 Nim files touched by this PR. Pure formatting: 17 files re-styled, no semantic change. Suite still 69/69. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Fix build, add local-storage-path config, lazy init of Persistency from Waku start * fix: fix nix deps * fixes for nix build, regenerate deps * reverting accidental dependency changes * Fixing deps * Apply suggestions from code review Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com> * persistency tests: migrate to suite / asyncTest / await Match the in-tree test convention (procSuite -> suite, sync test + waitFor -> asyncTest + await): - procSuite "X": -> suite "X": - For tests doing async work: test -> asyncTest, waitFor -> await. - Poll helpers (proc waitFor(t: Job, ...) in test_lifecycle.nim, proc waitUntilExists(...) in test_facade.nim and test_string_lookup.nim) -> Future[bool] {.async.}, internal `waitFor X` -> `await X`, internal `sleep(N)` -> `await sleepAsync(chronos.milliseconds(N))`. - Renamed test_lifecycle.nim's helper proc from `waitFor(t: Job, ...)` -> `pollExists(t: Job, ...)`; the previous name shadowed chronos.waitFor in the chronos macro expansion. - `chronos.milliseconds(N)` explicitly qualified because `std/times` also exports `milliseconds` (returning TimeInterval, not Duration). - `check await x` -> `let okN = await x; check okN` to dodge chronos's "yield in expr not lowered" with await-as-macro-argument. - `(await x).foo()` -> `let awN = await x; ... awN.foo() ...` for the same reason. waku/persistency/persistency.nim: nph also pulled the proc signatures across multiple lines; restored explicit `Future[void] {.async.}` return types after the colon (an intermediate nph pass had elided them). Suite: 71 / 71 OK against the new async write surface. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * use idiomatic valueOr instead of ifs * Reworked persistency shutdown, remove not necessary teardown mechanism * Use const for DefaultStoragePath * format to follow coding guidelines - no use of result and explicit returns - no functional change --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2026-05-16 00:09:07 +02:00
proc teardown(manager: SendEventListenerManager) {.async.} =
await MessageSentEvent.dropListener(manager.brokerCtx, manager.sentListener)
await MessageErrorEvent.dropListener(manager.brokerCtx, manager.errorListener)
await MessagePropagatedEvent.dropListener(
manager.brokerCtx, manager.propagatedListener
)
feat: waku api send (#3669) * Introduce api/send Added events and requests for support. Reworked delivery_monitor into a featured devlivery_service, that - supports relay publish and lightpush depending on configuration but with fallback options - if available and configured it utilizes store api to confirm message delivery - emits message delivery events accordingly prepare for use in api_example * Fix edge mode config and test added * Fix some import issues, start and stop waku shall not throw exception but return with result properly * Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover * add api_example app to examples2 * Adapt after merge from master * Adapt code for using broker context * Fix brokerCtx settings for all usedbrokers, cover locked node init * Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api * More test added * Fix multi propagate event emit, fix fail send test case * Fix rebase * Fix PushMessageHandlers in tests * adapt libwaku to api changes * Fix relay test by adapting publish return error in case NoPeersToPublish * Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos * Fix rln relay broker, missed brokerCtx * Fix rest relay test failed, due to publish will fail if no peer avail * ignore anvil test state file * Make terst_wakunode_rln_relay broker context aware to fix * Fix waku rln tests by having them broker context aware * fix typo in test_app.nim
2026-01-30 01:06:00 +01:00
proc waitForEvents(
manager: SendEventListenerManager, timeout: Duration
): Future[bool] {.async.} =
return await allFutures(
manager.sentFuture, manager.propagatedFuture, manager.errorFuture
)
nim nph 0.7.0 formatting (#3759)
2026-03-17 10:15:35 -03:00
.withTimeout(timeout)
feat: waku api send (#3669) * Introduce api/send Added events and requests for support. Reworked delivery_monitor into a featured devlivery_service, that - supports relay publish and lightpush depending on configuration but with fallback options - if available and configured it utilizes store api to confirm message delivery - emits message delivery events accordingly prepare for use in api_example * Fix edge mode config and test added * Fix some import issues, start and stop waku shall not throw exception but return with result properly * Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover * add api_example app to examples2 * Adapt after merge from master * Adapt code for using broker context * Fix brokerCtx settings for all usedbrokers, cover locked node init * Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api * More test added * Fix multi propagate event emit, fix fail send test case * Fix rebase * Fix PushMessageHandlers in tests * adapt libwaku to api changes * Fix relay test by adapting publish return error in case NoPeersToPublish * Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos * Fix rln relay broker, missed brokerCtx * Fix rest relay test failed, due to publish will fail if no peer avail * ignore anvil test state file * Make terst_wakunode_rln_relay broker context aware to fix * Fix waku rln tests by having them broker context aware * fix typo in test_app.nim
2026-01-30 01:06:00 +01:00
proc outcomes(manager: SendEventListenerManager): set[SendEventOutcome] =
if manager.sentFuture.completed():
result.incl(SendEventOutcome.Sent)
if manager.propagatedFuture.completed():
result.incl(SendEventOutcome.Propagated)
if manager.errorFuture.failed():
result.incl(SendEventOutcome.Error)
proc validate(manager: SendEventListenerManager, expected: set[SendEventOutcome]) =
echo "EVENT COUNTS: sent=",
manager.sentCount, ", propagated=", manager.propagatedCount, ", error=",
manager.errorCount
check manager.outcomes() == expected
proc validate(
manager: SendEventListenerManager,
expected: set[SendEventOutcome],
expectedRequestId: RequestId,
) =
manager.validate(expected)
for requestId in manager.sentRequestIds:
check requestId == expectedRequestId
for requestId in manager.propagatedRequestIds:
check requestId == expectedRequestId
for requestId in manager.errorRequestIds:
check requestId == expectedRequestId
Chore: adapt cli args for delivery api (#3744) * LogosDeliveryAPI: NodeConfig -> WakluNodeConf + mode selector and logos.dev preset * Adjustment made on test, logos.dev preset * change default agentString from nwaku to logos-delivery * Add p2pReliability switch to presets and make it default to true. * Borrow entryNode idea from NodeConfig to WakuNodeConf to easy shortcut among diffrent bootstrap node list all needs different formats * Fix rateLimit assignment for builder * Remove Core mode default as we already have a defaul, user must override * Removed obsolate API createNode with NodeConfig - tests are refactored for WakuNodeConf usage * Fix failing test due to twn-clusterid(1) default has overwrite for maxMessagSize. Fix readme.
2026-03-03 19:17:54 +01:00
proc createApiNodeConf(mode: cli_args.WakuMode = cli_args.WakuMode.Core): WakuNodeConf =
var conf = defaultWakuNodeConf().valueOr:
raiseAssert error
conf.mode = mode
conf.listenAddress = parseIpAddress("0.0.0.0")
conf.tcpPort = Port(0)
conf.discv5UdpPort = Port(0)
conf.clusterId = 3'u16
conf.numShardsInNetwork = 1
conf.reliabilityEnabled = true
conf.rest = false
result = conf
feat: waku api send (#3669) * Introduce api/send Added events and requests for support. Reworked delivery_monitor into a featured devlivery_service, that - supports relay publish and lightpush depending on configuration but with fallback options - if available and configured it utilizes store api to confirm message delivery - emits message delivery events accordingly prepare for use in api_example * Fix edge mode config and test added * Fix some import issues, start and stop waku shall not throw exception but return with result properly * Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover * add api_example app to examples2 * Adapt after merge from master * Adapt code for using broker context * Fix brokerCtx settings for all usedbrokers, cover locked node init * Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api * More test added * Fix multi propagate event emit, fix fail send test case * Fix rebase * Fix PushMessageHandlers in tests * adapt libwaku to api changes * Fix relay test by adapting publish return error in case NoPeersToPublish * Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos * Fix rln relay broker, missed brokerCtx * Fix rest relay test failed, due to publish will fail if no peer avail * ignore anvil test state file * Make terst_wakunode_rln_relay broker context aware to fix * Fix waku rln tests by having them broker context aware * fix typo in test_app.nim
2026-01-30 01:06:00 +01:00
suite "Waku API - Send":
var
relayNode1 {.threadvar.}: WakuNode
relayNode1PeerInfo {.threadvar.}: RemotePeerInfo
relayNode1PeerId {.threadvar.}: PeerId
relayNode2 {.threadvar.}: WakuNode
relayNode2PeerInfo {.threadvar.}: RemotePeerInfo
relayNode2PeerId {.threadvar.}: PeerId
lightpushNode {.threadvar.}: WakuNode
lightpushNodePeerInfo {.threadvar.}: RemotePeerInfo
lightpushNodePeerId {.threadvar.}: PeerId
storeNode {.threadvar.}: WakuNode
storeNodePeerInfo {.threadvar.}: RemotePeerInfo
storeNodePeerId {.threadvar.}: PeerId
asyncSetup:
lockNewGlobalBrokerContext:
relayNode1 =
newTestWakuNode(generateSecp256k1Key(), parseIpAddress("0.0.0.0"), Port(0))
Chore: adapt cli args for delivery api (#3744) * LogosDeliveryAPI: NodeConfig -> WakluNodeConf + mode selector and logos.dev preset * Adjustment made on test, logos.dev preset * change default agentString from nwaku to logos-delivery * Add p2pReliability switch to presets and make it default to true. * Borrow entryNode idea from NodeConfig to WakuNodeConf to easy shortcut among diffrent bootstrap node list all needs different formats * Fix rateLimit assignment for builder * Remove Core mode default as we already have a defaul, user must override * Removed obsolate API createNode with NodeConfig - tests are refactored for WakuNodeConf usage * Fix failing test due to twn-clusterid(1) default has overwrite for maxMessagSize. Fix readme.
2026-03-03 19:17:54 +01:00
relayNode1.mountMetadata(3, @[0'u16]).isOkOr:
feat: waku api send (#3669) * Introduce api/send Added events and requests for support. Reworked delivery_monitor into a featured devlivery_service, that - supports relay publish and lightpush depending on configuration but with fallback options - if available and configured it utilizes store api to confirm message delivery - emits message delivery events accordingly prepare for use in api_example * Fix edge mode config and test added * Fix some import issues, start and stop waku shall not throw exception but return with result properly * Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover * add api_example app to examples2 * Adapt after merge from master * Adapt code for using broker context * Fix brokerCtx settings for all usedbrokers, cover locked node init * Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api * More test added * Fix multi propagate event emit, fix fail send test case * Fix rebase * Fix PushMessageHandlers in tests * adapt libwaku to api changes * Fix relay test by adapting publish return error in case NoPeersToPublish * Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos * Fix rln relay broker, missed brokerCtx * Fix rest relay test failed, due to publish will fail if no peer avail * ignore anvil test state file * Make terst_wakunode_rln_relay broker context aware to fix * Fix waku rln tests by having them broker context aware * fix typo in test_app.nim
2026-01-30 01:06:00 +01:00
raiseAssert "Failed to mount metadata: " & error
(await relayNode1.mountRelay()).isOkOr:
raiseAssert "Failed to mount relay"
await relayNode1.mountLibp2pPing()
await relayNode1.start()
lockNewGlobalBrokerContext:
relayNode2 =
newTestWakuNode(generateSecp256k1Key(), parseIpAddress("0.0.0.0"), Port(0))
Chore: adapt cli args for delivery api (#3744) * LogosDeliveryAPI: NodeConfig -> WakluNodeConf + mode selector and logos.dev preset * Adjustment made on test, logos.dev preset * change default agentString from nwaku to logos-delivery * Add p2pReliability switch to presets and make it default to true. * Borrow entryNode idea from NodeConfig to WakuNodeConf to easy shortcut among diffrent bootstrap node list all needs different formats * Fix rateLimit assignment for builder * Remove Core mode default as we already have a defaul, user must override * Removed obsolate API createNode with NodeConfig - tests are refactored for WakuNodeConf usage * Fix failing test due to twn-clusterid(1) default has overwrite for maxMessagSize. Fix readme.
2026-03-03 19:17:54 +01:00
relayNode2.mountMetadata(3, @[0'u16]).isOkOr:
feat: waku api send (#3669) * Introduce api/send Added events and requests for support. Reworked delivery_monitor into a featured devlivery_service, that - supports relay publish and lightpush depending on configuration but with fallback options - if available and configured it utilizes store api to confirm message delivery - emits message delivery events accordingly prepare for use in api_example * Fix edge mode config and test added * Fix some import issues, start and stop waku shall not throw exception but return with result properly * Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover * add api_example app to examples2 * Adapt after merge from master * Adapt code for using broker context * Fix brokerCtx settings for all usedbrokers, cover locked node init * Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api * More test added * Fix multi propagate event emit, fix fail send test case * Fix rebase * Fix PushMessageHandlers in tests * adapt libwaku to api changes * Fix relay test by adapting publish return error in case NoPeersToPublish * Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos * Fix rln relay broker, missed brokerCtx * Fix rest relay test failed, due to publish will fail if no peer avail * ignore anvil test state file * Make terst_wakunode_rln_relay broker context aware to fix * Fix waku rln tests by having them broker context aware * fix typo in test_app.nim
2026-01-30 01:06:00 +01:00
raiseAssert "Failed to mount metadata: " & error
(await relayNode2.mountRelay()).isOkOr:
raiseAssert "Failed to mount relay"
await relayNode2.mountLibp2pPing()
await relayNode2.start()
lockNewGlobalBrokerContext:
lightpushNode =
newTestWakuNode(generateSecp256k1Key(), parseIpAddress("0.0.0.0"), Port(0))
Chore: adapt cli args for delivery api (#3744) * LogosDeliveryAPI: NodeConfig -> WakluNodeConf + mode selector and logos.dev preset * Adjustment made on test, logos.dev preset * change default agentString from nwaku to logos-delivery * Add p2pReliability switch to presets and make it default to true. * Borrow entryNode idea from NodeConfig to WakuNodeConf to easy shortcut among diffrent bootstrap node list all needs different formats * Fix rateLimit assignment for builder * Remove Core mode default as we already have a defaul, user must override * Removed obsolate API createNode with NodeConfig - tests are refactored for WakuNodeConf usage * Fix failing test due to twn-clusterid(1) default has overwrite for maxMessagSize. Fix readme.
2026-03-03 19:17:54 +01:00
lightpushNode.mountMetadata(3, @[0'u16]).isOkOr:
feat: waku api send (#3669) * Introduce api/send Added events and requests for support. Reworked delivery_monitor into a featured devlivery_service, that - supports relay publish and lightpush depending on configuration but with fallback options - if available and configured it utilizes store api to confirm message delivery - emits message delivery events accordingly prepare for use in api_example * Fix edge mode config and test added * Fix some import issues, start and stop waku shall not throw exception but return with result properly * Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover * add api_example app to examples2 * Adapt after merge from master * Adapt code for using broker context * Fix brokerCtx settings for all usedbrokers, cover locked node init * Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api * More test added * Fix multi propagate event emit, fix fail send test case * Fix rebase * Fix PushMessageHandlers in tests * adapt libwaku to api changes * Fix relay test by adapting publish return error in case NoPeersToPublish * Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos * Fix rln relay broker, missed brokerCtx * Fix rest relay test failed, due to publish will fail if no peer avail * ignore anvil test state file * Make terst_wakunode_rln_relay broker context aware to fix * Fix waku rln tests by having them broker context aware * fix typo in test_app.nim
2026-01-30 01:06:00 +01:00
raiseAssert "Failed to mount metadata: " & error
(await lightpushNode.mountRelay()).isOkOr:
raiseAssert "Failed to mount relay"
(await lightpushNode.mountLightPush()).isOkOr:
raiseAssert "Failed to mount lightpush"
await lightpushNode.mountLibp2pPing()
await lightpushNode.start()
lockNewGlobalBrokerContext:
storeNode =
newTestWakuNode(generateSecp256k1Key(), parseIpAddress("0.0.0.0"), Port(0))
Chore: adapt cli args for delivery api (#3744) * LogosDeliveryAPI: NodeConfig -> WakluNodeConf + mode selector and logos.dev preset * Adjustment made on test, logos.dev preset * change default agentString from nwaku to logos-delivery * Add p2pReliability switch to presets and make it default to true. * Borrow entryNode idea from NodeConfig to WakuNodeConf to easy shortcut among diffrent bootstrap node list all needs different formats * Fix rateLimit assignment for builder * Remove Core mode default as we already have a defaul, user must override * Removed obsolate API createNode with NodeConfig - tests are refactored for WakuNodeConf usage * Fix failing test due to twn-clusterid(1) default has overwrite for maxMessagSize. Fix readme.
2026-03-03 19:17:54 +01:00
storeNode.mountMetadata(3, @[0'u16]).isOkOr:
feat: waku api send (#3669) * Introduce api/send Added events and requests for support. Reworked delivery_monitor into a featured devlivery_service, that - supports relay publish and lightpush depending on configuration but with fallback options - if available and configured it utilizes store api to confirm message delivery - emits message delivery events accordingly prepare for use in api_example * Fix edge mode config and test added * Fix some import issues, start and stop waku shall not throw exception but return with result properly * Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover * add api_example app to examples2 * Adapt after merge from master * Adapt code for using broker context * Fix brokerCtx settings for all usedbrokers, cover locked node init * Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api * More test added * Fix multi propagate event emit, fix fail send test case * Fix rebase * Fix PushMessageHandlers in tests * adapt libwaku to api changes * Fix relay test by adapting publish return error in case NoPeersToPublish * Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos * Fix rln relay broker, missed brokerCtx * Fix rest relay test failed, due to publish will fail if no peer avail * ignore anvil test state file * Make terst_wakunode_rln_relay broker context aware to fix * Fix waku rln tests by having them broker context aware * fix typo in test_app.nim
2026-01-30 01:06:00 +01:00
raiseAssert "Failed to mount metadata: " & error
(await storeNode.mountRelay()).isOkOr:
raiseAssert "Failed to mount relay"
# Mount archive so store can persist messages
let archiveDriver = newSqliteArchiveDriver()
storeNode.mountArchive(archiveDriver).isOkOr:
raiseAssert "Failed to mount archive: " & error
await storeNode.mountStore()
await storeNode.mountLibp2pPing()
await storeNode.start()
relayNode1PeerInfo = relayNode1.peerInfo.toRemotePeerInfo()
relayNode1PeerId = relayNode1.peerInfo.peerId
relayNode2PeerInfo = relayNode2.peerInfo.toRemotePeerInfo()
relayNode2PeerId = relayNode2.peerInfo.peerId
lightpushNodePeerInfo = lightpushNode.peerInfo.toRemotePeerInfo()
lightpushNodePeerId = lightpushNode.peerInfo.peerId
storeNodePeerInfo = storeNode.peerInfo.toRemotePeerInfo()
storeNodePeerId = storeNode.peerInfo.peerId
# Subscribe all relay nodes to the default shard topic
Chore: adapt cli args for delivery api (#3744) * LogosDeliveryAPI: NodeConfig -> WakluNodeConf + mode selector and logos.dev preset * Adjustment made on test, logos.dev preset * change default agentString from nwaku to logos-delivery * Add p2pReliability switch to presets and make it default to true. * Borrow entryNode idea from NodeConfig to WakuNodeConf to easy shortcut among diffrent bootstrap node list all needs different formats * Fix rateLimit assignment for builder * Remove Core mode default as we already have a defaul, user must override * Removed obsolate API createNode with NodeConfig - tests are refactored for WakuNodeConf usage * Fix failing test due to twn-clusterid(1) default has overwrite for maxMessagSize. Fix readme.
2026-03-03 19:17:54 +01:00
const testPubsubTopic = PubsubTopic("/waku/2/rs/3/0")
feat: waku api send (#3669) * Introduce api/send Added events and requests for support. Reworked delivery_monitor into a featured devlivery_service, that - supports relay publish and lightpush depending on configuration but with fallback options - if available and configured it utilizes store api to confirm message delivery - emits message delivery events accordingly prepare for use in api_example * Fix edge mode config and test added * Fix some import issues, start and stop waku shall not throw exception but return with result properly * Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover * add api_example app to examples2 * Adapt after merge from master * Adapt code for using broker context * Fix brokerCtx settings for all usedbrokers, cover locked node init * Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api * More test added * Fix multi propagate event emit, fix fail send test case * Fix rebase * Fix PushMessageHandlers in tests * adapt libwaku to api changes * Fix relay test by adapting publish return error in case NoPeersToPublish * Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos * Fix rln relay broker, missed brokerCtx * Fix rest relay test failed, due to publish will fail if no peer avail * ignore anvil test state file * Make terst_wakunode_rln_relay broker context aware to fix * Fix waku rln tests by having them broker context aware * fix typo in test_app.nim
2026-01-30 01:06:00 +01:00
proc dummyHandler(
topic: PubsubTopic, msg: WakuMessage
): Future[void] {.async, gcsafe.} =
discard
relayNode1.subscribe((kind: PubsubSub, topic: testPubsubTopic), dummyHandler).isOkOr:
raiseAssert "Failed to subscribe relayNode1: " & error
relayNode2.subscribe((kind: PubsubSub, topic: testPubsubTopic), dummyHandler).isOkOr:
raiseAssert "Failed to subscribe relayNode2: " & error
lightpushNode.subscribe((kind: PubsubSub, topic: testPubsubTopic), dummyHandler).isOkOr:
raiseAssert "Failed to subscribe lightpushNode: " & error
storeNode.subscribe((kind: PubsubSub, topic: testPubsubTopic), dummyHandler).isOkOr:
raiseAssert "Failed to subscribe storeNode: " & error
# Subscribe all relay nodes to the default shard topic
await relayNode1.connectToNodes(@[relayNode2PeerInfo, storeNodePeerInfo])
await lightpushNode.connectToNodes(@[relayNode2PeerInfo])
asyncTeardown:
await allFutures(
relayNode1.stop(), relayNode2.stop(), lightpushNode.stop(), storeNode.stop()
)
asyncTest "Check API availability (unhealthy node)":
var node: Waku
lockNewGlobalBrokerContext:
node = (await createNode(createApiNodeConf())).valueOr:
raiseAssert error
(await startWaku(addr node)).isOkOr:
raiseAssert "Failed to start Waku node: " & error
# node is not connected !
let envelope = MessageEnvelope.init(
ContentTopic("/waku/2/default-content/proto"), "test payload"
)
let sendResult = await node.send(envelope)
feat: implement Waku API Health spec (#3689) * Fix protocol strength metric to consider connected peers only * Remove polling loop; event-driven node connection health updates * Remove 10s WakuRelay topic health polling loop; now event-driven * Change NodeHealthStatus to ConnectionStatus * Change new nodeState (rest API /health) field to connectionStatus * Add getSyncProtocolHealthInfo and getSyncNodeHealthReport * Add ConnectionStatusChangeEvent * Add RequestHealthReport * Refactor sync/async protocol health queries in the health monitor * Add EventRelayTopicHealthChange * Add EventWakuPeer emitted by PeerManager * Add Edge support for topics health requests and events * Rename "RelayTopic" -> "Topic" * Add RequestContentTopicsHealth sync request * Add EventContentTopicHealthChange * Rename RequestTopicsHealth -> RequestShardTopicsHealth * Remove health check gating from checkApiAvailability * Add basic health smoke tests * Other misc improvements, refactors, fixes Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com> Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2026-02-12 14:52:39 -03:00
# TODO: The API is not enforcing a health check before the send,
# so currently this test cannot successfully fail to send.
check sendResult.isOk()
feat: waku api send (#3669) * Introduce api/send Added events and requests for support. Reworked delivery_monitor into a featured devlivery_service, that - supports relay publish and lightpush depending on configuration but with fallback options - if available and configured it utilizes store api to confirm message delivery - emits message delivery events accordingly prepare for use in api_example * Fix edge mode config and test added * Fix some import issues, start and stop waku shall not throw exception but return with result properly * Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover * add api_example app to examples2 * Adapt after merge from master * Adapt code for using broker context * Fix brokerCtx settings for all usedbrokers, cover locked node init * Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api * More test added * Fix multi propagate event emit, fix fail send test case * Fix rebase * Fix PushMessageHandlers in tests * adapt libwaku to api changes * Fix relay test by adapting publish return error in case NoPeersToPublish * Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos * Fix rln relay broker, missed brokerCtx * Fix rest relay test failed, due to publish will fail if no peer avail * ignore anvil test state file * Make terst_wakunode_rln_relay broker context aware to fix * Fix waku rln tests by having them broker context aware * fix typo in test_app.nim
2026-01-30 01:06:00 +01:00
(await node.stop()).isOkOr:
raiseAssert "Failed to stop node: " & error
asyncTest "Send fully validated":
var node: Waku
lockNewGlobalBrokerContext:
node = (await createNode(createApiNodeConf())).valueOr:
raiseAssert error
(await startWaku(addr node)).isOkOr:
raiseAssert "Failed to start Waku node: " & error
await node.node.connectToNodes(
@[relayNode1PeerInfo, lightpushNodePeerInfo, storeNodePeerInfo]
)
let eventManager = newSendEventListenerManager(node.brokerCtx)
defer:
feat: persistency (#3880) * persistency: per-job SQLite-backed storage layer (singleton, brokered) Adds a backend-neutral CRUD library at waku/persistency/, plus the nim-brokers dependency swap that enables it. Architecture (ports-and-adapters): * Persistency: process-wide singleton, one root directory. * Job: one tenant, one DB file, one worker thread, one BrokerContext. * Backend: SQLite via waku/common/databases/db_sqlite. Uniform schema kv(category BLOB, key BLOB, payload BLOB) PRIMARY KEY (category, key) WITHOUT ROWID, WAL mode. * Writes are fire-and-forget via EventBroker(mt) PersistEvent. * Reads are async via five RequestBroker(mt) shapes (KvGet, KvExists, KvScan, KvCount, KvDelete). Reads return Result[T, PersistencyError]. * One storage thread per job; tenants isolated by BrokerContext. Public surface (waku/persistency/persistency.nim): Persistency.instance(rootDir) / Persistency.instance() / Persistency.reset() p.openJob(id) / p.closeJob(id) / p.dropJob(id) / p.close() p.job(id) / p[id] / p.hasJob(id) Writes (Job form & string-id form, fire-and-forget): persist / persistPut / persistDelete / persistEncoded Reads (Job form & string-id form, async Result): get / exists / scan / scanPrefix / count / deleteAcked Key & payload encoding (keys.nim, payload.nim): * encodePart family + variadic key(...) / payload(...) macros + single-value toKey / toPayload. * Primitives: string and openArray[byte] are 2-byte BE length + bytes; int{8..64} are sign-flipped 8-byte BE; uint{16..64} are 8-byte BE; bool/byte/char are 1 byte; enums are int64(ord(v)). * Generic encodePart[T: tuple | object] recurses through fields() so any composite Nim type is encodable without ceremony. * Stable across Nim/C compiler upgrades: no sizeof, no memcpy, no cast on pointers, no host-endianness dependency. * `rawKey(bytes)` + `persistPut(..., openArray[byte])` let callers bypass the built-in encoder with their own format (CBOR, protobuf...). Lifecycle: * Persistency.new is private; Persistency.instance is the only public constructor. Same rootDir is idempotent; conflicting rootDir is peInvalidArgument. Persistency.reset for test/restart paths. * openJob opens-or-creates the per-job SQLite file; an existing file is reused with its data preserved. * Teardown integration: Persistency.instance registers a Teardown MultiRequestBroker provider that closes all jobs and clears the singleton slot when Waku.stop() issues Teardown.request. Internal layering: types.nim pure value types (Key, KeyRange, KvRow, TxOp, PersistencyError) keys.nim encodePart primitives + key(...) macro payload.nim toPayload + payload(...) macro schema.nim CREATE TABLE + connection pragmas + user_version backend_sqlite.nim KvBackend, applyOps (single source of write SQL), getOne/existsOne/deleteOne, scanRange (asc/desc, half-open ranges, open-ended stop), countRange backend_comm.nim EventBroker(mt) PersistEvent + 5 RequestBroker(mt) declarations; encodeErr/decodeErr boundary helpers backend_thread.nim startStorageThread / stopStorageThread (shared allocShared0 arg, cstring dbPath, atomic ready/shutdown flags); per-thread provider registration persistency.nim Persistency + Job types, singleton state, public facade ../requests/lifecycle_requests.nim Teardown MultiRequestBroker Tests (69 cases, all passing): test_keys.nim sort-order invariants (length-prefix strings, sign-flipped ints, composite tuples, prefix range) test_backend.nim round-trip / replace / delete-return-value / batched atomicity / asc-desc-half-open-open- ended scans / category isolation / batch txDelete test_lifecycle.nim open-or-create rootDir / non-dir collision / reopen across sessions / idempotent openJob / two-tenant parallel isolation / closeJob joins worker / dropJob removes file / acked delete test_facade.nim put-then-get / atomic batch / scanPrefix asc/desc / deleteAcked hit-miss / fire-and-forget delete / two-tenant facade isolation test_encoding.nim tuple/named-tuple/object keys, embedded Key, enum encoding, field-major composite sort, payload struct encoding, end-to-end struct round-trip through SQLite test_string_lookup.nim peJobNotFound semantics / hasJob / subscript / persistPut+get via id / reads short-circuit / writes drop+warn / persistEncoded via id / scan parity Job-ref vs id test_singleton.nim idempotent same-rootDir / different-rootDir rejection / no-arg instance lifecycle / reset retargets / reset idempotence / Teardown.request end-to-end Prerequisite delivered in the same series: replace the in-tree broker implementation with the external nim-brokers package; update all broker call-sites (waku_filter_v2, waku_relay, waku_rln_relay, delivery_service, peer_manager, requests/*, factory/*, api tests, etc.) to the new package API; chat2 made to compile again. Note: SDS adapter (Phase 5 of the design) is deferred -- nim-sds is still developed side-by-side and the persistency layer is intentionally SDS-agnostic. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * persistency: pin nim-brokers by URL+commit (workaround for stale registry) The bare `brokers >= 2.0.1` form cannot resolve on machines where the local nimble SAT solver enumerates only the registry-recorded 0.1.0 for brokers. The nim-lang/packages entry for `brokers` carries no per-tag metadata (only the URL), so until that registry entry is refreshed the SAT solver clamps the available-versions list to 0.1.0 and rejects the >= 2.0.1 constraint -- even though pkgs2 and pkgcache both have v2.0.1 cloned locally. Pinning by URL+commit bypasses the registry path entirely. Inline comment in waku.nimble documents the situation and the path back to the bare form once nim-lang/packages is updated. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * persistency: nph format pass Run `nph` on all 57 Nim files touched by this PR. Pure formatting: 17 files re-styled, no semantic change. Suite still 69/69. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Fix build, add local-storage-path config, lazy init of Persistency from Waku start * fix: fix nix deps * fixes for nix build, regenerate deps * reverting accidental dependency changes * Fixing deps * Apply suggestions from code review Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com> * persistency tests: migrate to suite / asyncTest / await Match the in-tree test convention (procSuite -> suite, sync test + waitFor -> asyncTest + await): - procSuite "X": -> suite "X": - For tests doing async work: test -> asyncTest, waitFor -> await. - Poll helpers (proc waitFor(t: Job, ...) in test_lifecycle.nim, proc waitUntilExists(...) in test_facade.nim and test_string_lookup.nim) -> Future[bool] {.async.}, internal `waitFor X` -> `await X`, internal `sleep(N)` -> `await sleepAsync(chronos.milliseconds(N))`. - Renamed test_lifecycle.nim's helper proc from `waitFor(t: Job, ...)` -> `pollExists(t: Job, ...)`; the previous name shadowed chronos.waitFor in the chronos macro expansion. - `chronos.milliseconds(N)` explicitly qualified because `std/times` also exports `milliseconds` (returning TimeInterval, not Duration). - `check await x` -> `let okN = await x; check okN` to dodge chronos's "yield in expr not lowered" with await-as-macro-argument. - `(await x).foo()` -> `let awN = await x; ... awN.foo() ...` for the same reason. waku/persistency/persistency.nim: nph also pulled the proc signatures across multiple lines; restored explicit `Future[void] {.async.}` return types after the colon (an intermediate nph pass had elided them). Suite: 71 / 71 OK against the new async write surface. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * use idiomatic valueOr instead of ifs * Reworked persistency shutdown, remove not necessary teardown mechanism * Use const for DefaultStoragePath * format to follow coding guidelines - no use of result and explicit returns - no functional change --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2026-05-16 00:09:07 +02:00
await eventManager.teardown()
feat: waku api send (#3669) * Introduce api/send Added events and requests for support. Reworked delivery_monitor into a featured devlivery_service, that - supports relay publish and lightpush depending on configuration but with fallback options - if available and configured it utilizes store api to confirm message delivery - emits message delivery events accordingly prepare for use in api_example * Fix edge mode config and test added * Fix some import issues, start and stop waku shall not throw exception but return with result properly * Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover * add api_example app to examples2 * Adapt after merge from master * Adapt code for using broker context * Fix brokerCtx settings for all usedbrokers, cover locked node init * Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api * More test added * Fix multi propagate event emit, fix fail send test case * Fix rebase * Fix PushMessageHandlers in tests * adapt libwaku to api changes * Fix relay test by adapting publish return error in case NoPeersToPublish * Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos * Fix rln relay broker, missed brokerCtx * Fix rest relay test failed, due to publish will fail if no peer avail * ignore anvil test state file * Make terst_wakunode_rln_relay broker context aware to fix * Fix waku rln tests by having them broker context aware * fix typo in test_app.nim
2026-01-30 01:06:00 +01:00
let envelope = MessageEnvelope.init(
ContentTopic("/waku/2/default-content/proto"), "test payload"
)
let requestId = (await node.send(envelope)).valueOr:
raiseAssert error
# Wait for events with timeout
const eventTimeout = 10.seconds
discard await eventManager.waitForEvents(eventTimeout)
eventManager.validate(
{SendEventOutcome.Sent, SendEventOutcome.Propagated}, requestId
)
(await node.stop()).isOkOr:
raiseAssert "Failed to stop node: " & error
asyncTest "Send only propagates":
var node: Waku
lockNewGlobalBrokerContext:
node = (await createNode(createApiNodeConf())).valueOr:
raiseAssert error
(await startWaku(addr node)).isOkOr:
raiseAssert "Failed to start Waku node: " & error
await node.node.connectToNodes(@[relayNode1PeerInfo])
let eventManager = newSendEventListenerManager(node.brokerCtx)
defer:
feat: persistency (#3880) * persistency: per-job SQLite-backed storage layer (singleton, brokered) Adds a backend-neutral CRUD library at waku/persistency/, plus the nim-brokers dependency swap that enables it. Architecture (ports-and-adapters): * Persistency: process-wide singleton, one root directory. * Job: one tenant, one DB file, one worker thread, one BrokerContext. * Backend: SQLite via waku/common/databases/db_sqlite. Uniform schema kv(category BLOB, key BLOB, payload BLOB) PRIMARY KEY (category, key) WITHOUT ROWID, WAL mode. * Writes are fire-and-forget via EventBroker(mt) PersistEvent. * Reads are async via five RequestBroker(mt) shapes (KvGet, KvExists, KvScan, KvCount, KvDelete). Reads return Result[T, PersistencyError]. * One storage thread per job; tenants isolated by BrokerContext. Public surface (waku/persistency/persistency.nim): Persistency.instance(rootDir) / Persistency.instance() / Persistency.reset() p.openJob(id) / p.closeJob(id) / p.dropJob(id) / p.close() p.job(id) / p[id] / p.hasJob(id) Writes (Job form & string-id form, fire-and-forget): persist / persistPut / persistDelete / persistEncoded Reads (Job form & string-id form, async Result): get / exists / scan / scanPrefix / count / deleteAcked Key & payload encoding (keys.nim, payload.nim): * encodePart family + variadic key(...) / payload(...) macros + single-value toKey / toPayload. * Primitives: string and openArray[byte] are 2-byte BE length + bytes; int{8..64} are sign-flipped 8-byte BE; uint{16..64} are 8-byte BE; bool/byte/char are 1 byte; enums are int64(ord(v)). * Generic encodePart[T: tuple | object] recurses through fields() so any composite Nim type is encodable without ceremony. * Stable across Nim/C compiler upgrades: no sizeof, no memcpy, no cast on pointers, no host-endianness dependency. * `rawKey(bytes)` + `persistPut(..., openArray[byte])` let callers bypass the built-in encoder with their own format (CBOR, protobuf...). Lifecycle: * Persistency.new is private; Persistency.instance is the only public constructor. Same rootDir is idempotent; conflicting rootDir is peInvalidArgument. Persistency.reset for test/restart paths. * openJob opens-or-creates the per-job SQLite file; an existing file is reused with its data preserved. * Teardown integration: Persistency.instance registers a Teardown MultiRequestBroker provider that closes all jobs and clears the singleton slot when Waku.stop() issues Teardown.request. Internal layering: types.nim pure value types (Key, KeyRange, KvRow, TxOp, PersistencyError) keys.nim encodePart primitives + key(...) macro payload.nim toPayload + payload(...) macro schema.nim CREATE TABLE + connection pragmas + user_version backend_sqlite.nim KvBackend, applyOps (single source of write SQL), getOne/existsOne/deleteOne, scanRange (asc/desc, half-open ranges, open-ended stop), countRange backend_comm.nim EventBroker(mt) PersistEvent + 5 RequestBroker(mt) declarations; encodeErr/decodeErr boundary helpers backend_thread.nim startStorageThread / stopStorageThread (shared allocShared0 arg, cstring dbPath, atomic ready/shutdown flags); per-thread provider registration persistency.nim Persistency + Job types, singleton state, public facade ../requests/lifecycle_requests.nim Teardown MultiRequestBroker Tests (69 cases, all passing): test_keys.nim sort-order invariants (length-prefix strings, sign-flipped ints, composite tuples, prefix range) test_backend.nim round-trip / replace / delete-return-value / batched atomicity / asc-desc-half-open-open- ended scans / category isolation / batch txDelete test_lifecycle.nim open-or-create rootDir / non-dir collision / reopen across sessions / idempotent openJob / two-tenant parallel isolation / closeJob joins worker / dropJob removes file / acked delete test_facade.nim put-then-get / atomic batch / scanPrefix asc/desc / deleteAcked hit-miss / fire-and-forget delete / two-tenant facade isolation test_encoding.nim tuple/named-tuple/object keys, embedded Key, enum encoding, field-major composite sort, payload struct encoding, end-to-end struct round-trip through SQLite test_string_lookup.nim peJobNotFound semantics / hasJob / subscript / persistPut+get via id / reads short-circuit / writes drop+warn / persistEncoded via id / scan parity Job-ref vs id test_singleton.nim idempotent same-rootDir / different-rootDir rejection / no-arg instance lifecycle / reset retargets / reset idempotence / Teardown.request end-to-end Prerequisite delivered in the same series: replace the in-tree broker implementation with the external nim-brokers package; update all broker call-sites (waku_filter_v2, waku_relay, waku_rln_relay, delivery_service, peer_manager, requests/*, factory/*, api tests, etc.) to the new package API; chat2 made to compile again. Note: SDS adapter (Phase 5 of the design) is deferred -- nim-sds is still developed side-by-side and the persistency layer is intentionally SDS-agnostic. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * persistency: pin nim-brokers by URL+commit (workaround for stale registry) The bare `brokers >= 2.0.1` form cannot resolve on machines where the local nimble SAT solver enumerates only the registry-recorded 0.1.0 for brokers. The nim-lang/packages entry for `brokers` carries no per-tag metadata (only the URL), so until that registry entry is refreshed the SAT solver clamps the available-versions list to 0.1.0 and rejects the >= 2.0.1 constraint -- even though pkgs2 and pkgcache both have v2.0.1 cloned locally. Pinning by URL+commit bypasses the registry path entirely. Inline comment in waku.nimble documents the situation and the path back to the bare form once nim-lang/packages is updated. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * persistency: nph format pass Run `nph` on all 57 Nim files touched by this PR. Pure formatting: 17 files re-styled, no semantic change. Suite still 69/69. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Fix build, add local-storage-path config, lazy init of Persistency from Waku start * fix: fix nix deps * fixes for nix build, regenerate deps * reverting accidental dependency changes * Fixing deps * Apply suggestions from code review Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com> * persistency tests: migrate to suite / asyncTest / await Match the in-tree test convention (procSuite -> suite, sync test + waitFor -> asyncTest + await): - procSuite "X": -> suite "X": - For tests doing async work: test -> asyncTest, waitFor -> await. - Poll helpers (proc waitFor(t: Job, ...) in test_lifecycle.nim, proc waitUntilExists(...) in test_facade.nim and test_string_lookup.nim) -> Future[bool] {.async.}, internal `waitFor X` -> `await X`, internal `sleep(N)` -> `await sleepAsync(chronos.milliseconds(N))`. - Renamed test_lifecycle.nim's helper proc from `waitFor(t: Job, ...)` -> `pollExists(t: Job, ...)`; the previous name shadowed chronos.waitFor in the chronos macro expansion. - `chronos.milliseconds(N)` explicitly qualified because `std/times` also exports `milliseconds` (returning TimeInterval, not Duration). - `check await x` -> `let okN = await x; check okN` to dodge chronos's "yield in expr not lowered" with await-as-macro-argument. - `(await x).foo()` -> `let awN = await x; ... awN.foo() ...` for the same reason. waku/persistency/persistency.nim: nph also pulled the proc signatures across multiple lines; restored explicit `Future[void] {.async.}` return types after the colon (an intermediate nph pass had elided them). Suite: 71 / 71 OK against the new async write surface. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * use idiomatic valueOr instead of ifs * Reworked persistency shutdown, remove not necessary teardown mechanism * Use const for DefaultStoragePath * format to follow coding guidelines - no use of result and explicit returns - no functional change --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2026-05-16 00:09:07 +02:00
await eventManager.teardown()
feat: waku api send (#3669) * Introduce api/send Added events and requests for support. Reworked delivery_monitor into a featured devlivery_service, that - supports relay publish and lightpush depending on configuration but with fallback options - if available and configured it utilizes store api to confirm message delivery - emits message delivery events accordingly prepare for use in api_example * Fix edge mode config and test added * Fix some import issues, start and stop waku shall not throw exception but return with result properly * Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover * add api_example app to examples2 * Adapt after merge from master * Adapt code for using broker context * Fix brokerCtx settings for all usedbrokers, cover locked node init * Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api * More test added * Fix multi propagate event emit, fix fail send test case * Fix rebase * Fix PushMessageHandlers in tests * adapt libwaku to api changes * Fix relay test by adapting publish return error in case NoPeersToPublish * Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos * Fix rln relay broker, missed brokerCtx * Fix rest relay test failed, due to publish will fail if no peer avail * ignore anvil test state file * Make terst_wakunode_rln_relay broker context aware to fix * Fix waku rln tests by having them broker context aware * fix typo in test_app.nim
2026-01-30 01:06:00 +01:00
let envelope = MessageEnvelope.init(
ContentTopic("/waku/2/default-content/proto"), "test payload"
)
let requestId = (await node.send(envelope)).valueOr:
raiseAssert error
# Wait for events with timeout
const eventTimeout = 10.seconds
discard await eventManager.waitForEvents(eventTimeout)
eventManager.validate({SendEventOutcome.Propagated}, requestId)
(await node.stop()).isOkOr:
raiseAssert "Failed to stop node: " & error
asyncTest "Send only propagates fallback to lightpush":
var node: Waku
lockNewGlobalBrokerContext:
node = (await createNode(createApiNodeConf())).valueOr:
raiseAssert error
(await startWaku(addr node)).isOkOr:
raiseAssert "Failed to start Waku node: " & error
await node.node.connectToNodes(@[lightpushNodePeerInfo])
let eventManager = newSendEventListenerManager(node.brokerCtx)
defer:
feat: persistency (#3880) * persistency: per-job SQLite-backed storage layer (singleton, brokered) Adds a backend-neutral CRUD library at waku/persistency/, plus the nim-brokers dependency swap that enables it. Architecture (ports-and-adapters): * Persistency: process-wide singleton, one root directory. * Job: one tenant, one DB file, one worker thread, one BrokerContext. * Backend: SQLite via waku/common/databases/db_sqlite. Uniform schema kv(category BLOB, key BLOB, payload BLOB) PRIMARY KEY (category, key) WITHOUT ROWID, WAL mode. * Writes are fire-and-forget via EventBroker(mt) PersistEvent. * Reads are async via five RequestBroker(mt) shapes (KvGet, KvExists, KvScan, KvCount, KvDelete). Reads return Result[T, PersistencyError]. * One storage thread per job; tenants isolated by BrokerContext. Public surface (waku/persistency/persistency.nim): Persistency.instance(rootDir) / Persistency.instance() / Persistency.reset() p.openJob(id) / p.closeJob(id) / p.dropJob(id) / p.close() p.job(id) / p[id] / p.hasJob(id) Writes (Job form & string-id form, fire-and-forget): persist / persistPut / persistDelete / persistEncoded Reads (Job form & string-id form, async Result): get / exists / scan / scanPrefix / count / deleteAcked Key & payload encoding (keys.nim, payload.nim): * encodePart family + variadic key(...) / payload(...) macros + single-value toKey / toPayload. * Primitives: string and openArray[byte] are 2-byte BE length + bytes; int{8..64} are sign-flipped 8-byte BE; uint{16..64} are 8-byte BE; bool/byte/char are 1 byte; enums are int64(ord(v)). * Generic encodePart[T: tuple | object] recurses through fields() so any composite Nim type is encodable without ceremony. * Stable across Nim/C compiler upgrades: no sizeof, no memcpy, no cast on pointers, no host-endianness dependency. * `rawKey(bytes)` + `persistPut(..., openArray[byte])` let callers bypass the built-in encoder with their own format (CBOR, protobuf...). Lifecycle: * Persistency.new is private; Persistency.instance is the only public constructor. Same rootDir is idempotent; conflicting rootDir is peInvalidArgument. Persistency.reset for test/restart paths. * openJob opens-or-creates the per-job SQLite file; an existing file is reused with its data preserved. * Teardown integration: Persistency.instance registers a Teardown MultiRequestBroker provider that closes all jobs and clears the singleton slot when Waku.stop() issues Teardown.request. Internal layering: types.nim pure value types (Key, KeyRange, KvRow, TxOp, PersistencyError) keys.nim encodePart primitives + key(...) macro payload.nim toPayload + payload(...) macro schema.nim CREATE TABLE + connection pragmas + user_version backend_sqlite.nim KvBackend, applyOps (single source of write SQL), getOne/existsOne/deleteOne, scanRange (asc/desc, half-open ranges, open-ended stop), countRange backend_comm.nim EventBroker(mt) PersistEvent + 5 RequestBroker(mt) declarations; encodeErr/decodeErr boundary helpers backend_thread.nim startStorageThread / stopStorageThread (shared allocShared0 arg, cstring dbPath, atomic ready/shutdown flags); per-thread provider registration persistency.nim Persistency + Job types, singleton state, public facade ../requests/lifecycle_requests.nim Teardown MultiRequestBroker Tests (69 cases, all passing): test_keys.nim sort-order invariants (length-prefix strings, sign-flipped ints, composite tuples, prefix range) test_backend.nim round-trip / replace / delete-return-value / batched atomicity / asc-desc-half-open-open- ended scans / category isolation / batch txDelete test_lifecycle.nim open-or-create rootDir / non-dir collision / reopen across sessions / idempotent openJob / two-tenant parallel isolation / closeJob joins worker / dropJob removes file / acked delete test_facade.nim put-then-get / atomic batch / scanPrefix asc/desc / deleteAcked hit-miss / fire-and-forget delete / two-tenant facade isolation test_encoding.nim tuple/named-tuple/object keys, embedded Key, enum encoding, field-major composite sort, payload struct encoding, end-to-end struct round-trip through SQLite test_string_lookup.nim peJobNotFound semantics / hasJob / subscript / persistPut+get via id / reads short-circuit / writes drop+warn / persistEncoded via id / scan parity Job-ref vs id test_singleton.nim idempotent same-rootDir / different-rootDir rejection / no-arg instance lifecycle / reset retargets / reset idempotence / Teardown.request end-to-end Prerequisite delivered in the same series: replace the in-tree broker implementation with the external nim-brokers package; update all broker call-sites (waku_filter_v2, waku_relay, waku_rln_relay, delivery_service, peer_manager, requests/*, factory/*, api tests, etc.) to the new package API; chat2 made to compile again. Note: SDS adapter (Phase 5 of the design) is deferred -- nim-sds is still developed side-by-side and the persistency layer is intentionally SDS-agnostic. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * persistency: pin nim-brokers by URL+commit (workaround for stale registry) The bare `brokers >= 2.0.1` form cannot resolve on machines where the local nimble SAT solver enumerates only the registry-recorded 0.1.0 for brokers. The nim-lang/packages entry for `brokers` carries no per-tag metadata (only the URL), so until that registry entry is refreshed the SAT solver clamps the available-versions list to 0.1.0 and rejects the >= 2.0.1 constraint -- even though pkgs2 and pkgcache both have v2.0.1 cloned locally. Pinning by URL+commit bypasses the registry path entirely. Inline comment in waku.nimble documents the situation and the path back to the bare form once nim-lang/packages is updated. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * persistency: nph format pass Run `nph` on all 57 Nim files touched by this PR. Pure formatting: 17 files re-styled, no semantic change. Suite still 69/69. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Fix build, add local-storage-path config, lazy init of Persistency from Waku start * fix: fix nix deps * fixes for nix build, regenerate deps * reverting accidental dependency changes * Fixing deps * Apply suggestions from code review Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com> * persistency tests: migrate to suite / asyncTest / await Match the in-tree test convention (procSuite -> suite, sync test + waitFor -> asyncTest + await): - procSuite "X": -> suite "X": - For tests doing async work: test -> asyncTest, waitFor -> await. - Poll helpers (proc waitFor(t: Job, ...) in test_lifecycle.nim, proc waitUntilExists(...) in test_facade.nim and test_string_lookup.nim) -> Future[bool] {.async.}, internal `waitFor X` -> `await X`, internal `sleep(N)` -> `await sleepAsync(chronos.milliseconds(N))`. - Renamed test_lifecycle.nim's helper proc from `waitFor(t: Job, ...)` -> `pollExists(t: Job, ...)`; the previous name shadowed chronos.waitFor in the chronos macro expansion. - `chronos.milliseconds(N)` explicitly qualified because `std/times` also exports `milliseconds` (returning TimeInterval, not Duration). - `check await x` -> `let okN = await x; check okN` to dodge chronos's "yield in expr not lowered" with await-as-macro-argument. - `(await x).foo()` -> `let awN = await x; ... awN.foo() ...` for the same reason. waku/persistency/persistency.nim: nph also pulled the proc signatures across multiple lines; restored explicit `Future[void] {.async.}` return types after the colon (an intermediate nph pass had elided them). Suite: 71 / 71 OK against the new async write surface. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * use idiomatic valueOr instead of ifs * Reworked persistency shutdown, remove not necessary teardown mechanism * Use const for DefaultStoragePath * format to follow coding guidelines - no use of result and explicit returns - no functional change --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2026-05-16 00:09:07 +02:00
await eventManager.teardown()
feat: waku api send (#3669) * Introduce api/send Added events and requests for support. Reworked delivery_monitor into a featured devlivery_service, that - supports relay publish and lightpush depending on configuration but with fallback options - if available and configured it utilizes store api to confirm message delivery - emits message delivery events accordingly prepare for use in api_example * Fix edge mode config and test added * Fix some import issues, start and stop waku shall not throw exception but return with result properly * Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover * add api_example app to examples2 * Adapt after merge from master * Adapt code for using broker context * Fix brokerCtx settings for all usedbrokers, cover locked node init * Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api * More test added * Fix multi propagate event emit, fix fail send test case * Fix rebase * Fix PushMessageHandlers in tests * adapt libwaku to api changes * Fix relay test by adapting publish return error in case NoPeersToPublish * Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos * Fix rln relay broker, missed brokerCtx * Fix rest relay test failed, due to publish will fail if no peer avail * ignore anvil test state file * Make terst_wakunode_rln_relay broker context aware to fix * Fix waku rln tests by having them broker context aware * fix typo in test_app.nim
2026-01-30 01:06:00 +01:00
let envelope = MessageEnvelope.init(
ContentTopic("/waku/2/default-content/proto"), "test payload"
)
let requestId = (await node.send(envelope)).valueOr:
raiseAssert error
# Wait for events with timeout
const eventTimeout = 10.seconds
discard await eventManager.waitForEvents(eventTimeout)
eventManager.validate({SendEventOutcome.Propagated}, requestId)
(await node.stop()).isOkOr:
raiseAssert "Failed to stop node: " & error
asyncTest "Send fully validates fallback to lightpush":
var node: Waku
lockNewGlobalBrokerContext:
node = (await createNode(createApiNodeConf())).valueOr:
raiseAssert error
(await startWaku(addr node)).isOkOr:
raiseAssert "Failed to start Waku node: " & error
await node.node.connectToNodes(@[lightpushNodePeerInfo, storeNodePeerInfo])
let eventManager = newSendEventListenerManager(node.brokerCtx)
defer:
feat: persistency (#3880) * persistency: per-job SQLite-backed storage layer (singleton, brokered) Adds a backend-neutral CRUD library at waku/persistency/, plus the nim-brokers dependency swap that enables it. Architecture (ports-and-adapters): * Persistency: process-wide singleton, one root directory. * Job: one tenant, one DB file, one worker thread, one BrokerContext. * Backend: SQLite via waku/common/databases/db_sqlite. Uniform schema kv(category BLOB, key BLOB, payload BLOB) PRIMARY KEY (category, key) WITHOUT ROWID, WAL mode. * Writes are fire-and-forget via EventBroker(mt) PersistEvent. * Reads are async via five RequestBroker(mt) shapes (KvGet, KvExists, KvScan, KvCount, KvDelete). Reads return Result[T, PersistencyError]. * One storage thread per job; tenants isolated by BrokerContext. Public surface (waku/persistency/persistency.nim): Persistency.instance(rootDir) / Persistency.instance() / Persistency.reset() p.openJob(id) / p.closeJob(id) / p.dropJob(id) / p.close() p.job(id) / p[id] / p.hasJob(id) Writes (Job form & string-id form, fire-and-forget): persist / persistPut / persistDelete / persistEncoded Reads (Job form & string-id form, async Result): get / exists / scan / scanPrefix / count / deleteAcked Key & payload encoding (keys.nim, payload.nim): * encodePart family + variadic key(...) / payload(...) macros + single-value toKey / toPayload. * Primitives: string and openArray[byte] are 2-byte BE length + bytes; int{8..64} are sign-flipped 8-byte BE; uint{16..64} are 8-byte BE; bool/byte/char are 1 byte; enums are int64(ord(v)). * Generic encodePart[T: tuple | object] recurses through fields() so any composite Nim type is encodable without ceremony. * Stable across Nim/C compiler upgrades: no sizeof, no memcpy, no cast on pointers, no host-endianness dependency. * `rawKey(bytes)` + `persistPut(..., openArray[byte])` let callers bypass the built-in encoder with their own format (CBOR, protobuf...). Lifecycle: * Persistency.new is private; Persistency.instance is the only public constructor. Same rootDir is idempotent; conflicting rootDir is peInvalidArgument. Persistency.reset for test/restart paths. * openJob opens-or-creates the per-job SQLite file; an existing file is reused with its data preserved. * Teardown integration: Persistency.instance registers a Teardown MultiRequestBroker provider that closes all jobs and clears the singleton slot when Waku.stop() issues Teardown.request. Internal layering: types.nim pure value types (Key, KeyRange, KvRow, TxOp, PersistencyError) keys.nim encodePart primitives + key(...) macro payload.nim toPayload + payload(...) macro schema.nim CREATE TABLE + connection pragmas + user_version backend_sqlite.nim KvBackend, applyOps (single source of write SQL), getOne/existsOne/deleteOne, scanRange (asc/desc, half-open ranges, open-ended stop), countRange backend_comm.nim EventBroker(mt) PersistEvent + 5 RequestBroker(mt) declarations; encodeErr/decodeErr boundary helpers backend_thread.nim startStorageThread / stopStorageThread (shared allocShared0 arg, cstring dbPath, atomic ready/shutdown flags); per-thread provider registration persistency.nim Persistency + Job types, singleton state, public facade ../requests/lifecycle_requests.nim Teardown MultiRequestBroker Tests (69 cases, all passing): test_keys.nim sort-order invariants (length-prefix strings, sign-flipped ints, composite tuples, prefix range) test_backend.nim round-trip / replace / delete-return-value / batched atomicity / asc-desc-half-open-open- ended scans / category isolation / batch txDelete test_lifecycle.nim open-or-create rootDir / non-dir collision / reopen across sessions / idempotent openJob / two-tenant parallel isolation / closeJob joins worker / dropJob removes file / acked delete test_facade.nim put-then-get / atomic batch / scanPrefix asc/desc / deleteAcked hit-miss / fire-and-forget delete / two-tenant facade isolation test_encoding.nim tuple/named-tuple/object keys, embedded Key, enum encoding, field-major composite sort, payload struct encoding, end-to-end struct round-trip through SQLite test_string_lookup.nim peJobNotFound semantics / hasJob / subscript / persistPut+get via id / reads short-circuit / writes drop+warn / persistEncoded via id / scan parity Job-ref vs id test_singleton.nim idempotent same-rootDir / different-rootDir rejection / no-arg instance lifecycle / reset retargets / reset idempotence / Teardown.request end-to-end Prerequisite delivered in the same series: replace the in-tree broker implementation with the external nim-brokers package; update all broker call-sites (waku_filter_v2, waku_relay, waku_rln_relay, delivery_service, peer_manager, requests/*, factory/*, api tests, etc.) to the new package API; chat2 made to compile again. Note: SDS adapter (Phase 5 of the design) is deferred -- nim-sds is still developed side-by-side and the persistency layer is intentionally SDS-agnostic. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * persistency: pin nim-brokers by URL+commit (workaround for stale registry) The bare `brokers >= 2.0.1` form cannot resolve on machines where the local nimble SAT solver enumerates only the registry-recorded 0.1.0 for brokers. The nim-lang/packages entry for `brokers` carries no per-tag metadata (only the URL), so until that registry entry is refreshed the SAT solver clamps the available-versions list to 0.1.0 and rejects the >= 2.0.1 constraint -- even though pkgs2 and pkgcache both have v2.0.1 cloned locally. Pinning by URL+commit bypasses the registry path entirely. Inline comment in waku.nimble documents the situation and the path back to the bare form once nim-lang/packages is updated. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * persistency: nph format pass Run `nph` on all 57 Nim files touched by this PR. Pure formatting: 17 files re-styled, no semantic change. Suite still 69/69. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Fix build, add local-storage-path config, lazy init of Persistency from Waku start * fix: fix nix deps * fixes for nix build, regenerate deps * reverting accidental dependency changes * Fixing deps * Apply suggestions from code review Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com> * persistency tests: migrate to suite / asyncTest / await Match the in-tree test convention (procSuite -> suite, sync test + waitFor -> asyncTest + await): - procSuite "X": -> suite "X": - For tests doing async work: test -> asyncTest, waitFor -> await. - Poll helpers (proc waitFor(t: Job, ...) in test_lifecycle.nim, proc waitUntilExists(...) in test_facade.nim and test_string_lookup.nim) -> Future[bool] {.async.}, internal `waitFor X` -> `await X`, internal `sleep(N)` -> `await sleepAsync(chronos.milliseconds(N))`. - Renamed test_lifecycle.nim's helper proc from `waitFor(t: Job, ...)` -> `pollExists(t: Job, ...)`; the previous name shadowed chronos.waitFor in the chronos macro expansion. - `chronos.milliseconds(N)` explicitly qualified because `std/times` also exports `milliseconds` (returning TimeInterval, not Duration). - `check await x` -> `let okN = await x; check okN` to dodge chronos's "yield in expr not lowered" with await-as-macro-argument. - `(await x).foo()` -> `let awN = await x; ... awN.foo() ...` for the same reason. waku/persistency/persistency.nim: nph also pulled the proc signatures across multiple lines; restored explicit `Future[void] {.async.}` return types after the colon (an intermediate nph pass had elided them). Suite: 71 / 71 OK against the new async write surface. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * use idiomatic valueOr instead of ifs * Reworked persistency shutdown, remove not necessary teardown mechanism * Use const for DefaultStoragePath * format to follow coding guidelines - no use of result and explicit returns - no functional change --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2026-05-16 00:09:07 +02:00
await eventManager.teardown()
feat: waku api send (#3669) * Introduce api/send Added events and requests for support. Reworked delivery_monitor into a featured devlivery_service, that - supports relay publish and lightpush depending on configuration but with fallback options - if available and configured it utilizes store api to confirm message delivery - emits message delivery events accordingly prepare for use in api_example * Fix edge mode config and test added * Fix some import issues, start and stop waku shall not throw exception but return with result properly * Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover * add api_example app to examples2 * Adapt after merge from master * Adapt code for using broker context * Fix brokerCtx settings for all usedbrokers, cover locked node init * Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api * More test added * Fix multi propagate event emit, fix fail send test case * Fix rebase * Fix PushMessageHandlers in tests * adapt libwaku to api changes * Fix relay test by adapting publish return error in case NoPeersToPublish * Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos * Fix rln relay broker, missed brokerCtx * Fix rest relay test failed, due to publish will fail if no peer avail * ignore anvil test state file * Make terst_wakunode_rln_relay broker context aware to fix * Fix waku rln tests by having them broker context aware * fix typo in test_app.nim
2026-01-30 01:06:00 +01:00
let envelope = MessageEnvelope.init(
ContentTopic("/waku/2/default-content/proto"), "test payload"
)
let requestId = (await node.send(envelope)).valueOr:
raiseAssert error
# Wait for events with timeout
const eventTimeout = 10.seconds
discard await eventManager.waitForEvents(eventTimeout)
eventManager.validate(
{SendEventOutcome.Propagated, SendEventOutcome.Sent}, requestId
)
(await node.stop()).isOkOr:
raiseAssert "Failed to stop node: " & error
asyncTest "Send fails with event":
var fakeLightpushNode: WakuNode
lockNewGlobalBrokerContext:
fakeLightpushNode =
newTestWakuNode(generateSecp256k1Key(), parseIpAddress("0.0.0.0"), Port(0))
Chore: adapt cli args for delivery api (#3744) * LogosDeliveryAPI: NodeConfig -> WakluNodeConf + mode selector and logos.dev preset * Adjustment made on test, logos.dev preset * change default agentString from nwaku to logos-delivery * Add p2pReliability switch to presets and make it default to true. * Borrow entryNode idea from NodeConfig to WakuNodeConf to easy shortcut among diffrent bootstrap node list all needs different formats * Fix rateLimit assignment for builder * Remove Core mode default as we already have a defaul, user must override * Removed obsolate API createNode with NodeConfig - tests are refactored for WakuNodeConf usage * Fix failing test due to twn-clusterid(1) default has overwrite for maxMessagSize. Fix readme.
2026-03-03 19:17:54 +01:00
fakeLightpushNode.mountMetadata(3, @[0'u16]).isOkOr:
feat: waku api send (#3669) * Introduce api/send Added events and requests for support. Reworked delivery_monitor into a featured devlivery_service, that - supports relay publish and lightpush depending on configuration but with fallback options - if available and configured it utilizes store api to confirm message delivery - emits message delivery events accordingly prepare for use in api_example * Fix edge mode config and test added * Fix some import issues, start and stop waku shall not throw exception but return with result properly * Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover * add api_example app to examples2 * Adapt after merge from master * Adapt code for using broker context * Fix brokerCtx settings for all usedbrokers, cover locked node init * Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api * More test added * Fix multi propagate event emit, fix fail send test case * Fix rebase * Fix PushMessageHandlers in tests * adapt libwaku to api changes * Fix relay test by adapting publish return error in case NoPeersToPublish * Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos * Fix rln relay broker, missed brokerCtx * Fix rest relay test failed, due to publish will fail if no peer avail * ignore anvil test state file * Make terst_wakunode_rln_relay broker context aware to fix * Fix waku rln tests by having them broker context aware * fix typo in test_app.nim
2026-01-30 01:06:00 +01:00
raiseAssert "Failed to mount metadata: " & error
(await fakeLightpushNode.mountRelay()).isOkOr:
raiseAssert "Failed to mount relay"
(await fakeLightpushNode.mountLightPush()).isOkOr:
raiseAssert "Failed to mount lightpush"
await fakeLightpushNode.mountLibp2pPing()
await fakeLightpushNode.start()
let fakeLightpushNodePeerInfo = fakeLightpushNode.peerInfo.toRemotePeerInfo()
proc dummyHandler(
topic: PubsubTopic, msg: WakuMessage
): Future[void] {.async, gcsafe.} =
discard
fakeLightpushNode.subscribe(
Chore: adapt cli args for delivery api (#3744) * LogosDeliveryAPI: NodeConfig -> WakluNodeConf + mode selector and logos.dev preset * Adjustment made on test, logos.dev preset * change default agentString from nwaku to logos-delivery * Add p2pReliability switch to presets and make it default to true. * Borrow entryNode idea from NodeConfig to WakuNodeConf to easy shortcut among diffrent bootstrap node list all needs different formats * Fix rateLimit assignment for builder * Remove Core mode default as we already have a defaul, user must override * Removed obsolate API createNode with NodeConfig - tests are refactored for WakuNodeConf usage * Fix failing test due to twn-clusterid(1) default has overwrite for maxMessagSize. Fix readme.
2026-03-03 19:17:54 +01:00
(kind: PubsubSub, topic: PubsubTopic("/waku/2/rs/3/0")), dummyHandler
feat: waku api send (#3669) * Introduce api/send Added events and requests for support. Reworked delivery_monitor into a featured devlivery_service, that - supports relay publish and lightpush depending on configuration but with fallback options - if available and configured it utilizes store api to confirm message delivery - emits message delivery events accordingly prepare for use in api_example * Fix edge mode config and test added * Fix some import issues, start and stop waku shall not throw exception but return with result properly * Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover * add api_example app to examples2 * Adapt after merge from master * Adapt code for using broker context * Fix brokerCtx settings for all usedbrokers, cover locked node init * Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api * More test added * Fix multi propagate event emit, fix fail send test case * Fix rebase * Fix PushMessageHandlers in tests * adapt libwaku to api changes * Fix relay test by adapting publish return error in case NoPeersToPublish * Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos * Fix rln relay broker, missed brokerCtx * Fix rest relay test failed, due to publish will fail if no peer avail * ignore anvil test state file * Make terst_wakunode_rln_relay broker context aware to fix * Fix waku rln tests by having them broker context aware * fix typo in test_app.nim
2026-01-30 01:06:00 +01:00
).isOkOr:
raiseAssert "Failed to subscribe fakeLightpushNode: " & error
var node: Waku
lockNewGlobalBrokerContext:
Chore: adapt cli args for delivery api (#3744) * LogosDeliveryAPI: NodeConfig -> WakluNodeConf + mode selector and logos.dev preset * Adjustment made on test, logos.dev preset * change default agentString from nwaku to logos-delivery * Add p2pReliability switch to presets and make it default to true. * Borrow entryNode idea from NodeConfig to WakuNodeConf to easy shortcut among diffrent bootstrap node list all needs different formats * Fix rateLimit assignment for builder * Remove Core mode default as we already have a defaul, user must override * Removed obsolate API createNode with NodeConfig - tests are refactored for WakuNodeConf usage * Fix failing test due to twn-clusterid(1) default has overwrite for maxMessagSize. Fix readme.
2026-03-03 19:17:54 +01:00
node = (await createNode(createApiNodeConf(cli_args.WakuMode.Edge))).valueOr:
feat: waku api send (#3669) * Introduce api/send Added events and requests for support. Reworked delivery_monitor into a featured devlivery_service, that - supports relay publish and lightpush depending on configuration but with fallback options - if available and configured it utilizes store api to confirm message delivery - emits message delivery events accordingly prepare for use in api_example * Fix edge mode config and test added * Fix some import issues, start and stop waku shall not throw exception but return with result properly * Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover * add api_example app to examples2 * Adapt after merge from master * Adapt code for using broker context * Fix brokerCtx settings for all usedbrokers, cover locked node init * Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api * More test added * Fix multi propagate event emit, fix fail send test case * Fix rebase * Fix PushMessageHandlers in tests * adapt libwaku to api changes * Fix relay test by adapting publish return error in case NoPeersToPublish * Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos * Fix rln relay broker, missed brokerCtx * Fix rest relay test failed, due to publish will fail if no peer avail * ignore anvil test state file * Make terst_wakunode_rln_relay broker context aware to fix * Fix waku rln tests by having them broker context aware * fix typo in test_app.nim
2026-01-30 01:06:00 +01:00
raiseAssert error
(await startWaku(addr node)).isOkOr:
raiseAssert "Failed to start Waku node: " & error
await node.node.connectToNodes(@[fakeLightpushNodePeerInfo])
let eventManager = newSendEventListenerManager(node.brokerCtx)
defer:
feat: persistency (#3880) * persistency: per-job SQLite-backed storage layer (singleton, brokered) Adds a backend-neutral CRUD library at waku/persistency/, plus the nim-brokers dependency swap that enables it. Architecture (ports-and-adapters): * Persistency: process-wide singleton, one root directory. * Job: one tenant, one DB file, one worker thread, one BrokerContext. * Backend: SQLite via waku/common/databases/db_sqlite. Uniform schema kv(category BLOB, key BLOB, payload BLOB) PRIMARY KEY (category, key) WITHOUT ROWID, WAL mode. * Writes are fire-and-forget via EventBroker(mt) PersistEvent. * Reads are async via five RequestBroker(mt) shapes (KvGet, KvExists, KvScan, KvCount, KvDelete). Reads return Result[T, PersistencyError]. * One storage thread per job; tenants isolated by BrokerContext. Public surface (waku/persistency/persistency.nim): Persistency.instance(rootDir) / Persistency.instance() / Persistency.reset() p.openJob(id) / p.closeJob(id) / p.dropJob(id) / p.close() p.job(id) / p[id] / p.hasJob(id) Writes (Job form & string-id form, fire-and-forget): persist / persistPut / persistDelete / persistEncoded Reads (Job form & string-id form, async Result): get / exists / scan / scanPrefix / count / deleteAcked Key & payload encoding (keys.nim, payload.nim): * encodePart family + variadic key(...) / payload(...) macros + single-value toKey / toPayload. * Primitives: string and openArray[byte] are 2-byte BE length + bytes; int{8..64} are sign-flipped 8-byte BE; uint{16..64} are 8-byte BE; bool/byte/char are 1 byte; enums are int64(ord(v)). * Generic encodePart[T: tuple | object] recurses through fields() so any composite Nim type is encodable without ceremony. * Stable across Nim/C compiler upgrades: no sizeof, no memcpy, no cast on pointers, no host-endianness dependency. * `rawKey(bytes)` + `persistPut(..., openArray[byte])` let callers bypass the built-in encoder with their own format (CBOR, protobuf...). Lifecycle: * Persistency.new is private; Persistency.instance is the only public constructor. Same rootDir is idempotent; conflicting rootDir is peInvalidArgument. Persistency.reset for test/restart paths. * openJob opens-or-creates the per-job SQLite file; an existing file is reused with its data preserved. * Teardown integration: Persistency.instance registers a Teardown MultiRequestBroker provider that closes all jobs and clears the singleton slot when Waku.stop() issues Teardown.request. Internal layering: types.nim pure value types (Key, KeyRange, KvRow, TxOp, PersistencyError) keys.nim encodePart primitives + key(...) macro payload.nim toPayload + payload(...) macro schema.nim CREATE TABLE + connection pragmas + user_version backend_sqlite.nim KvBackend, applyOps (single source of write SQL), getOne/existsOne/deleteOne, scanRange (asc/desc, half-open ranges, open-ended stop), countRange backend_comm.nim EventBroker(mt) PersistEvent + 5 RequestBroker(mt) declarations; encodeErr/decodeErr boundary helpers backend_thread.nim startStorageThread / stopStorageThread (shared allocShared0 arg, cstring dbPath, atomic ready/shutdown flags); per-thread provider registration persistency.nim Persistency + Job types, singleton state, public facade ../requests/lifecycle_requests.nim Teardown MultiRequestBroker Tests (69 cases, all passing): test_keys.nim sort-order invariants (length-prefix strings, sign-flipped ints, composite tuples, prefix range) test_backend.nim round-trip / replace / delete-return-value / batched atomicity / asc-desc-half-open-open- ended scans / category isolation / batch txDelete test_lifecycle.nim open-or-create rootDir / non-dir collision / reopen across sessions / idempotent openJob / two-tenant parallel isolation / closeJob joins worker / dropJob removes file / acked delete test_facade.nim put-then-get / atomic batch / scanPrefix asc/desc / deleteAcked hit-miss / fire-and-forget delete / two-tenant facade isolation test_encoding.nim tuple/named-tuple/object keys, embedded Key, enum encoding, field-major composite sort, payload struct encoding, end-to-end struct round-trip through SQLite test_string_lookup.nim peJobNotFound semantics / hasJob / subscript / persistPut+get via id / reads short-circuit / writes drop+warn / persistEncoded via id / scan parity Job-ref vs id test_singleton.nim idempotent same-rootDir / different-rootDir rejection / no-arg instance lifecycle / reset retargets / reset idempotence / Teardown.request end-to-end Prerequisite delivered in the same series: replace the in-tree broker implementation with the external nim-brokers package; update all broker call-sites (waku_filter_v2, waku_relay, waku_rln_relay, delivery_service, peer_manager, requests/*, factory/*, api tests, etc.) to the new package API; chat2 made to compile again. Note: SDS adapter (Phase 5 of the design) is deferred -- nim-sds is still developed side-by-side and the persistency layer is intentionally SDS-agnostic. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * persistency: pin nim-brokers by URL+commit (workaround for stale registry) The bare `brokers >= 2.0.1` form cannot resolve on machines where the local nimble SAT solver enumerates only the registry-recorded 0.1.0 for brokers. The nim-lang/packages entry for `brokers` carries no per-tag metadata (only the URL), so until that registry entry is refreshed the SAT solver clamps the available-versions list to 0.1.0 and rejects the >= 2.0.1 constraint -- even though pkgs2 and pkgcache both have v2.0.1 cloned locally. Pinning by URL+commit bypasses the registry path entirely. Inline comment in waku.nimble documents the situation and the path back to the bare form once nim-lang/packages is updated. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * persistency: nph format pass Run `nph` on all 57 Nim files touched by this PR. Pure formatting: 17 files re-styled, no semantic change. Suite still 69/69. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Fix build, add local-storage-path config, lazy init of Persistency from Waku start * fix: fix nix deps * fixes for nix build, regenerate deps * reverting accidental dependency changes * Fixing deps * Apply suggestions from code review Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com> * persistency tests: migrate to suite / asyncTest / await Match the in-tree test convention (procSuite -> suite, sync test + waitFor -> asyncTest + await): - procSuite "X": -> suite "X": - For tests doing async work: test -> asyncTest, waitFor -> await. - Poll helpers (proc waitFor(t: Job, ...) in test_lifecycle.nim, proc waitUntilExists(...) in test_facade.nim and test_string_lookup.nim) -> Future[bool] {.async.}, internal `waitFor X` -> `await X`, internal `sleep(N)` -> `await sleepAsync(chronos.milliseconds(N))`. - Renamed test_lifecycle.nim's helper proc from `waitFor(t: Job, ...)` -> `pollExists(t: Job, ...)`; the previous name shadowed chronos.waitFor in the chronos macro expansion. - `chronos.milliseconds(N)` explicitly qualified because `std/times` also exports `milliseconds` (returning TimeInterval, not Duration). - `check await x` -> `let okN = await x; check okN` to dodge chronos's "yield in expr not lowered" with await-as-macro-argument. - `(await x).foo()` -> `let awN = await x; ... awN.foo() ...` for the same reason. waku/persistency/persistency.nim: nph also pulled the proc signatures across multiple lines; restored explicit `Future[void] {.async.}` return types after the colon (an intermediate nph pass had elided them). Suite: 71 / 71 OK against the new async write surface. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * use idiomatic valueOr instead of ifs * Reworked persistency shutdown, remove not necessary teardown mechanism * Use const for DefaultStoragePath * format to follow coding guidelines - no use of result and explicit returns - no functional change --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2026-05-16 00:09:07 +02:00
await eventManager.teardown()
feat: waku api send (#3669) * Introduce api/send Added events and requests for support. Reworked delivery_monitor into a featured devlivery_service, that - supports relay publish and lightpush depending on configuration but with fallback options - if available and configured it utilizes store api to confirm message delivery - emits message delivery events accordingly prepare for use in api_example * Fix edge mode config and test added * Fix some import issues, start and stop waku shall not throw exception but return with result properly * Utlize sync RequestBroker, adapt to non-async broker usage and gcsafe where appropriate, removed leftover * add api_example app to examples2 * Adapt after merge from master * Adapt code for using broker context * Fix brokerCtx settings for all usedbrokers, cover locked node init * Various fixes upon test failures. Added initial of subscribe API and auto-subscribe for send api * More test added * Fix multi propagate event emit, fix fail send test case * Fix rebase * Fix PushMessageHandlers in tests * adapt libwaku to api changes * Fix relay test by adapting publish return error in case NoPeersToPublish * Addressing all remaining review findings. Removed leftovers. Fixed loggings and typos * Fix rln relay broker, missed brokerCtx * Fix rest relay test failed, due to publish will fail if no peer avail * ignore anvil test state file * Make terst_wakunode_rln_relay broker context aware to fix * Fix waku rln tests by having them broker context aware * fix typo in test_app.nim
2026-01-30 01:06:00 +01:00
let envelope = MessageEnvelope.init(
ContentTopic("/waku/2/default-content/proto"), "test payload"
)
let requestId = (await node.send(envelope)).valueOr:
raiseAssert error
echo "Sent message with requestId=", requestId
# Wait for events with timeout
const eventTimeout = 62.seconds
discard await eventManager.waitForEvents(eventTimeout)
eventManager.validate({SendEventOutcome.Error}, requestId)
(await node.stop()).isOkOr:
raiseAssert "Failed to stop node: " & error
Reference in New Issue Copy Permalink