logos-messaging/logos-messaging-nim
1
0
Fork 0
mirror of https://github.com/logos-messaging/logos-messaging-nim.git synced 2026-05-18 00:09:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: persistency (#3880)

Browse Source 
* persistency: per-job SQLite-backed storage layer (singleton, brokered)

Adds a backend-neutral CRUD library at waku/persistency/, plus the
nim-brokers dependency swap that enables it.

Architecture (ports-and-adapters):
  * Persistency: process-wide singleton, one root directory.
  * Job: one tenant, one DB file, one worker thread, one BrokerContext.
  * Backend: SQLite via waku/common/databases/db_sqlite. Uniform schema
    kv(category BLOB, key BLOB, payload BLOB) PRIMARY KEY (category, key)
    WITHOUT ROWID, WAL mode.
  * Writes are fire-and-forget via EventBroker(mt) PersistEvent.
  * Reads are async via five RequestBroker(mt) shapes (KvGet, KvExists,
    KvScan, KvCount, KvDelete). Reads return Result[T, PersistencyError].
  * One storage thread per job; tenants isolated by BrokerContext.

Public surface (waku/persistency/persistency.nim):
  Persistency.instance(rootDir) / Persistency.instance() / Persistency.reset()
  p.openJob(id) / p.closeJob(id) / p.dropJob(id) / p.close()
  p.job(id) / p[id] / p.hasJob(id)
  Writes (Job form & string-id form, fire-and-forget):
    persist / persistPut / persistDelete / persistEncoded
  Reads (Job form & string-id form, async Result):
    get / exists / scan / scanPrefix / count / deleteAcked

Key & payload encoding (keys.nim, payload.nim):
  * encodePart family + variadic key(...) / payload(...) macros +
    single-value toKey / toPayload.
  * Primitives: string and openArray[byte] are 2-byte BE length + bytes;
    int{8..64} are sign-flipped 8-byte BE; uint{16..64} are 8-byte BE;
    bool/byte/char are 1 byte; enums are int64(ord(v)).
  * Generic encodePart[T: tuple | object] recurses through fields() so
    any composite Nim type is encodable without ceremony.
  * Stable across Nim/C compiler upgrades: no sizeof, no memcpy, no
    cast on pointers, no host-endianness dependency.
  * `rawKey(bytes)` + `persistPut(..., openArray[byte])` let callers
    bypass the built-in encoder with their own format (CBOR, protobuf...).

Lifecycle:
  * Persistency.new is private; Persistency.instance is the only public
    constructor. Same rootDir is idempotent; conflicting rootDir is
    peInvalidArgument. Persistency.reset for test/restart paths.
  * openJob opens-or-creates the per-job SQLite file; an existing file
    is reused with its data preserved.
  * Teardown integration: Persistency.instance registers a Teardown
    MultiRequestBroker provider that closes all jobs and clears the
    singleton slot when Waku.stop() issues Teardown.request.

Internal layering:
  types.nim          pure value types (Key, KeyRange, KvRow, TxOp,
                     PersistencyError)
  keys.nim           encodePart primitives + key(...) macro
  payload.nim        toPayload + payload(...) macro
  schema.nim         CREATE TABLE + connection pragmas + user_version
  backend_sqlite.nim KvBackend, applyOps (single source of write SQL),
                     getOne/existsOne/deleteOne, scanRange (asc/desc,
                     half-open ranges, open-ended stop), countRange
  backend_comm.nim   EventBroker(mt) PersistEvent + 5 RequestBroker(mt)
                     declarations; encodeErr/decodeErr boundary helpers
  backend_thread.nim startStorageThread / stopStorageThread (shared
                     allocShared0 arg, cstring dbPath, atomic
                     ready/shutdown flags); per-thread provider
                     registration
  persistency.nim    Persistency + Job types, singleton state, public
                     facade
  ../requests/lifecycle_requests.nim
                     Teardown MultiRequestBroker

Tests (69 cases, all passing):
  test_keys.nim          sort-order invariants (length-prefix strings,
                         sign-flipped ints, composite tuples, prefix
                         range)
  test_backend.nim       round-trip / replace / delete-return-value /
                         batched atomicity / asc-desc-half-open-open-
                         ended scans / category isolation / batch
                         txDelete
  test_lifecycle.nim     open-or-create rootDir / non-dir collision /
                         reopen across sessions / idempotent openJob /
                         two-tenant parallel isolation / closeJob joins
                         worker / dropJob removes file / acked delete
  test_facade.nim        put-then-get / atomic batch / scanPrefix
                         asc/desc / deleteAcked hit-miss /
                         fire-and-forget delete / two-tenant facade
                         isolation
  test_encoding.nim      tuple/named-tuple/object keys, embedded Key,
                         enum encoding, field-major composite sort,
                         payload struct encoding, end-to-end struct
                         round-trip through SQLite
  test_string_lookup.nim peJobNotFound semantics / hasJob / subscript /
                         persistPut+get via id / reads short-circuit /
                         writes drop+warn / persistEncoded via id /
                         scan parity Job-ref vs id
  test_singleton.nim     idempotent same-rootDir / different-rootDir
                         rejection / no-arg instance lifecycle / reset
                         retargets / reset idempotence / Teardown.request
                         end-to-end

Prerequisite delivered in the same series: replace the in-tree broker
implementation with the external nim-brokers package; update all
broker call-sites (waku_filter_v2, waku_relay, waku_rln_relay,
delivery_service, peer_manager, requests/*, factory/*, api tests, etc.)
to the new package API; chat2 made to compile again.

Note: SDS adapter (Phase 5 of the design) is deferred -- nim-sds is
still developed side-by-side and the persistency layer is intentionally
SDS-agnostic.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* persistency: pin nim-brokers by URL+commit (workaround for stale registry)

The bare `brokers >= 2.0.1` form cannot resolve on machines where the
local nimble SAT solver enumerates only the registry-recorded 0.1.0 for
brokers. The nim-lang/packages entry for `brokers` carries no per-tag
metadata (only the URL), so until that registry entry is refreshed the
SAT solver clamps the available-versions list to 0.1.0 and rejects the
>= 2.0.1 constraint -- even though pkgs2 and pkgcache both have v2.0.1
cloned locally.

Pinning by URL+commit bypasses the registry path entirely. Inline
comment in waku.nimble documents the situation and the path back to
the bare form once nim-lang/packages is updated.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* persistency: nph format pass

Run `nph` on all 57 Nim files touched by this PR. Pure formatting:
17 files re-styled, no semantic change. Suite still 69/69.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* Fix build, add local-storage-path config, lazy init of Persistency from Waku start

* fix: fix nix deps

* fixes for nix build, regenerate deps

* reverting accidental dependency changes

* Fixing deps

* Apply suggestions from code review

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>

* persistency tests: migrate to suite / asyncTest / await

Match the in-tree test convention (procSuite -> suite, sync test +
waitFor -> asyncTest + await):

- procSuite "X": -> suite "X":
- For tests doing async work: test -> asyncTest, waitFor -> await.
- Poll helpers (proc waitFor(t: Job, ...) in test_lifecycle.nim,
  proc waitUntilExists(...) in test_facade.nim and
  test_string_lookup.nim) -> Future[bool] {.async.}, internal
  `waitFor X` -> `await X`, internal `sleep(N)` ->
  `await sleepAsync(chronos.milliseconds(N))`.
- Renamed test_lifecycle.nim's helper proc from `waitFor(t: Job, ...)`
  -> `pollExists(t: Job, ...)`; the previous name shadowed
  chronos.waitFor in the chronos macro expansion.
- `chronos.milliseconds(N)` explicitly qualified because `std/times`
  also exports `milliseconds` (returning TimeInterval, not Duration).
- `check await x` -> `let okN = await x; check okN` to dodge chronos's
  "yield in expr not lowered" with await-as-macro-argument.
- `(await x).foo()` -> `let awN = await x; ... awN.foo() ...` for the
  same reason.

waku/persistency/persistency.nim: nph also pulled the proc signatures
across multiple lines; restored explicit `Future[void] {.async.}`
return types after the colon (an intermediate nph pass had elided them).

Suite: 71 / 71 OK against the new async write surface.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* use idiomatic valueOr instead of ifs

* Reworked persistency shutdown, remove not necessary teardown mechanism

* Use const for DefaultStoragePath

* format to follow coding guidelines - no use of result and explicit returns - no functional change

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
This commit is contained in:
NagyZoltanPeter 2026-05-16 00:09:07 +02:00 committed by GitHub
parent 34c197c5cd
commit 42e0aa43d1
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
73 changed files with 3006 additions and 3607 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -85,3 +85,4 @@ nimble.paths
nimbledeps
**/anvil_state/state-deployed-contracts-mint-and-approved.json
.gitnexus

42
AGENTS.md
View File

@ -506,4 +506,46 @@ Language: Nim 2.x | License: MIT or Apache 2.0
Note: For specific version requirements, check `waku.nimble`.
<!-- gitnexus:start -->
# GitNexus — Code Intelligence
This project is indexed by GitNexus as **logos-delivery** (2076 symbols, 2564 relationships, 12 execution flows). Use the GitNexus MCP tools to understand code, assess impact, and navigate safely.
> If any GitNexus tool warns the index is stale, run `npx gitnexus analyze` in terminal first.
## Always Do
- **MUST run impact analysis before editing any symbol.** Before modifying a function, class, or method, run `gitnexus_impact({target: "symbolName", direction: "upstream"})` and report the blast radius (direct callers, affected processes, risk level) to the user.
- **MUST run `gitnexus_detect_changes()` before committing** to verify your changes only affect expected symbols and execution flows.
- **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits.
- When exploring unfamiliar code, use `gitnexus_query({query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance.
- When you need full context on a specific symbol — callers, callees, which execution flows it participates in — use `gitnexus_context({name: "symbolName"})`.
## Never Do
- NEVER edit a function, class, or method without first running `gitnexus_impact` on it.
- NEVER ignore HIGH or CRITICAL risk warnings from impact analysis.
- NEVER rename symbols with find-and-replace — use `gitnexus_rename` which understands the call graph.
- NEVER commit changes without running `gitnexus_detect_changes()` to check affected scope.
## Resources
| Resource | Use for |
|----------|---------|
| `gitnexus://repo/logos-delivery/context` | Codebase overview, check index freshness |
| `gitnexus://repo/logos-delivery/clusters` | All functional areas |
| `gitnexus://repo/logos-delivery/processes` | All execution flows |
| `gitnexus://repo/logos-delivery/process/{name}` | Step-by-step execution trace |
## CLI
| Task | Read this skill file |
|------|---------------------|
| Understand architecture / "How does X work?" | `.claude/skills/gitnexus/gitnexus-exploring/SKILL.md` |
| Blast radius / "What breaks if I change X?" | `.claude/skills/gitnexus/gitnexus-impact-analysis/SKILL.md` |
| Trace bugs / "Why is X failing?" | `.claude/skills/gitnexus/gitnexus-debugging/SKILL.md` |
| Rename / extract / split / refactor | `.claude/skills/gitnexus/gitnexus-refactoring/SKILL.md` |
| Tools, resources, schema reference | `.claude/skills/gitnexus/gitnexus-guide/SKILL.md` |
| Index, status, clean, wiki CLI commands | `.claude/skills/gitnexus/gitnexus-cli/SKILL.md` |
<!-- gitnexus:end -->

3
apps/chat2/chat2.nim
View File

@ -13,7 +13,8 @@ import
chronos,
eth/keys,
bearssl,
stew/[byteutils, results],
stew/[byteutils],
results,
metrics,
metrics/chronos_httpserver
import

10
apps/chat2/config_chat2.nim
View File

@ -140,7 +140,8 @@ type
metricsServerAddress* {.
desc: "Listening address of the metrics server.",
defaultValue: parseIpAddress("127.0.0.1"),
defaultValue:
IpAddress(family: IpAddressFamily.IPv4, address_v4: [127'u8, 0, 0, 1]),
name: "metrics-server-address"
.}: IpAddress
@ -173,7 +174,10 @@ type
dnsAddrsNameServers* {.
desc:
"DNS name server IPs to query for DNS multiaddrs resolution. Argument may be repeated.",
defaultValue: @[parseIpAddress("1.1.1.1"), parseIpAddress("1.0.0.1")],
defaultValue: @[
IpAddress(family: IpAddressFamily.IPv4, address_v4: [1'u8, 1, 1, 1]),
IpAddress(family: IpAddressFamily.IPv4, address_v4: [1'u8, 0, 0, 1]),
],
name: "dns-addrs-name-server"
.}: seq[IpAddress]
@ -348,4 +352,4 @@ proc parseCmdArg*(T: type EthRpcUrl, s: string): T =
func defaultListenAddress*(conf: Chat2Conf): IpAddress =
# TODO: How should we select between IPv4 and IPv6
# Maybe there should be a config option for this.
(static parseIpAddress("0.0.0.0"))
(static IpAddress(family: IpAddressFamily.IPv4, address_v4: [0'u8, 0, 0, 0]))

2
config.nims
View File

@ -117,7 +117,7 @@ if defined(android):
switch("passL", "--sysroot=" & sysRoot)
switch("cincludes", sysRoot & "/usr/include/")
# begin Nimble config (version 2)
--noNimblePath
when withDir(thisDir(), system.fileExists("nimble.paths")):
--noNimblePath
include "nimble.paths"
# end Nimble config

6
examples/api_example/api_example.nim
View File

@ -33,9 +33,9 @@ proc periodicSender(w: Waku): Future[void] {.async.} =
return
defer:
MessageSentEvent.dropListener(sentListener)
MessageErrorEvent.dropListener(errorListener)
MessagePropagatedEvent.dropListener(propagatedListener)
await MessageSentEvent.dropListener(sentListener)
await MessageErrorEvent.dropListener(errorListener)
await MessagePropagatedEvent.dropListener(propagatedListener)
## Periodically sends a Waku message every 30 seconds
var counter = 0

10
liblogosdelivery/logos_delivery_api/node_api.nim
View File

@ -184,11 +184,11 @@ proc logosdelivery_stop_node(
requireInitializedNode(ctx, "STOP_NODE"):
return err(errMsg)
MessageErrorEvent.dropAllListeners(ctx.myLib[].brokerCtx)
MessageSentEvent.dropAllListeners(ctx.myLib[].brokerCtx)
MessagePropagatedEvent.dropAllListeners(ctx.myLib[].brokerCtx)
MessageReceivedEvent.dropAllListeners(ctx.myLib[].brokerCtx)
EventConnectionStatusChange.dropAllListeners(ctx.myLib[].brokerCtx)
await MessageErrorEvent.dropAllListeners(ctx.myLib[].brokerCtx)
await MessageSentEvent.dropAllListeners(ctx.myLib[].brokerCtx)
await MessagePropagatedEvent.dropAllListeners(ctx.myLib[].brokerCtx)
await MessageReceivedEvent.dropAllListeners(ctx.myLib[].brokerCtx)
await EventConnectionStatusChange.dropAllListeners(ctx.myLib[].brokerCtx)
(await ctx.myLib[].stop()).isOkOr:
let errMsg = $error

32
nimble.lock
View File

@ -263,6 +263,21 @@
"sha1": "8bc8c30b107fdba73b677e5f257c6c42ae1cdc8e"
}
},
"cbor_serialization": {
"version": "0.3.0",
"vcsRevision": "1664160e04d153573373afddc552b9cbf6fbe4dc",
"url": "https://github.com/vacp2p/nim-cbor-serialization",
"downloadMethod": "git",
"dependencies": [
"nim",
"serialization",
"stew",
"results"
],
"checksums": {
"sha1": "ab126eae09a6e39c72972a6a0b83cb06a2ffe8f0"
}
},
"json_serialization": {
"version": "0.4.4",
"vcsRevision": "c343b0e243d9e17e2c40f3a8a24340f7c4a71d44",
@ -312,6 +327,23 @@
"sha1": "8df97c45683abe2337bdff43b844c4fbcc124ca2"
}
},
"brokers": {
"version": "#v2.0.1",
"vcsRevision": "2093ca4d50e581adda73fee7fd16231f990f4cbe",
"url": "https://github.com/NagyZoltanPeter/nim-brokers.git",
"downloadMethod": "git",
"dependencies": [
"nim",
"chronos",
"results",
"chronicles",
"testutils",
"cbor_serialization"
],
"checksums": {
"sha1": "cc74c987af94537e9d44d1b0143aa417299040c5"
}
},
"stint": {
"version": "0.8.2",
"vcsRevision": "470b7892561b5179ab20bd389a69217d6213fe58",

21
nix/deps.nix
View File

@ -129,6 +129,13 @@
fetchSubmodules = true;
};
cbor_serialization = pkgs.fetchgit {
url = "https://github.com/vacp2p/nim-cbor-serialization";
rev = "1664160e04d153573373afddc552b9cbf6fbe4dc";
sha256 = "0c1rj4fk0fcqvsf0yqhxvm8h10aww75gi4yfsjhlczh88ypywii2";
fetchSubmodules = true;
};
json_serialization = pkgs.fetchgit {
url = "https://github.com/status-im/nim-json-serialization";
rev = "c343b0e243d9e17e2c40f3a8a24340f7c4a71d44";
@ -150,6 +157,13 @@
fetchSubmodules = true;
};
brokers = pkgs.fetchgit {
url = "https://github.com/NagyZoltanPeter/nim-brokers.git";
rev = "2093ca4d50e581adda73fee7fd16231f990f4cbe";
sha256 = "0a4ix2q6riqfrd0hfnajisy159qdmk5imwzymppj23rwc8n7d2dx";
fetchSubmodules = true;
};
stint = pkgs.fetchgit {
url = "https://github.com/status-im/nim-stint";
rev = "470b7892561b5179ab20bd389a69217d6213fe58";
@ -262,6 +276,13 @@
fetchSubmodules = true;
};
sds = pkgs.fetchgit {
url = "https://github.com/logos-messaging/nim-sds.git";
rev = "2e9a7683f0e180bf112135fae3a3803eed8490d4";
sha256 = "1dbpvp3zhvdlfxdyggz5waga1vg3b6ndd3acfzhnx8k1wdr01c6f";
fetchSubmodules = true;
};
ffi = pkgs.fetchgit {
url = "https://github.com/logos-messaging/nim-ffi";
rev = "06111de155253b34e47ed2aaed1d61d08d62cc1b";

3
tests/all_tests_waku.nim
View File

@ -85,3 +85,6 @@ import ./api/test_all
# Waku tools tests
import ./tools/test_all
# Persistency library tests
import ./persistency/test_all

7
tests/api/test_api_health.nim
View File

@ -2,11 +2,12 @@
import std/[options, sequtils, times]
import chronos, testutils/unittests, stew/byteutils, libp2p/[switch, peerinfo]
import brokers/broker_context
import ../testlib/[common, wakucore, wakunode, testasync]
import
waku,
waku/[waku_node, waku_core, waku_relay/protocol, common/broker/broker_context],
waku/[waku_node, waku_core, waku_relay/protocol],
waku/node/health_monitor/[topic_health, health_status, protocol_health, health_report],
waku/requests/health_requests,
waku/requests/node_requests,
@ -43,7 +44,7 @@ proc waitForConnectionStatus(
if not await future.withTimeout(TestTimeout):
raiseAssert "Timeout waiting for status: " & $expected
finally:
EventConnectionStatusChange.dropListener(brokerCtx, handle)
await EventConnectionStatusChange.dropListener(brokerCtx, handle)
proc waitForShardHealthy(
brokerCtx: BrokerContext
@ -67,7 +68,7 @@ proc waitForShardHealthy(
else:
raiseAssert "Timeout waiting for shard health event"
finally:
EventShardTopicHealthChange.dropListener(brokerCtx, handle)
await EventShardTopicHealthChange.dropListener(brokerCtx, handle)
suite "LM API health checking":
var

8
tests/api/test_api_receive.nim
View File

@ -3,6 +3,7 @@
import std/[options, sequtils, net, sets]
import chronos, testutils/unittests, stew/byteutils
import libp2p/[peerid, peerinfo, crypto/crypto]
import brokers/broker_context
import ../testlib/[common, wakucore, wakunode, testasync]
import ../waku_archive/archive_utils
@ -11,7 +12,6 @@ import
waku/[
waku_node,
waku_core,
common/broker/broker_context,
events/message_events,
waku_relay/protocol,
waku_archive,
@ -52,8 +52,8 @@ proc newReceiveEventListenerManager(
return manager
proc teardown(manager: ReceiveEventListenerManager) =
MessageReceivedEvent.dropListener(manager.brokerCtx, manager.receivedListener)
proc teardown(manager: ReceiveEventListenerManager) {.async.} =
await MessageReceivedEvent.dropListener(manager.brokerCtx, manager.receivedListener)
proc waitForEvents(
manager: ReceiveEventListenerManager, timeout: Duration
@ -182,7 +182,7 @@ suite "Messaging API, Receive Service (store recovery)":
# listen before triggering store check
let eventManager = newReceiveEventListenerManager(subscriber.brokerCtx, 1)
defer:
eventManager.teardown()
await eventManager.teardown()
# trigger store check, should recover and deliver via MessageReceivedEvent
await subscriber.deliveryService.recvService.checkStore()

24
tests/api/test_api_send.nim
View File

@ -2,10 +2,10 @@
import std/strutils
import chronos, testutils/unittests, stew/byteutils, libp2p/[switch, peerinfo]
import brokers/broker_context
import ../testlib/[common, wakucore, wakunode, testasync]
import ../waku_archive/archive_utils
import
waku, waku/[waku_node, waku_core, waku_relay/protocol, common/broker/broker_context]
import waku, waku/[waku_node, waku_core, waku_relay/protocol]
import waku/factory/waku_conf
import tools/confutils/cli_args
@ -77,10 +77,12 @@ proc newSendEventListenerManager(brokerCtx: BrokerContext): SendEventListenerMan
return manager
proc teardown(manager: SendEventListenerManager) =
MessageSentEvent.dropListener(manager.brokerCtx, manager.sentListener)
MessageErrorEvent.dropListener(manager.brokerCtx, manager.errorListener)
MessagePropagatedEvent.dropListener(manager.brokerCtx, manager.propagatedListener)
proc teardown(manager: SendEventListenerManager) {.async.} =
await MessageSentEvent.dropListener(manager.brokerCtx, manager.sentListener)
await MessageErrorEvent.dropListener(manager.brokerCtx, manager.errorListener)
await MessagePropagatedEvent.dropListener(
manager.brokerCtx, manager.propagatedListener
)
proc waitForEvents(
manager: SendEventListenerManager, timeout: Duration
@ -270,7 +272,7 @@ suite "Waku API - Send":
let eventManager = newSendEventListenerManager(node.brokerCtx)
defer:
eventManager.teardown()
await eventManager.teardown()
let envelope = MessageEnvelope.init(
ContentTopic("/waku/2/default-content/proto"), "test payload"
@ -302,7 +304,7 @@ suite "Waku API - Send":
let eventManager = newSendEventListenerManager(node.brokerCtx)
defer:
eventManager.teardown()
await eventManager.teardown()
let envelope = MessageEnvelope.init(
ContentTopic("/waku/2/default-content/proto"), "test payload"
@ -332,7 +334,7 @@ suite "Waku API - Send":
let eventManager = newSendEventListenerManager(node.brokerCtx)
defer:
eventManager.teardown()
await eventManager.teardown()
let envelope = MessageEnvelope.init(
ContentTopic("/waku/2/default-content/proto"), "test payload"
@ -362,7 +364,7 @@ suite "Waku API - Send":
let eventManager = newSendEventListenerManager(node.brokerCtx)
defer:
eventManager.teardown()
await eventManager.teardown()
let envelope = MessageEnvelope.init(
ContentTopic("/waku/2/default-content/proto"), "test payload"
@ -416,7 +418,7 @@ suite "Waku API - Send":
let eventManager = newSendEventListenerManager(node.brokerCtx)
defer:
eventManager.teardown()
await eventManager.teardown()
let envelope = MessageEnvelope.init(
ContentTopic("/waku/2/default-content/proto"), "test payload"

42
tests/api/test_api_subscription.nim
View File

@ -3,6 +3,7 @@
import std/[strutils, sequtils, net, options, sets, tables]
import chronos, testutils/unittests, stew/byteutils
import libp2p/[peerid, peerinfo, multiaddress, crypto/crypto]
import brokers/broker_context
import ../testlib/[common, wakucore, wakunode, testasync]
import
@ -10,7 +11,6 @@ import
waku/[
waku_node,
waku_core,
common/broker/broker_context,
events/message_events,
waku_relay/protocol,
node/kernel_api/filter,
@ -51,8 +51,8 @@ proc newReceiveEventListenerManager(
return manager
proc teardown(manager: ReceiveEventListenerManager) =
MessageReceivedEvent.dropListener(manager.brokerCtx, manager.receivedListener)
proc teardown(manager: ReceiveEventListenerManager) {.async.} =
await MessageReceivedEvent.dropListener(manager.brokerCtx, manager.receivedListener)
proc waitForEvents(
manager: ReceiveEventListenerManager, timeout: Duration
@ -208,7 +208,7 @@ suite "Messaging API, SubscriptionManager":
let eventManager = newReceiveEventListenerManager(net.subscriber.brokerCtx, 1)
defer:
eventManager.teardown()
await eventManager.teardown()
discard (await net.publishToMesh(testTopic, "Hello, world!".toBytes())).expect(
"Publish failed"
@ -229,7 +229,7 @@ suite "Messaging API, SubscriptionManager":
let eventManager = newReceiveEventListenerManager(net.subscriber.brokerCtx, 1)
defer:
eventManager.teardown()
await eventManager.teardown()
discard (await net.publishToMesh(ignoredTopic, "Ghost Msg".toBytes())).expect(
"Publish failed"
@ -250,7 +250,7 @@ suite "Messaging API, SubscriptionManager":
let eventManager = newReceiveEventListenerManager(net.subscriber.brokerCtx, 1)
defer:
eventManager.teardown()
await eventManager.teardown()
discard (await net.publishToMesh(testTopic, "Should be dropped".toBytes())).expect(
"Publish failed"
@ -271,7 +271,7 @@ suite "Messaging API, SubscriptionManager":
let eventManager = newReceiveEventListenerManager(net.subscriber.brokerCtx, 1)
defer:
eventManager.teardown()
await eventManager.teardown()
net.subscriber.unsubscribe(topicA).expect("failed to unsub A")
@ -298,7 +298,7 @@ suite "Messaging API, SubscriptionManager":
let eventManager = newReceiveEventListenerManager(net.subscriber.brokerCtx, 1)
defer:
eventManager.teardown()
await eventManager.teardown()
discard (await net.publishToMesh(glitchTopic, "Ghost Msg".toBytes())).expect(
"Publish failed"
@ -322,7 +322,7 @@ suite "Messaging API, SubscriptionManager":
(await net.publishToMesh(testTopic, "Msg 1".toBytes())).expect("Pub 1 failed")
require await eventManager.waitForEvents(TestTimeout)
eventManager.teardown()
await eventManager.teardown()
# Unsubscribe and verify teardown
net.subscriber.unsubscribe(testTopic).expect("Unsub failed")
@ -332,7 +332,7 @@ suite "Messaging API, SubscriptionManager":
(await net.publishToMesh(testTopic, "Ghost".toBytes())).expect("Ghost pub failed")
check not await eventManager.waitForEvents(NegativeTestTimeout)
eventManager.teardown()
await eventManager.teardown()
# Resubscribe
(await net.subscriber.subscribe(testTopic)).expect("Resub failed")
@ -364,7 +364,7 @@ suite "Messaging API, SubscriptionManager":
let eventManager = newReceiveEventListenerManager(net.subscriber.brokerCtx, 2)
defer:
eventManager.teardown()
await eventManager.teardown()
discard (await net.publishToMesh(topicA, "Msg on Shard A".toBytes())).expect(
"Publish A failed"
@ -400,7 +400,7 @@ suite "Messaging API, SubscriptionManager":
# here we just give a chance for any messages that we don't expect to arrive
await sleepAsync(1.seconds)
eventManager.teardown()
await eventManager.teardown()
# weak check (but catches most bugs)
require eventManager.receivedMessages.len == expected.len
@ -451,7 +451,7 @@ suite "Messaging API, SubscriptionManager":
let eventManager = newReceiveEventListenerManager(net.subscriber.brokerCtx, 1)
defer:
eventManager.teardown()
await eventManager.teardown()
discard (await net.publishToMeshAfterEdgeReady(testTopic, "Hello, edge!".toBytes())).expect(
"Publish failed"
@ -472,7 +472,7 @@ suite "Messaging API, SubscriptionManager":
let eventManager = newReceiveEventListenerManager(net.subscriber.brokerCtx, 1)
defer:
eventManager.teardown()
await eventManager.teardown()
discard (await net.publishToMesh(ignoredTopic, "Ghost Msg".toBytes())).expect(
"Publish failed"
@ -493,7 +493,7 @@ suite "Messaging API, SubscriptionManager":
let eventManager = newReceiveEventListenerManager(net.subscriber.brokerCtx, 1)
defer:
eventManager.teardown()
await eventManager.teardown()
discard (await net.publishToMesh(testTopic, "Should be dropped".toBytes())).expect(
"Publish failed"
@ -517,7 +517,7 @@ suite "Messaging API, SubscriptionManager":
let eventManager = newReceiveEventListenerManager(net.subscriber.brokerCtx, 1)
defer:
eventManager.teardown()
await eventManager.teardown()
net.subscriber.unsubscribe(topicA).expect("failed to unsub A")
@ -546,7 +546,7 @@ suite "Messaging API, SubscriptionManager":
)
require await eventManager.waitForEvents(TestTimeout)
eventManager.teardown()
await eventManager.teardown()
net.subscriber.unsubscribe(testTopic).expect("Unsub failed")
eventManager = newReceiveEventListenerManager(net.subscriber.brokerCtx, 1)
@ -555,7 +555,7 @@ suite "Messaging API, SubscriptionManager":
(await net.publishToMesh(testTopic, "Ghost".toBytes())).expect("Ghost pub failed")
check not await eventManager.waitForEvents(NegativeTestTimeout)
eventManager.teardown()
await eventManager.teardown()
(await net.subscriber.subscribe(testTopic)).expect("Resub failed")
eventManager = newReceiveEventListenerManager(net.subscriber.brokerCtx, 1)
@ -653,7 +653,7 @@ suite "Messaging API, SubscriptionManager":
require await eventManager.waitForEvents(TestTimeout)
check eventManager.receivedMessages[0].payload == "Before failover".toBytes()
eventManager.teardown()
await eventManager.teardown()
# Disconnect meshBuddy from edge (keeps relay mesh alive for publishing)
await subscriber.node.disconnectNode(meshBuddyPeerInfo)
@ -678,7 +678,7 @@ suite "Messaging API, SubscriptionManager":
require await eventManager.waitForEvents(TestTimeout)
check eventManager.receivedMessages[0].payload == "After failover".toBytes()
eventManager.teardown()
await eventManager.teardown()
(await subscriber.stop()).expect("Failed to stop subscriber")
await meshBuddy.stop()
@ -801,7 +801,7 @@ suite "Messaging API, SubscriptionManager":
require await eventManager.waitForEvents(TestTimeout)
check eventManager.receivedMessages[0].payload == "After replacement".toBytes()
eventManager.teardown()
await eventManager.teardown()
(await subscriber.stop()).expect("Failed to stop subscriber")
await sparePeer.stop()

5
tests/common/test_all.nim
View File

@ -8,7 +8,4 @@ import
./test_parse_size,
./test_requestratelimiter,
./test_ratelimit_setting,
./test_timed_map,
./test_event_broker,
./test_request_broker,
./test_multi_request_broker
./test_timed_map

201
tests/common/test_event_broker.nim
View File

@ -1,201 +0,0 @@
import chronos
import std/sequtils
import testutils/unittests
import waku/common/broker/event_broker
type ExternalDefinedEventType = object
label*: string
EventBroker:
type IntEvent = int
EventBroker:
type ExternalAliasEvent = distinct ExternalDefinedEventType
EventBroker:
type SampleEvent = object
value*: int
label*: string
EventBroker:
type BinaryEvent = object
flag*: bool
EventBroker:
type RefEvent = ref object
payload*: seq[int]
template waitForListeners() =
waitFor sleepAsync(1.milliseconds)
suite "EventBroker":
test "delivers events to all listeners":
var seen: seq[(int, string)] = @[]
discard SampleEvent.listen(
proc(evt: SampleEvent): Future[void] {.async: (raises: []).} =
seen.add((evt.value, evt.label))
)
discard SampleEvent.listen(
proc(evt: SampleEvent): Future[void] {.async: (raises: []).} =
seen.add((evt.value * 2, evt.label & "!"))
)
let evt = SampleEvent(value: 5, label: "hi")
SampleEvent.emit(evt)
waitForListeners()
check seen.len == 2
check seen.anyIt(it == (5, "hi"))
check seen.anyIt(it == (10, "hi!"))
SampleEvent.dropAllListeners()
test "forget removes a single listener":
var counter = 0
let handleA = SampleEvent.listen(
proc(evt: SampleEvent): Future[void] {.async: (raises: []).} =
inc counter
)
let handleB = SampleEvent.listen(
proc(evt: SampleEvent): Future[void] {.async: (raises: []).} =
inc(counter, 2)
)
SampleEvent.dropListener(handleA.get())
let eventVal = SampleEvent(value: 1, label: "one")
SampleEvent.emit(eventVal)
waitForListeners()
check counter == 2
SampleEvent.dropAllListeners()
test "forgetAll clears every listener":
var triggered = false
let handle1 = SampleEvent.listen(
proc(evt: SampleEvent): Future[void] {.async: (raises: []).} =
triggered = true
)
let handle2 = SampleEvent.listen(
proc(evt: SampleEvent): Future[void] {.async: (raises: []).} =
discard
)
SampleEvent.dropAllListeners()
SampleEvent.emit(42, "noop")
SampleEvent.emit(label = "noop", value = 42)
waitForListeners()
check not triggered
let freshHandle = SampleEvent.listen(
proc(evt: SampleEvent): Future[void] {.async: (raises: []).} =
discard
)
check freshHandle.get().id > 0'u64
SampleEvent.dropListener(freshHandle.get())
test "broker helpers operate via typedesc":
var toggles: seq[bool] = @[]
let handle = BinaryEvent.listen(
proc(evt: BinaryEvent): Future[void] {.async: (raises: []).} =
toggles.add(evt.flag)
)
BinaryEvent(flag: true).emit()
waitForListeners()
let binaryEvent = BinaryEvent(flag: false)
BinaryEvent.emit(binaryEvent)
waitForListeners()
check toggles == @[true, false]
BinaryEvent.dropAllListeners()
test "ref typed event":
var counter: int = 0
let handle = RefEvent.listen(
proc(evt: RefEvent): Future[void] {.async: (raises: []).} =
for n in evt.payload:
counter += n
)
RefEvent(payload: @[1, 2, 3]).emit()
waitForListeners()
RefEvent.emit(payload = @[4, 5, 6])
waitForListeners()
check counter == 21 # 1+2+3 + 4+5+6
RefEvent.dropAllListeners()
test "supports BrokerContext-scoped listeners":
SampleEvent.dropAllListeners()
let ctxA = NewBrokerContext()
let ctxB = NewBrokerContext()
var seenA: seq[int] = @[]
var seenB: seq[int] = @[]
discard SampleEvent.listen(
ctxA,
proc(evt: SampleEvent): Future[void] {.async: (raises: []).} =
seenA.add(evt.value),
)
discard SampleEvent.listen(
ctxB,
proc(evt: SampleEvent): Future[void] {.async: (raises: []).} =
seenB.add(evt.value),
)
SampleEvent.emit(ctxA, SampleEvent(value: 1, label: "a"))
SampleEvent.emit(ctxB, SampleEvent(value: 2, label: "b"))
waitForListeners()
check seenA == @[1]
check seenB == @[2]
SampleEvent.dropAllListeners(ctxA)
SampleEvent.emit(ctxA, SampleEvent(value: 3, label: "a2"))
SampleEvent.emit(ctxB, SampleEvent(value: 4, label: "b2"))
waitForListeners()
check seenA == @[1]
check seenB == @[2, 4]
SampleEvent.dropAllListeners(ctxB)
test "supports non-object event types (auto-distinct)":
var seen: seq[int] = @[]
discard IntEvent.listen(
proc(evt: IntEvent): Future[void] {.async: (raises: []).} =
seen.add(int(evt))
)
IntEvent.emit(IntEvent(42))
waitForListeners()
check seen == @[42]
IntEvent.dropAllListeners()
test "supports externally-defined type aliases (auto-distinct)":
var seen: seq[string] = @[]
discard ExternalAliasEvent.listen(
proc(evt: ExternalAliasEvent): Future[void] {.async: (raises: []).} =
let base = ExternalDefinedEventType(evt)
seen.add(base.label)
)
ExternalAliasEvent.emit(ExternalAliasEvent(ExternalDefinedEventType(label: "x")))
waitForListeners()
check seen == @["x"]
ExternalAliasEvent.dropAllListeners()

343
tests/common/test_multi_request_broker.nim
View File

@ -1,343 +0,0 @@
{.used.}
import testutils/unittests
import chronos
import std/sequtils
import std/strutils
import waku/common/broker/multi_request_broker
MultiRequestBroker:
type NoArgResponse = object
label*: string
proc signatureFetch*(): Future[Result[NoArgResponse, string]] {.async.}
MultiRequestBroker:
type ArgResponse = object
id*: string
proc signatureFetch*(
suffix: string, numsuffix: int
): Future[Result[ArgResponse, string]] {.async.}
MultiRequestBroker:
type DualResponse = ref object
note*: string
suffix*: string
proc signatureBase*(): Future[Result[DualResponse, string]] {.async.}
proc signatureWithInput*(
suffix: string
): Future[Result[DualResponse, string]] {.async.}
type ExternalBaseType = string
MultiRequestBroker:
type NativeIntResponse = int
proc signatureFetch*(): Future[Result[NativeIntResponse, string]] {.async.}
MultiRequestBroker:
type ExternalAliasResponse = ExternalBaseType
proc signatureFetch*(): Future[Result[ExternalAliasResponse, string]] {.async.}
MultiRequestBroker:
type AlreadyDistinctResponse = distinct int
proc signatureFetch*(): Future[Result[AlreadyDistinctResponse, string]] {.async.}
suite "MultiRequestBroker":
test "aggregates zero-argument providers":
discard NoArgResponse.setProvider(
proc(): Future[Result[NoArgResponse, string]] {.async.} =
ok(NoArgResponse(label: "one"))
)
discard NoArgResponse.setProvider(
proc(): Future[Result[NoArgResponse, string]] {.async.} =
discard catch:
await sleepAsync(1.milliseconds)
ok(NoArgResponse(label: "two"))
)
let responses = waitFor NoArgResponse.request()
check responses.get().len == 2
check responses.get().anyIt(it.label == "one")
check responses.get().anyIt(it.label == "two")
NoArgResponse.clearProviders()
test "aggregates argument providers":
discard ArgResponse.setProvider(
proc(suffix: string, num: int): Future[Result[ArgResponse, string]] {.async.} =
ok(ArgResponse(id: suffix & "-a-" & $num))
)
discard ArgResponse.setProvider(
proc(suffix: string, num: int): Future[Result[ArgResponse, string]] {.async.} =
ok(ArgResponse(id: suffix & "-b-" & $num))
)
let keyed = waitFor ArgResponse.request("topic", 1)
check keyed.get().len == 2
check keyed.get().anyIt(it.id == "topic-a-1")
check keyed.get().anyIt(it.id == "topic-b-1")
ArgResponse.clearProviders()
test "clearProviders resets both provider lists":
discard DualResponse.setProvider(
proc(): Future[Result[DualResponse, string]] {.async.} =
ok(DualResponse(note: "base", suffix: ""))
)
discard DualResponse.setProvider(
proc(suffix: string): Future[Result[DualResponse, string]] {.async.} =
ok(DualResponse(note: "base" & suffix, suffix: suffix))
)
let noArgs = waitFor DualResponse.request()
check noArgs.get().len == 1
let param = waitFor DualResponse.request("-extra")
check param.get().len == 1
check param.get()[0].suffix == "-extra"
DualResponse.clearProviders()
let emptyNoArgs = waitFor DualResponse.request()
check emptyNoArgs.get().len == 0
let emptyWithArgs = waitFor DualResponse.request("-extra")
check emptyWithArgs.get().len == 0
test "request returns empty seq when no providers registered":
let empty = waitFor NoArgResponse.request()
check empty.get().len == 0
test "failed providers will fail the request":
NoArgResponse.clearProviders()
discard NoArgResponse.setProvider(
proc(): Future[Result[NoArgResponse, string]] {.async.} =
err("boom")
)
discard NoArgResponse.setProvider(
proc(): Future[Result[NoArgResponse, string]] {.async.} =
ok(NoArgResponse(label: "survivor"))
)
let filtered = waitFor NoArgResponse.request()
check filtered.isErr()
NoArgResponse.clearProviders()
test "deduplicates identical zero-argument providers":
NoArgResponse.clearProviders()
var invocations = 0
let sharedHandler = proc(): Future[Result[NoArgResponse, string]] {.async.} =
inc invocations
ok(NoArgResponse(label: "dup"))
let first = NoArgResponse.setProvider(sharedHandler)
let second = NoArgResponse.setProvider(sharedHandler)
check first.get().id == second.get().id
check first.get().kind == second.get().kind
let dupResponses = waitFor NoArgResponse.request()
check dupResponses.get().len == 1
check invocations == 1
NoArgResponse.clearProviders()
test "removeProvider deletes registered handlers":
var removedCalled = false
var keptCalled = false
let removable = NoArgResponse.setProvider(
proc(): Future[Result[NoArgResponse, string]] {.async.} =
removedCalled = true
ok(NoArgResponse(label: "removed"))
)
discard NoArgResponse.setProvider(
proc(): Future[Result[NoArgResponse, string]] {.async.} =
keptCalled = true
ok(NoArgResponse(label: "kept"))
)
NoArgResponse.removeProvider(removable.get())
let afterRemoval = (waitFor NoArgResponse.request()).valueOr:
assert false, "request failed"
@[]
check afterRemoval.len == 1
check afterRemoval[0].label == "kept"
check not removedCalled
check keptCalled
NoArgResponse.clearProviders()
test "removeProvider works for argument signatures":
var invoked: seq[string] = @[]
discard ArgResponse.setProvider(
proc(suffix: string, num: int): Future[Result[ArgResponse, string]] {.async.} =
invoked.add("first" & suffix)
ok(ArgResponse(id: suffix & "-one-" & $num))
)
let handle = ArgResponse.setProvider(
proc(suffix: string, num: int): Future[Result[ArgResponse, string]] {.async.} =
invoked.add("second" & suffix)
ok(ArgResponse(id: suffix & "-two-" & $num))
)
ArgResponse.removeProvider(handle.get())
let single = (waitFor ArgResponse.request("topic", 1)).valueOr:
assert false, "request failed"
@[]
check single.len == 1
check single[0].id == "topic-one-1"
check invoked == @["firsttopic"]
ArgResponse.clearProviders()
test "catches exception from providers and report error":
let firstHandler = NoArgResponse.setProvider(
proc(): Future[Result[NoArgResponse, string]] {.async.} =
raise newException(ValueError, "first handler raised")
)
discard NoArgResponse.setProvider(
proc(): Future[Result[NoArgResponse, string]] {.async.} =
ok(NoArgResponse(label: "just ok"))
)
let afterException = waitFor NoArgResponse.request()
check afterException.isErr()
check afterException.error().contains("first handler raised")
NoArgResponse.clearProviders()
test "ref providers returning nil fail request":
DualResponse.clearProviders()
test "supports native request types":
NativeIntResponse.clearProviders()
discard NativeIntResponse.setProvider(
proc(): Future[Result[NativeIntResponse, string]] {.async.} =
ok(NativeIntResponse(1))
)
discard NativeIntResponse.setProvider(
proc(): Future[Result[NativeIntResponse, string]] {.async.} =
ok(NativeIntResponse(2))
)
let res = waitFor NativeIntResponse.request()
check res.isOk()
check res.get().len == 2
check res.get().anyIt(int(it) == 1)
check res.get().anyIt(int(it) == 2)
NativeIntResponse.clearProviders()
test "supports external request types":
ExternalAliasResponse.clearProviders()
discard ExternalAliasResponse.setProvider(
proc(): Future[Result[ExternalAliasResponse, string]] {.async.} =
ok(ExternalAliasResponse("hello"))
)
let res = waitFor ExternalAliasResponse.request()
check res.isOk()
check res.get().len == 1
check ExternalBaseType(res.get()[0]) == "hello"
ExternalAliasResponse.clearProviders()
test "supports already-distinct request types":
AlreadyDistinctResponse.clearProviders()
discard AlreadyDistinctResponse.setProvider(
proc(): Future[Result[AlreadyDistinctResponse, string]] {.async.} =
ok(AlreadyDistinctResponse(7))
)
let res = waitFor AlreadyDistinctResponse.request()
check res.isOk()
check res.get().len == 1
check int(res.get()[0]) == 7
AlreadyDistinctResponse.clearProviders()
test "context-aware providers are isolated":
NoArgResponse.clearProviders()
let ctxA = NewBrokerContext()
let ctxB = NewBrokerContext()
discard NoArgResponse.setProvider(
ctxA,
proc(): Future[Result[NoArgResponse, string]] {.async.} =
ok(NoArgResponse(label: "a")),
)
discard NoArgResponse.setProvider(
ctxB,
proc(): Future[Result[NoArgResponse, string]] {.async.} =
ok(NoArgResponse(label: "b")),
)
let resA = waitFor NoArgResponse.request(ctxA)
check resA.isOk()
check resA.get().len == 1
check resA.get()[0].label == "a"
let resB = waitFor NoArgResponse.request(ctxB)
check resB.isOk()
check resB.get().len == 1
check resB.get()[0].label == "b"
let resDefault = waitFor NoArgResponse.request()
check resDefault.isOk()
check resDefault.get().len == 0
NoArgResponse.clearProviders(ctxA)
let clearedA = waitFor NoArgResponse.request(ctxA)
check clearedA.isOk()
check clearedA.get().len == 0
let stillB = waitFor NoArgResponse.request(ctxB)
check stillB.isOk()
check stillB.get().len == 1
check stillB.get()[0].label == "b"
NoArgResponse.clearProviders(ctxB)
discard DualResponse.setProvider(
proc(): Future[Result[DualResponse, string]] {.async.} =
let nilResponse: DualResponse = nil
ok(nilResponse)
)
let zeroArg = waitFor DualResponse.request()
check zeroArg.isErr()
DualResponse.clearProviders()
discard DualResponse.setProvider(
proc(suffix: string): Future[Result[DualResponse, string]] {.async.} =
let nilResponse: DualResponse = nil
ok(nilResponse)
)
let withInput = waitFor DualResponse.request("-extra")
check withInput.isErr()
DualResponse.clearProviders()

675
tests/common/test_request_broker.nim
View File

@ -1,675 +0,0 @@
{.used.}
import testutils/unittests
import chronos
import std/strutils
import waku/common/broker/request_broker
## ---------------------------------------------------------------------------
## Async-mode brokers + tests
## ---------------------------------------------------------------------------
RequestBroker:
type SimpleResponse = object
value*: string
proc signatureFetch*(): Future[Result[SimpleResponse, string]] {.async.}
RequestBroker:
type KeyedResponse = object
key*: string
payload*: string
proc signatureFetchWithKey*(
key: string, subKey: int
): Future[Result[KeyedResponse, string]] {.async.}
RequestBroker:
type DualResponse = object
note*: string
count*: int
proc signatureNoInput*(): Future[Result[DualResponse, string]] {.async.}
proc signatureWithInput*(
suffix: string
): Future[Result[DualResponse, string]] {.async.}
RequestBroker(async):
type ImplicitResponse = ref object
note*: string
static:
doAssert typeof(SimpleResponse.request()) is Future[Result[SimpleResponse, string]]
suite "RequestBroker macro (async mode)":
test "serves zero-argument providers":
check SimpleResponse
.setProvider(
proc(): Future[Result[SimpleResponse, string]] {.async.} =
ok(SimpleResponse(value: "hi"))
)
.isOk()
let res = waitFor SimpleResponse.request()
check res.isOk()
check res.value.value == "hi"
SimpleResponse.clearProvider()
test "zero-argument request errors when unset":
let res = waitFor SimpleResponse.request()
check res.isErr()
check res.error.contains("no zero-arg provider")
test "serves input-based providers":
var seen: seq[string] = @[]
check KeyedResponse
.setProvider(
proc(
key: string, subKey: int
): Future[Result[KeyedResponse, string]] {.async.} =
seen.add(key)
ok(KeyedResponse(key: key, payload: key & "-payload+" & $subKey))
)
.isOk()
let res = waitFor KeyedResponse.request("topic", 1)
check res.isOk()
check res.value.key == "topic"
check res.value.payload == "topic-payload+1"
check seen == @["topic"]
KeyedResponse.clearProvider()
test "catches provider exception":
check KeyedResponse
.setProvider(
proc(
key: string, subKey: int
): Future[Result[KeyedResponse, string]] {.async.} =
raise newException(ValueError, "simulated failure")
)
.isOk()
let res = waitFor KeyedResponse.request("neglected", 11)
check res.isErr()
check res.error.contains("simulated failure")
KeyedResponse.clearProvider()
test "input request errors when unset":
let res = waitFor KeyedResponse.request("foo", 2)
check res.isErr()
check res.error.contains("input signature")
test "supports both provider types simultaneously":
check DualResponse
.setProvider(
proc(): Future[Result[DualResponse, string]] {.async.} =
ok(DualResponse(note: "base", count: 1))
)
.isOk()
check DualResponse
.setProvider(
proc(suffix: string): Future[Result[DualResponse, string]] {.async.} =
ok(DualResponse(note: "base" & suffix, count: suffix.len))
)
.isOk()
let noInput = waitFor DualResponse.request()
check noInput.isOk()
check noInput.value.note == "base"
let withInput = waitFor DualResponse.request("-extra")
check withInput.isOk()
check withInput.value.note == "base-extra"
check withInput.value.count == 6
DualResponse.clearProvider()
test "clearProvider resets both entries":
check DualResponse
.setProvider(
proc(): Future[Result[DualResponse, string]] {.async.} =
ok(DualResponse(note: "temp", count: 0))
)
.isOk()
DualResponse.clearProvider()
let res = waitFor DualResponse.request()
check res.isErr()
test "implicit zero-argument provider works by default":
check ImplicitResponse
.setProvider(
proc(): Future[Result[ImplicitResponse, string]] {.async.} =
ok(ImplicitResponse(note: "auto"))
)
.isOk()
let res = waitFor ImplicitResponse.request()
check res.isOk()
ImplicitResponse.clearProvider()
check res.value.note == "auto"
test "implicit zero-argument request errors when unset":
let res = waitFor ImplicitResponse.request()
check res.isErr()
check res.error.contains("no zero-arg provider")
test "no provider override":
check DualResponse
.setProvider(
proc(): Future[Result[DualResponse, string]] {.async.} =
ok(DualResponse(note: "base", count: 1))
)
.isOk()
check DualResponse
.setProvider(
proc(suffix: string): Future[Result[DualResponse, string]] {.async.} =
ok(DualResponse(note: "base" & suffix, count: suffix.len))
)
.isOk()
let overrideProc = proc(): Future[Result[DualResponse, string]] {.async.} =
ok(DualResponse(note: "something else", count: 1))
check DualResponse.setProvider(overrideProc).isErr()
let noInput = waitFor DualResponse.request()
check noInput.isOk()
check noInput.value.note == "base"
let stillResponse = waitFor DualResponse.request(" still works")
check stillResponse.isOk()
check stillResponse.value.note.contains("base still works")
DualResponse.clearProvider()
let noResponse = waitFor DualResponse.request()
check noResponse.isErr()
check noResponse.error.contains("no zero-arg provider")
let noResponseArg = waitFor DualResponse.request("Should not work")
check noResponseArg.isErr()
check noResponseArg.error.contains("no provider")
check DualResponse.setProvider(overrideProc).isOk()
let nowSuccWithOverride = waitFor DualResponse.request()
check nowSuccWithOverride.isOk()
check nowSuccWithOverride.value.note == "something else"
check nowSuccWithOverride.value.count == 1
DualResponse.clearProvider()
test "supports keyed providers (async, zero-arg)":
SimpleResponse.clearProvider()
check SimpleResponse
.setProvider(
proc(): Future[Result[SimpleResponse, string]] {.async.} =
ok(SimpleResponse(value: "default"))
)
.isOk()
check SimpleResponse
.setProvider(
BrokerContext(0x11111111'u32),
proc(): Future[Result[SimpleResponse, string]] {.async.} =
ok(SimpleResponse(value: "one")),
)
.isOk()
check SimpleResponse
.setProvider(
BrokerContext(0x22222222'u32),
proc(): Future[Result[SimpleResponse, string]] {.async.} =
ok(SimpleResponse(value: "two")),
)
.isOk()
let defaultRes = waitFor SimpleResponse.request()
check defaultRes.isOk()
check defaultRes.value.value == "default"
let res1 = waitFor SimpleResponse.request(BrokerContext(0x11111111'u32))
check res1.isOk()
check res1.value.value == "one"
let res2 = waitFor SimpleResponse.request(BrokerContext(0x22222222'u32))
check res2.isOk()
check res2.value.value == "two"
let missing = waitFor SimpleResponse.request(BrokerContext(0x33333333'u32))
check missing.isErr()
check missing.error.contains("no provider registered for broker context")
check SimpleResponse
.setProvider(
BrokerContext(0x11111111'u32),
proc(): Future[Result[SimpleResponse, string]] {.async.} =
ok(SimpleResponse(value: "dup")),
)
.isErr()
SimpleResponse.clearProvider()
test "supports keyed providers (async, with args)":
KeyedResponse.clearProvider()
check KeyedResponse
.setProvider(
proc(
key: string, subKey: int
): Future[Result[KeyedResponse, string]] {.async.} =
ok(KeyedResponse(key: "default-" & key, payload: $subKey))
)
.isOk()
check KeyedResponse
.setProvider(
BrokerContext(0xABCDEF01'u32),
proc(
key: string, subKey: int
): Future[Result[KeyedResponse, string]] {.async.} =
ok(KeyedResponse(key: "k1-" & key, payload: "p" & $subKey)),
)
.isOk()
check KeyedResponse
.setProvider(
BrokerContext(0xABCDEF02'u32),
proc(
key: string, subKey: int
): Future[Result[KeyedResponse, string]] {.async.} =
ok(KeyedResponse(key: "k2-" & key, payload: "q" & $subKey)),
)
.isOk()
let d = waitFor KeyedResponse.request("topic", 7)
check d.isOk()
check d.value.key == "default-topic"
let k1 = waitFor KeyedResponse.request(BrokerContext(0xABCDEF01'u32), "topic", 7)
check k1.isOk()
check k1.value.key == "k1-topic"
check k1.value.payload == "p7"
let k2 = waitFor KeyedResponse.request(BrokerContext(0xABCDEF02'u32), "topic", 7)
check k2.isOk()
check k2.value.key == "k2-topic"
check k2.value.payload == "q7"
let miss = waitFor KeyedResponse.request(BrokerContext(0xDEADBEEF'u32), "topic", 7)
check miss.isErr()
check miss.error.contains("no provider registered for broker context")
KeyedResponse.clearProvider()
## ---------------------------------------------------------------------------
## Sync-mode brokers + tests
## ---------------------------------------------------------------------------
RequestBroker(sync):
type SimpleResponseSync = object
value*: string
proc signatureFetch*(): Result[SimpleResponseSync, string]
RequestBroker(sync):
type KeyedResponseSync = object
key*: string
payload*: string
proc signatureFetchWithKey*(
key: string, subKey: int
): Result[KeyedResponseSync, string]
RequestBroker(sync):
type DualResponseSync = object
note*: string
count*: int
proc signatureNoInput*(): Result[DualResponseSync, string]
proc signatureWithInput*(suffix: string): Result[DualResponseSync, string]
RequestBroker(sync):
type ImplicitResponseSync = ref object
note*: string
static:
doAssert typeof(SimpleResponseSync.request()) is Result[SimpleResponseSync, string]
doAssert not (
typeof(SimpleResponseSync.request()) is Future[Result[SimpleResponseSync, string]]
)
doAssert typeof(KeyedResponseSync.request("topic", 1)) is
Result[KeyedResponseSync, string]
suite "RequestBroker macro (sync mode)":
test "serves zero-argument providers (sync)":
check SimpleResponseSync
.setProvider(
proc(): Result[SimpleResponseSync, string] =
ok(SimpleResponseSync(value: "hi"))
)
.isOk()
let res = SimpleResponseSync.request()
check res.isOk()
check res.value.value == "hi"
SimpleResponseSync.clearProvider()
test "zero-argument request errors when unset (sync)":
let res = SimpleResponseSync.request()
check res.isErr()
check res.error.contains("no zero-arg provider")
test "serves input-based providers (sync)":
var seen: seq[string] = @[]
check KeyedResponseSync
.setProvider(
proc(key: string, subKey: int): Result[KeyedResponseSync, string] =
seen.add(key)
ok(KeyedResponseSync(key: key, payload: key & "-payload+" & $subKey))
)
.isOk()
let res = KeyedResponseSync.request("topic", 1)
check res.isOk()
check res.value.key == "topic"
check res.value.payload == "topic-payload+1"
check seen == @["topic"]
KeyedResponseSync.clearProvider()
test "catches provider exception (sync)":
check KeyedResponseSync
.setProvider(
proc(key: string, subKey: int): Result[KeyedResponseSync, string] =
raise newException(ValueError, "simulated failure")
)
.isOk()
let res = KeyedResponseSync.request("neglected", 11)
check res.isErr()
check res.error.contains("simulated failure")
KeyedResponseSync.clearProvider()
test "input request errors when unset (sync)":
let res = KeyedResponseSync.request("foo", 2)
check res.isErr()
check res.error.contains("input signature")
test "supports both provider types simultaneously (sync)":
check DualResponseSync
.setProvider(
proc(): Result[DualResponseSync, string] =
ok(DualResponseSync(note: "base", count: 1))
)
.isOk()
check DualResponseSync
.setProvider(
proc(suffix: string): Result[DualResponseSync, string] =
ok(DualResponseSync(note: "base" & suffix, count: suffix.len))
)
.isOk()
let noInput = DualResponseSync.request()
check noInput.isOk()
check noInput.value.note == "base"
let withInput = DualResponseSync.request("-extra")
check withInput.isOk()
check withInput.value.note == "base-extra"
check withInput.value.count == 6
DualResponseSync.clearProvider()
test "clearProvider resets both entries (sync)":
check DualResponseSync
.setProvider(
proc(): Result[DualResponseSync, string] =
ok(DualResponseSync(note: "temp", count: 0))
)
.isOk()
DualResponseSync.clearProvider()
let res = DualResponseSync.request()
check res.isErr()
test "implicit zero-argument provider works by default (sync)":
check ImplicitResponseSync
.setProvider(
proc(): Result[ImplicitResponseSync, string] =
ok(ImplicitResponseSync(note: "auto"))
)
.isOk()
let res = ImplicitResponseSync.request()
check res.isOk()
ImplicitResponseSync.clearProvider()
check res.value.note == "auto"
test "implicit zero-argument request errors when unset (sync)":
let res = ImplicitResponseSync.request()
check res.isErr()
check res.error.contains("no zero-arg provider")
test "implicit zero-argument provider raises error (sync)":
check ImplicitResponseSync
.setProvider(
proc(): Result[ImplicitResponseSync, string] =
raise newException(ValueError, "simulated failure")
)
.isOk()
let res = ImplicitResponseSync.request()
check res.isErr()
check res.error.contains("simulated failure")
ImplicitResponseSync.clearProvider()
test "supports keyed providers (sync, zero-arg)":
SimpleResponseSync.clearProvider()
check SimpleResponseSync
.setProvider(
proc(): Result[SimpleResponseSync, string] =
ok(SimpleResponseSync(value: "default"))
)
.isOk()
check SimpleResponseSync
.setProvider(
BrokerContext(0x10101010'u32),
proc(): Result[SimpleResponseSync, string] =
ok(SimpleResponseSync(value: "ten")),
)
.isOk()
let defaultRes = SimpleResponseSync.request()
check defaultRes.isOk()
check defaultRes.value.value == "default"
let keyedRes = SimpleResponseSync.request(BrokerContext(0x10101010'u32))
check keyedRes.isOk()
check keyedRes.value.value == "ten"
let miss = SimpleResponseSync.request(BrokerContext(0x20202020'u32))
check miss.isErr()
check miss.error.contains("no provider registered for broker context")
SimpleResponseSync.clearProvider()
test "supports keyed providers (sync, with args)":
KeyedResponseSync.clearProvider()
check KeyedResponseSync
.setProvider(
proc(key: string, subKey: int): Result[KeyedResponseSync, string] =
ok(KeyedResponseSync(key: "default-" & key, payload: $subKey))
)
.isOk()
check KeyedResponseSync
.setProvider(
BrokerContext(0xA0A0A0A0'u32),
proc(key: string, subKey: int): Result[KeyedResponseSync, string] =
ok(KeyedResponseSync(key: "k-" & key, payload: "p" & $subKey)),
)
.isOk()
let d = KeyedResponseSync.request("topic", 2)
check d.isOk()
check d.value.key == "default-topic"
let keyed = KeyedResponseSync.request(BrokerContext(0xA0A0A0A0'u32), "topic", 2)
check keyed.isOk()
check keyed.value.key == "k-topic"
check keyed.value.payload == "p2"
let miss = KeyedResponseSync.request(BrokerContext(0xB0B0B0B0'u32), "topic", 2)
check miss.isErr()
check miss.error.contains("no provider registered for broker context")
KeyedResponseSync.clearProvider()
## ---------------------------------------------------------------------------
## POD / external type brokers + tests (distinct/alias behavior)
## ---------------------------------------------------------------------------
type ExternalDefinedTypeAsync = object
label*: string
type ExternalDefinedTypeSync = object
label*: string
type ExternalDefinedTypeShared = object
label*: string
RequestBroker:
type PodResponse = int
proc signatureFetch*(): Future[Result[PodResponse, string]] {.async.}
RequestBroker:
type ExternalAliasedResponse = ExternalDefinedTypeAsync
proc signatureFetch*(): Future[Result[ExternalAliasedResponse, string]] {.async.}
RequestBroker(sync):
type ExternalAliasedResponseSync = ExternalDefinedTypeSync
proc signatureFetch*(): Result[ExternalAliasedResponseSync, string]
RequestBroker(sync):
type DistinctStringResponseA = distinct string
RequestBroker(sync):
type DistinctStringResponseB = distinct string
RequestBroker(sync):
type ExternalDistinctResponseA = distinct ExternalDefinedTypeShared
RequestBroker(sync):
type ExternalDistinctResponseB = distinct ExternalDefinedTypeShared
suite "RequestBroker macro (POD/external types)":
test "supports non-object response types (async)":
check PodResponse
.setProvider(
proc(): Future[Result[PodResponse, string]] {.async.} =
ok(PodResponse(123))
)
.isOk()
let res = waitFor PodResponse.request()
check res.isOk()
check int(res.value) == 123
PodResponse.clearProvider()
test "supports aliased external types (async)":
check ExternalAliasedResponse
.setProvider(
proc(): Future[Result[ExternalAliasedResponse, string]] {.async.} =
ok(ExternalAliasedResponse(ExternalDefinedTypeAsync(label: "ext")))
)
.isOk()
let res = waitFor ExternalAliasedResponse.request()
check res.isOk()
check ExternalDefinedTypeAsync(res.value).label == "ext"
ExternalAliasedResponse.clearProvider()
test "supports aliased external types (sync)":
check ExternalAliasedResponseSync
.setProvider(
proc(): Result[ExternalAliasedResponseSync, string] =
ok(ExternalAliasedResponseSync(ExternalDefinedTypeSync(label: "ext")))
)
.isOk()
let res = ExternalAliasedResponseSync.request()
check res.isOk()
check ExternalDefinedTypeSync(res.value).label == "ext"
ExternalAliasedResponseSync.clearProvider()
test "distinct response types avoid overload ambiguity (sync)":
check DistinctStringResponseA
.setProvider(
proc(): Result[DistinctStringResponseA, string] =
ok(DistinctStringResponseA("a"))
)
.isOk()
check DistinctStringResponseB
.setProvider(
proc(): Result[DistinctStringResponseB, string] =
ok(DistinctStringResponseB("b"))
)
.isOk()
check ExternalDistinctResponseA
.setProvider(
proc(): Result[ExternalDistinctResponseA, string] =
ok(ExternalDistinctResponseA(ExternalDefinedTypeShared(label: "ea")))
)
.isOk()
check ExternalDistinctResponseB
.setProvider(
proc(): Result[ExternalDistinctResponseB, string] =
ok(ExternalDistinctResponseB(ExternalDefinedTypeShared(label: "eb")))
)
.isOk()
let resA = DistinctStringResponseA.request()
let resB = DistinctStringResponseB.request()
check resA.isOk()
check resB.isOk()
check string(resA.value) == "a"
check string(resB.value) == "b"
let resEA = ExternalDistinctResponseA.request()
let resEB = ExternalDistinctResponseB.request()
check resEA.isOk()
check resEB.isOk()
check ExternalDefinedTypeShared(resEA.value).label == "ea"
check ExternalDefinedTypeShared(resEB.value).label == "eb"
DistinctStringResponseA.clearProvider()
DistinctStringResponseB.clearProvider()
ExternalDistinctResponseA.clearProvider()
ExternalDistinctResponseB.clearProvider()

8
tests/node/test_wakunode_health_monitor.nim
View File

@ -2,6 +2,7 @@
import
std/[json, options, sequtils, strutils, tables], testutils/unittests, chronos, results
import brokers/broker_context
import
waku/[
@ -23,7 +24,6 @@ import
events/health_events,
events/peer_events,
waku_archive,
common/broker/broker_context,
]
import ../testlib/[wakunode, wakucore], ../waku_archive/archive_utils
@ -277,7 +277,7 @@ suite "Health Monitor - events":
await nodeA.connectToNodes(@[nodeB.switch.peerInfo.toRemotePeerInfo()])
let metadataOk = await metadataFut.withTimeout(TestConnectivityTimeLimit)
WakuPeerEvent.dropListener(nodeA.brokerCtx, metadataLis)
await WakuPeerEvent.dropListener(nodeA.brokerCtx, metadataLis)
require metadataOk
let connectTimeLimit = Moment.now() + TestConnectivityTimeLimit
@ -380,7 +380,7 @@ suite "Health Monitor - events":
await nodeA.connectToNodes(@[nodeB.switch.peerInfo.toRemotePeerInfo()])
let metadataOk = await metadataFut.withTimeout(TestConnectivityTimeLimit)
WakuPeerEvent.dropListener(nodeA.brokerCtx, metadataLis)
await WakuPeerEvent.dropListener(nodeA.brokerCtx, metadataLis)
require metadataOk
var deadline = Moment.now() + TestConnectivityTimeLimit
@ -413,7 +413,7 @@ suite "Health Monitor - events":
subMgr.subscribe(contentTopic).expect("Failed to subscribe")
let shardHealthOk = await shardHealthFut.withTimeout(TestConnectivityTimeLimit)
EventShardTopicHealthChange.dropListener(nodeA.brokerCtx, shardHealthLis)
await EventShardTopicHealthChange.dropListener(nodeA.brokerCtx, shardHealthLis)
check shardHealthOk == true
check subMgr.edgeFilterSubStates.len > 0

9
tests/persistency/test_all.nim Normal file
View File

@ -0,0 +1,9 @@
{.used.}
import ./test_keys
import ./test_backend
import ./test_lifecycle
import ./test_facade
import ./test_encoding
import ./test_string_lookup
import ./test_singleton

195
tests/persistency/test_backend.nim Normal file
View File

@ -0,0 +1,195 @@
{.used.}
import std/options
import results
import testutils/unittests
import waku/persistency/[types, keys, backend_sqlite]
template str(b: seq[byte]): string =
var s = newString(b.len)
for i, x in b:
s[i] = char(x)
s
proc payload(s: string): seq[byte] =
result = newSeq[byte](s.len)
for i, c in s:
result[i] = byte(c)
suite "Persistency SQLite backend":
test "open in-memory backend and round-trip a single value":
let b = openBackendInMemory().get()
defer:
b.close()
b
.applyOps(
[
TxOp(
category: "msg",
key: key("c1", 1'i64),
kind: txPut,
payload: payload("hello"),
)
]
)
.get()
let got = b.getOne("msg", key("c1", 1'i64)).get()
check got.isSome
check str(got.get) == "hello"
check b.existsOne("msg", key("c1", 1'i64)).get()
check not b.existsOne("msg", key("c1", 2'i64)).get()
test "INSERT OR REPLACE overwrites payload for the same key":
let b = openBackendInMemory().get()
defer:
b.close()
let k = key("c1", 1'i64)
b.applyOps([TxOp(category: "msg", key: k, kind: txPut, payload: payload("v1"))]).get()
b.applyOps([TxOp(category: "msg", key: k, kind: txPut, payload: payload("v2"))]).get()
check str(b.getOne("msg", k).get().get) == "v2"
test "deleteOne reports whether the row existed":
let b = openBackendInMemory().get()
defer:
b.close()
let k = key("c1", 1'i64)
check not b.deleteOne("msg", k).get()
b.applyOps([TxOp(category: "msg", key: k, kind: txPut, payload: payload("x"))]).get()
check b.deleteOne("msg", k).get()
check not b.existsOne("msg", k).get()
test "applyOps batches multiple ops atomically":
let b = openBackendInMemory().get()
defer:
b.close()
b
.applyOps(
[
TxOp(
category: "msg", key: key("c1", 1'i64), kind: txPut, payload: payload("a")
),
TxOp(
category: "msg", key: key("c1", 2'i64), kind: txPut, payload: payload("b")
),
TxOp(
category: "msg", key: key("c1", 3'i64), kind: txPut, payload: payload("c")
),
]
)
.get()
check b.countRange("msg", prefixRange(key("c1"))).get() == 3
test "scanRange ascending yields rows in key order":
let b = openBackendInMemory().get()
defer:
b.close()
let inserts = @[5'i64, 1, 4, 2, 3]
var ops: seq[TxOp] = @[]
for i in inserts:
ops.add(
TxOp(category: "msg", key: key("c1", i), kind: txPut, payload: payload($i))
)
b.applyOps(ops).get()
let rows = b.scanRange("msg", prefixRange(key("c1"))).get()
check rows.len == 5
var seenOrder: seq[string]
for r in rows:
seenOrder.add(str(r.payload))
check seenOrder == @["1", "2", "3", "4", "5"]
test "scanRange descending yields rows in reverse key order":
let b = openBackendInMemory().get()
defer:
b.close()
for i in [1'i64, 2, 3]:
b
.applyOps(
[TxOp(category: "msg", key: key("c1", i), kind: txPut, payload: payload($i))]
)
.get()
let rows = b.scanRange("msg", prefixRange(key("c1")), reverse = true).get()
check rows.len == 3
check str(rows[0].payload) == "3"
check str(rows[2].payload) == "1"
test "scanRange respects half-open [start, stop) bounds":
let b = openBackendInMemory().get()
defer:
b.close()
for i in [1'i64, 2, 3, 4, 5]:
b
.applyOps(
[TxOp(category: "msg", key: key("c1", i), kind: txPut, payload: payload($i))]
)
.get()
let rng = KeyRange(start: key("c1", 2'i64), stop: key("c1", 4'i64))
let rows = b.scanRange("msg", rng).get()
check rows.len == 2 # 2 and 3, not 4
check str(rows[0].payload) == "2"
check str(rows[1].payload) == "3"
test "scanRange with empty stop is open-ended":
let b = openBackendInMemory().get()
defer:
b.close()
for i in [1'i64, 2, 3]:
b
.applyOps(
[TxOp(category: "msg", key: key("c1", i), kind: txPut, payload: payload($i))]
)
.get()
let rng = KeyRange(start: key("c1", 2'i64), stop: rawKey(@[]))
let rows = b.scanRange("msg", rng).get()
check rows.len == 2
check str(rows[1].payload) == "3"
test "categories isolate keyspaces":
let b = openBackendInMemory().get()
defer:
b.close()
let k = key("c1", 1'i64)
b
.applyOps(
[
TxOp(category: "log", key: k, kind: txPut, payload: payload("log-1")),
TxOp(
category: "outgoing", key: k, kind: txPut, payload: payload("outgoing-1")
),
]
)
.get()
check str(b.getOne("log", k).get().get) == "log-1"
check str(b.getOne("outgoing", k).get().get) == "outgoing-1"
check b.countRange("log", prefixRange(key("c1"))).get() == 1
check b.countRange("outgoing", prefixRange(key("c1"))).get() == 1
test "txDelete inside a batch removes the row":
let b = openBackendInMemory().get()
defer:
b.close()
let k = key("c1", 1'i64)
b
.applyOps(
[
TxOp(category: "msg", key: k, kind: txPut, payload: payload("v")),
TxOp(category: "msg", key: k, kind: txDelete),
]
)
.get()
check not b.existsOne("msg", k).get()
test "missing key returns none":
let b = openBackendInMemory().get()
defer:
b.close()
check b.getOne("msg", key("nope")).get().isNone
test "countRange of empty category is zero":
let b = openBackendInMemory().get()
defer:
b.close()
check b.countRange("msg", prefixRange(key("c1"))).get() == 0

154
tests/persistency/test_encoding.nim Normal file
View File

@ -0,0 +1,154 @@
{.used.}
import std/[algorithm, options, os, times]
import chronos, results
import testutils/unittests
import waku/persistency/persistency
# Reusable byte-wise comparator (Key has its own `<`, but we sometimes
# want to sort `seq[Key]` here without relying on it for double-checking).
proc cmpBytes(a, b: Key): int =
let ab = bytes(a)
let bb = bytes(b)
let n = min(ab.len, bb.len)
for i in 0 ..< n:
if ab[i] != bb[i]:
return cmp(ab[i], bb[i])
cmp(ab.len, bb.len)
template str(b: seq[byte]): string =
var s = newString(b.len)
for i, x in b:
s[i] = char(x)
s
# Shared payload types used by multiple tests.
type
Mood = enum
moodCalm
moodHappy
moodAngry
Header = object
sender: string
epoch: int64
Msg = object
header: Header
mood: Mood
body: seq[byte]
suite "Persistency generic encoding":
# ── Key macro: composite types ────────────────────────────────────────
test "key macro accepts plain tuples":
let k1 = key(("ch", 1'i64))
let k2 = key("ch", 1'i64)
# A plain tuple is encoded field-by-field, so the result is identical
# to passing the fields directly.
check k1 == k2
test "key macro accepts named tuples":
type Coord = tuple[lane: string, seqNum: int64]
let k = key((lane: "a", seqNum: 7'i64))
let kFlat = key("a", 7'i64)
check k == kFlat
test "key macro accepts a user object":
let k1 = key(Header(sender: "alice", epoch: 5'i64))
let k2 = key("alice", 5'i64)
check k1 == k2
test "key macro accepts nested object inside another arg":
let k1 = key("v1", Header(sender: "alice", epoch: 5'i64))
let k2 = key("v1", "alice", 5'i64)
check k1 == k2
test "key macro encodes enums":
let k1 = key(moodAngry)
let k2 = key(int64(ord(moodAngry)))
check k1 == k2
test "toKey is equivalent to single-arg key()":
check toKey("x") == key("x")
check toKey(42'i64) == key(42'i64)
check toKey(Header(sender: "a", epoch: 1)) == key("a", 1'i64)
test "tuple-encoded keys preserve field-major sort order":
let inputs = @[
key(("a", 0'i64)),
key(("a", 1'i64)),
key(("a", int64.high)),
key(("b", int64.low)),
key(("b", 0'i64)),
]
var shuffled = @[inputs[3], inputs[0], inputs[4], inputs[2], inputs[1]]
shuffled.sort(cmpBytes)
check shuffled == inputs
test "embedded Key encodes verbatim":
let inner = key("a", 7'i64)
let outer = key("prefix", inner)
# Expanded: bytes of "prefix" + raw bytes of inner.
let expanded = key("prefix", "a", 7'i64)
check outer == expanded
# ── Payload macro / toPayload ─────────────────────────────────────────
test "toPayload encodes primitives":
check str(toPayload("hi")).len == 4 # 2-byte len prefix + 2 chars
check toPayload(42'i64).len == 8
check toPayload(true) == @[1'u8]
check toPayload(false) == @[0'u8]
test "toPayload encodes objects field-by-field":
let m = Msg(
header: Header(sender: "alice", epoch: 9'i64),
mood: moodHappy,
body: @[0xAA'u8, 0xBB, 0xCC],
)
let p = toPayload(m)
let pManual = payload("alice", 9'i64, int64(ord(moodHappy)), @[0xAA'u8, 0xBB, 0xCC])
check p == pManual
test "payload macro concatenates parts":
let p = payload("v1", 1'i64, @[0xDE'u8, 0xAD])
# Same as building each piece separately.
var expected: seq[byte] = @[]
encodePart(expected, "v1")
encodePart(expected, 1'i64)
encodePart(expected, @[0xDE'u8, 0xAD])
check p == expected
# ── End-to-end through the facade ─────────────────────────────────────
asyncTest "persistEncoded round-trips a struct through SQLite":
let root = getTempDir() / ("persistency_enc_" & $epochTime().int)
removeDir(root)
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
let job = p.openJob("t").get()
let m = Msg(
header: Header(sender: "alice", epoch: 1'i64),
mood: moodHappy,
body: @[1'u8, 2, 3],
)
let k = key("channel-42", m.header.epoch)
await job.persistEncoded("msg", k, m)
# Poll for the row, then read it back as raw bytes.
let deadline = epochTime() + 1.0
var got: Option[seq[byte]]
while epochTime() < deadline:
let r = await job.get("msg", k)
check r.isOk
got = r.get()
if got.isSome:
break
await sleepAsync(chronos.milliseconds(2))
check got.isSome
check got.get == toPayload(m)

196
tests/persistency/test_facade.nim Normal file
View File

@ -0,0 +1,196 @@
{.used.}
import std/[options, os, strutils, times]
import chronos, results
import testutils/unittests
import waku/persistency/persistency
proc payload(s: string): seq[byte] =
result = newSeq[byte](s.len)
for i, c in s:
result[i] = byte(c)
template str(b: seq[byte]): string =
var s = newString(b.len)
for i, x in b:
s[i] = char(x)
s
proc tmpRoot(label: string): string =
let p = getTempDir() / ("persistency_facade_" & label & "_" & $epochTime().int)
removeDir(p)
p
# Bounded poll on exists() to bridge the documented persist->read race.
proc waitUntilExists(
t: Job, category: string, k: Key, timeoutMs = 1000
): Future[bool] {.async.} =
let deadline = epochTime() + (timeoutMs.float / 1000.0)
while epochTime() < deadline:
let r = await t.exists(category, k)
if r.isOk and r.get():
return true
await sleepAsync(chronos.milliseconds(2))
return false
suite "Persistency facade":
asyncTest "persistPut then get round-trips":
let root = tmpRoot("put_get")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
let t = p.openJob("t").get()
let k = key("c", 1'i64)
await t.persistPut("msg", k, payload("hi"))
let ckOk1 = await t.waitUntilExists("msg", k)
check ckOk1
let aw1 = await t.get("msg", k)
let got = aw1.get()
check got.isSome
check str(got.get) == "hi"
asyncTest "persist (batch) is atomic and visible together":
let root = tmpRoot("batch")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
let t = p.openJob("t").get()
var ops: seq[TxOp]
for i in 1'i64 .. 4:
ops.add(
TxOp(category: "msg", key: key("c", i), kind: txPut, payload: payload($i))
)
await t.persist(ops)
let ckOk2 = await t.waitUntilExists("msg", key("c", 4'i64))
check ckOk2
let aw2 = await t.count("msg", prefixRange(key("c")))
let cnt = aw2.get()
check cnt == 4
asyncTest "scanPrefix returns rows in key order":
let root = tmpRoot("scan")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
let t = p.openJob("t").get()
for i in [3'i64, 1, 4, 1, 5, 9, 2]:
await t.persistPut("msg", key("c", i), payload($i))
let ckOk3 = await t.waitUntilExists("msg", key("c", 9'i64))
check ckOk3
let aw3 = await t.scanPrefix("msg", key("c"))
let rows = aw3.get()
# 7 ops with duplicate key i=1 -> 6 distinct rows
check rows.len == 6
var seenOrder: seq[int]
for r in rows:
seenOrder.add(parseInt(str(r.payload)))
check seenOrder == @[1, 2, 3, 4, 5, 9]
asyncTest "scanPrefix reverse=true returns rows in reverse order":
let root = tmpRoot("scan_rev")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
let t = p.openJob("t").get()
for i in 1'i64 .. 3:
await t.persistPut("msg", key("c", i), payload($i))
let ckOk4 = await t.waitUntilExists("msg", key("c", 3'i64))
check ckOk4
let aw4 = await t.scanPrefix("msg", key("c"), reverse = true)
let rows = aw4.get()
check rows.len == 3
check str(rows[0].payload) == "3"
check str(rows[2].payload) == "1"
asyncTest "deleteAcked round-trips and reports row presence":
let root = tmpRoot("delete")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
let t = p.openJob("t").get()
let k = key("c", 1'i64)
let aw5 = await t.deleteAcked("msg", k)
let miss = aw5.get()
check miss == false
await t.persistPut("msg", k, payload("v"))
let ckOk5 = await t.waitUntilExists("msg", k)
check ckOk5
let aw6 = await t.deleteAcked("msg", k)
let hit = aw6.get()
check hit == true
let aw7 = await t.exists("msg", k)
check aw7.get() == false
asyncTest "persistDelete fire-and-forget removes the row":
let root = tmpRoot("fadel")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
let t = p.openJob("t").get()
let k = key("c", 1'i64)
await t.persistPut("msg", k, payload("v"))
let ckOk6 = await t.waitUntilExists("msg", k)
check ckOk6
await t.persistDelete("msg", k)
# Poll for absence.
let deadline = epochTime() + 1.0
var gone = false
while epochTime() < deadline:
let aw8 = await t.exists("msg", k)
if not aw8.get():
gone = true
break
await sleepAsync(chronos.milliseconds(2))
check gone
asyncTest "two jobs do not see each other's data via the facade":
let root = tmpRoot("iso")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
let a = p.openJob("a").get()
let b = p.openJob("b").get()
let k = key("c", 1'i64)
await a.persistPut("msg", k, payload("A"))
await b.persistPut("msg", k, payload("B"))
let ckOk7 = await a.waitUntilExists("msg", k)
check ckOk7
let ckOk8 = await b.waitUntilExists("msg", k)
check ckOk8
let aw9 = await a.get("msg", k)
check str(aw9.get().get) == "A"
let aw10 = await b.get("msg", k)
check str(aw10.get().get) == "B"
let aw11 = await a.count("msg", prefixRange(key("c")))
check aw11.get() == 1
let aw12 = await b.count("msg", prefixRange(key("c")))
check aw12.get() == 1

135
tests/persistency/test_keys.nim Normal file
View File

@ -0,0 +1,135 @@
{.used.}
import std/[algorithm, sequtils]
import testutils/unittests
import waku/persistency/[types, keys]
proc cmpBytes(a, b: Key): int =
let ab = bytes(a)
let bb = bytes(b)
let n = min(ab.len, bb.len)
for i in 0 ..< n:
if ab[i] != bb[i]:
return cmp(ab[i], bb[i])
cmp(ab.len, bb.len)
suite "Persistency keys":
test "string components sort by length, then byte order":
var ks = @[key("ab"), key(""), key("a"), key("aa"), key("b")]
ks.sort(cmpBytes)
# length-prefix encoding => shorter strings always sort before longer
# ones; same-length strings sort in byte order.
check ks == @[key(""), key("a"), key("b"), key("aa"), key("ab")]
test "same-length strings sort in byte order":
var ks = @[key("delta"), key("alpha"), key("gamma"), key("bravo")]
ks.sort(cmpBytes)
check ks == @[key("alpha"), key("bravo"), key("delta"), key("gamma")]
test "int64 sign-flip preserves order across negative/zero/positive":
let inputs = @[
key("c", int64.low),
key("c", -2'i64),
key("c", -1'i64),
key("c", 0'i64),
key("c", 1'i64),
key("c", 2'i64),
key("c", int64.high),
]
var shuffled = inputs
# rotate so the natural order is not the input order
shuffled = @[
shuffled[3],
shuffled[6],
shuffled[0],
shuffled[5],
shuffled[1],
shuffled[4],
shuffled[2],
]
shuffled.sort(cmpBytes)
check shuffled == inputs
test "uint64 big-endian preserves order":
let inputs = @[
key("u", 0'u64),
key("u", 1'u64),
key("u", 256'u64),
key("u", 1_000_000'u64),
key("u", uint64.high - 1),
key("u", uint64.high),
]
var shuffled = @[inputs[3], inputs[0], inputs[5], inputs[2], inputs[1], inputs[4]]
shuffled.sort(cmpBytes)
check shuffled == inputs
test "composite (string, string) tuple ordering":
# First component "a" / "b" — both length 1, so byte order applies.
# Second components grouped by first; within each group, again
# length-then-byte: "" (len 0) < "a","z" (len 1) < "ab" (len 2).
let inputs = @[
key("a", ""),
key("a", "a"),
key("a", "z"),
key("a", "ab"),
key("b", ""),
key("b", "a"),
]
var shuffled = inputs.reversed()
shuffled.sort(cmpBytes)
check shuffled == inputs
test "composite (string, int64) tuple ordering":
let inputs = @[
key("a", int64.low),
key("a", -1'i64),
key("a", 0'i64),
key("a", 1'i64),
key("b", int64.low),
key("b", 0'i64),
]
var shuffled = inputs.reversed()
shuffled.sort(cmpBytes)
check shuffled == inputs
test "shorter composite key precedes longer one sharing its prefix":
check key("a") < key("a", 0'i64)
check key("a") < key("a", "")
check key("a", "x") < key("a", "x", "y")
test "Key equality is byte-wise":
check key("a", 1'i64) == key("a", 1'i64)
check not (key("a", 1'i64) == key("a", 2'i64))
test "prefixRange.start equals prefix":
let r = prefixRange(key("a"))
check r.start == key("a")
test "prefixRange.stop excludes the prefix and admits all extensions":
let r = prefixRange(key("a"))
let extensions = @[
key("a"),
key("a", 0'i64),
key("a", int64.high),
key("a", "x"),
key("a", uint64.high),
]
for k in extensions:
check r.start <= k
check k < r.stop
test "prefixRange.stop excludes siblings outside the prefix":
let r = prefixRange(key("a"))
# "b" has the same encoded length as "a" but a higher last byte, so it
# should be at-or-above the exclusive stop.
check not (key("b") < r.stop)
# "ab" has more bytes — its 2-byte length prefix bumps it past stop.
check not (key("ab") < r.stop)
# The empty key sits before the start.
check key("") < r.start
test "prefixRange handles all-0xFF prefix as open-ended":
let prefix = rawKey(@[0xFF'u8, 0xFF, 0xFF])
let r = prefixRange(prefix)
check r.start == prefix
check bytes(r.stop).len == 0

302
tests/persistency/test_lifecycle.nim Normal file
View File

@ -0,0 +1,302 @@
{.used.}
import std/[options, os, times]
import chronos, results
import testutils/unittests
import brokers/[event_broker, request_broker]
import waku/persistency/persistency
import waku/persistency/backend_comm
proc payloadBytes(s: string): seq[byte] =
result = newSeq[byte](s.len)
for i, c in s:
result[i] = byte(c)
template str(b: seq[byte]): string =
var s = newString(b.len)
for i, x in b:
s[i] = char(x)
s
proc tmpRoot(label: string): string =
let p = getTempDir() / ("persistency_test_" & label & "_" & $epochTime().int)
removeDir(p)
p
# Cross-thread persist: emit a PersistEvent then poll until the row shows up
# via KvExists. The PersistEvent listener is fire-and-forget, so reads
# immediately after emit are racy by design (documented in v1).
proc pollExists(
t: Job, category: string, k: Key, timeoutMs = 1000
): Future[bool] {.async.} =
let deadline = epochTime() + (timeoutMs.float / 1000.0)
while epochTime() < deadline:
let r = await KvExists.request(t.context, category, k)
if r.isOk and r.get().value:
return true
await sleepAsync(chronos.milliseconds(2))
return false
suite "Persistency lifecycle":
test "Persistency.instance accepts a pre-existing rootDir":
let root = tmpRoot("preexisting")
defer:
removeDir(root)
createDir(root) # pretend a previous run left it
let marker = root / "do-not-touch.txt"
writeFile(marker, "hi")
defer:
removeFile(marker)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
# The pre-existing file is untouched.
check fileExists(marker)
check readFile(marker) == "hi"
test "Persistency.instance refuses a non-directory path":
let root = tmpRoot("collision")
defer:
removeFile(root)
writeFile(root, "im a file not a dir") # collide with rootDir name
let r = Persistency.instance(root)
check r.isErr
check r.error.kind == peInvalidArgument
test "Persistency.instance defers rootDir creation until first openJob":
let root = tmpRoot("lazy")
defer:
removeDir(root)
check not dirExists(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
# instance() must not have touched the filesystem
check not dirExists(root)
discard p.openJob("first").get()
# first openJob materialises the directory
check dirExists(root)
test "Persistency.instance refuses a path whose ancestor is not a directory":
let parent = tmpRoot("bad-parent")
defer:
removeFile(parent)
writeFile(parent, "not a directory")
let root = parent / "child"
let r = Persistency.instance(root)
check r.isErr
check r.error.kind == peInvalidArgument
asyncTest "openJob reuses an existing DB file across processes-of-one":
let root = tmpRoot("reopen")
defer:
removeDir(root)
# First "session": write something then close.
block firstSession:
let p = Persistency.instance(root).get()
let j = p.openJob("persist").get()
await j.persistPut("msg", key("c", 1'i64), payloadBytes("v1"))
let ckOk1 = await j.pollExists("msg", key("c", 1'i64))
check ckOk1
Persistency.reset()
check fileExists(root / "persist.db")
# Second "session": reopen and read the data back.
block secondSession:
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
let j = p.openJob("persist").get()
let aw1 = await KvGet.request(j.context, "msg", key("c", 1'i64))
let got = aw1.get()
check got.value.isSome
check str(got.value.get) == "v1"
test "openJob is idempotent within a session":
let root = tmpRoot("idem")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
let a = p.openJob("same").get()
let b = p.openJob("same").get()
check a.id == b.id
check a.context == b.context
test "openJob materialises rootDir and launches a worker":
let root = tmpRoot("basic")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
let t = p.openJob("alpha").get()
check t.id == "alpha"
check t.running
check fileExists(root / "alpha.db")
asyncTest "persist then read round-trips via brokers":
let root = tmpRoot("rw")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
let t = p.openJob("t1").get()
let k = key("c", 1'i64)
let ev = PersistEvent(
ops: @[TxOp(category: "msg", key: k, kind: txPut, payload: payloadBytes("hello"))]
)
await PersistEvent.emit(t.context, ev)
let ckOk2 = await t.pollExists("msg", k)
check ckOk2
let aw2 = await KvGet.request(t.context, "msg", k)
let got = aw2.get()
check got.value.isSome
check str(got.value.get) == "hello"
asyncTest "two jobs run in parallel with isolated DBs":
let root = tmpRoot("isolation")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
let a = p.openJob("alpha").get()
let b = p.openJob("beta").get()
check a.context != b.context
let k = key("shared", 1'i64)
await PersistEvent.emit(
a.context,
PersistEvent(
ops: @[
TxOp(
category: "msg", key: k, kind: txPut, payload: payloadBytes("from-alpha")
)
]
),
)
await PersistEvent.emit(
b.context,
PersistEvent(
ops: @[
TxOp(category: "msg", key: k, kind: txPut, payload: payloadBytes("from-beta"))
]
),
)
let ckOk3 = await a.pollExists("msg", k)
check ckOk3
let ckOk4 = await b.pollExists("msg", k)
check ckOk4
let aw3 = await KvGet.request(a.context, "msg", k)
let aGot = aw3.get()
let aw4 = await KvGet.request(b.context, "msg", k)
let bGot = aw4.get()
check str(aGot.value.get) == "from-alpha"
check str(bGot.value.get) == "from-beta"
# Each job has its own DB file.
check fileExists(root / "alpha.db")
check fileExists(root / "beta.db")
asyncTest "closeJob joins the worker and frees the slot":
let root = tmpRoot("close")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
let t = p.openJob("x").get()
let ctx = t.context
p.closeJob("x")
check not t.running
# After close, requests on the old context have no provider.
let r = await KvExists.request(ctx, "msg", key("k"))
check r.isErr
test "dropJob removes the DB file":
let root = tmpRoot("drop")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
discard p.openJob("ephemeral").get()
check fileExists(root / "ephemeral.db")
p.dropJob("ephemeral")
check not fileExists(root / "ephemeral.db")
asyncTest "scan and count over a range":
let root = tmpRoot("scan")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
let t = p.openJob("t").get()
var ops: seq[TxOp]
for i in 1'i64 .. 5:
ops.add(
TxOp(category: "msg", key: key("c", i), kind: txPut, payload: payloadBytes($i))
)
await PersistEvent.emit(t.context, PersistEvent(ops: ops))
# Wait for the last insert to land.
let ckOk5 = await t.pollExists("msg", key("c", 5'i64))
check ckOk5
let rng = prefixRange(key("c"))
let aw5 = await KvCount.request(t.context, "msg", rng)
let cnt = aw5.get()
check cnt.n == 5
let aw6 = await KvScan.request(t.context, "msg", rng, false)
let scn = aw6.get()
check scn.rows.len == 5
check str(scn.rows[0].payload) == "1"
check str(scn.rows[4].payload) == "5"
asyncTest "acked delete reports whether the row existed":
let root = tmpRoot("delete")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
let t = p.openJob("t").get()
let k = key("d", 1'i64)
let aw7 = await KvDelete.request(t.context, "msg", k)
let r1 = aw7.get()
check r1.existed == false
await PersistEvent.emit(
t.context,
PersistEvent(
ops: @[TxOp(category: "msg", key: k, kind: txPut, payload: payloadBytes("v"))]
),
)
let ckOk6 = await t.pollExists("msg", k)
check ckOk6
let aw8 = await KvDelete.request(t.context, "msg", k)
let r2 = aw8.get()
check r2.existed == true
let aw9 = await KvExists.request(t.context, "msg", k)
let r3 = aw9.get()
check r3.value == false

79
tests/persistency/test_singleton.nim Normal file
View File

@ -0,0 +1,79 @@
{.used.}
import std/[os, strutils, times]
import chronos, results
import testutils/unittests
import brokers/multi_request_broker
import waku/persistency/persistency
proc tmpRoot(label: string): string =
let p = getTempDir() / ("persistency_singleton_" & label & "_" & $epochTime().int)
removeDir(p)
p
suite "Persistency singleton":
test "instance(rootDir) is idempotent with the same rootDir":
let root = tmpRoot("idem")
defer:
removeDir(root)
defer:
Persistency.reset()
let p1 = Persistency.instance(root).get()
let p2 = Persistency.instance(root).get()
check p1 == p2
test "instance(rootDir) refuses re-init with a different rootDir":
let rootA = tmpRoot("a")
let rootB = tmpRoot("b")
defer:
removeDir(rootA)
defer:
removeDir(rootB)
defer:
Persistency.reset()
discard Persistency.instance(rootA).get()
let r = Persistency.instance(rootB)
check r.isErr
check r.error.kind == peInvalidArgument
test "no-arg instance() fails before init, succeeds after":
let root = tmpRoot("noarg")
defer:
removeDir(root)
defer:
Persistency.reset()
let before = Persistency.instance()
check before.isErr
check before.error.kind == peClosed
discard Persistency.instance(root).get()
let after = Persistency.instance()
check after.isOk
test "reset() makes the next instance() target a different rootDir":
let rootA = tmpRoot("rs-a")
let rootB = tmpRoot("rs-b")
defer:
removeDir(rootA)
defer:
removeDir(rootB)
defer:
Persistency.reset()
let pA = Persistency.instance(rootA).get()
check pA.rootDir == rootA
Persistency.reset()
let pB = Persistency.instance(rootB).get()
check pB.rootDir == rootB
check pA != pB
test "reset() is idempotent":
defer:
Persistency.reset()
Persistency.reset()
Persistency.reset()
check Persistency.instance().isErr

184
tests/persistency/test_string_lookup.nim Normal file
View File

@ -0,0 +1,184 @@
{.used.}
import std/[options, os, times]
import chronos, results
import testutils/unittests
import waku/persistency/persistency
proc payloadBytes(s: string): seq[byte] =
result = newSeq[byte](s.len)
for i, c in s:
result[i] = byte(c)
template str(b: seq[byte]): string =
var s = newString(b.len)
for i, x in b:
s[i] = char(x)
s
proc tmpRoot(label: string): string =
let p = getTempDir() / ("persistency_lookup_" & label & "_" & $epochTime().int)
removeDir(p)
p
# Bridge the persist->read race (writes are fire-and-forget in v1).
proc waitUntilExists(
p: Persistency, jobId, category: string, k: Key, timeoutMs = 1000
): Future[bool] {.async.} =
let deadline = epochTime() + (timeoutMs.float / 1000.0)
while epochTime() < deadline:
let r = await p.exists(jobId, category, k)
if r.isOk and r.get():
return true
await sleepAsync(chronos.milliseconds(2))
return false
suite "Persistency string-id lookup":
test "job(p, id) returns peJobNotFound when not open":
let root = tmpRoot("notfound")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
let r = p.job("nope")
check r.isErr
check r.error.kind == peJobNotFound
test "job(p, id) returns the Job after openJob":
let root = tmpRoot("found")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
let opened = p.openJob("alpha").get()
let looked = p.job("alpha").get()
check looked.id == "alpha"
check looked == opened # same ref, no need to peek at .context
test "hasJob mirrors p.job()":
let root = tmpRoot("has")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
check not p.hasJob("x")
discard p.openJob("x")
check p.hasJob("x")
p.closeJob("x")
check not p.hasJob("x")
test "subscript [] returns the open Job":
let root = tmpRoot("subscript")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
discard p.openJob("a").get()
let j = p["a"]
check j.id == "a"
asyncTest "string-lookup persistPut + get round-trips without a Job ref":
let root = tmpRoot("rw")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
discard p.openJob("svc").get()
let k = key("c", 1'i64)
await p.persistPut("svc", "msg", k, payloadBytes("hello"))
let ckOk1 = await p.waitUntilExists("svc", "msg", k)
check ckOk1
let aw1 = await p.get("svc", "msg", k)
let got = aw1.get()
check got.isSome
check str(got.get) == "hello"
asyncTest "string-lookup reads short-circuit with peJobNotFound":
let root = tmpRoot("missingread")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
let g = await p.get("nope", "msg", key("k"))
check g.isErr
check g.error.kind == peJobNotFound
let c = await p.count("nope", "msg", prefixRange(key("k")))
check c.isErr
check c.error.kind == peJobNotFound
let d = await p.deleteAcked("nope", "msg", key("k"))
check d.isErr
check d.error.kind == peJobNotFound
asyncTest "string-lookup writes to an unknown job are dropped, not raised":
let root = tmpRoot("missingwrite")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
# Should not raise and should not leak any state.
await p.persistPut("ghost", "msg", key("k"), payloadBytes("v"))
await p.persistDelete("ghost", "msg", key("k"))
await p.persistEncoded("ghost", "msg", key("k"), 42'i64)
check not p.hasJob("ghost")
asyncTest "string-lookup persistEncoded round-trips a struct":
let root = tmpRoot("encoded")
defer:
removeDir(root)
type Item = object
tag: string
n: int64
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
discard p.openJob("e").get()
let k = key("items", 1'i64)
await p.persistEncoded("e", "msg", k, Item(tag: "alpha", n: 7))
let ckOk2 = await p.waitUntilExists("e", "msg", k)
check ckOk2
let aw2 = await p.get("e", "msg", k)
let got = aw2.get()
check got.isSome
check got.get == toPayload(Item(tag: "alpha", n: 7))
asyncTest "string-lookup scan returns the same rows as Job-form":
let root = tmpRoot("scan")
defer:
removeDir(root)
let p = Persistency.instance(root).get()
defer:
Persistency.reset()
let j = p.openJob("s").get()
for i in 1'i64 .. 3:
await p.persistPut("s", "msg", key("c", i), payloadBytes($i))
let ckOk3 = await p.waitUntilExists("s", "msg", key("c", 3'i64))
check ckOk3
let aw3 = await p.scanPrefix("s", "msg", key("c"))
let viaId = aw3.get()
let aw4 = await j.scanPrefix("msg", key("c"))
let viaRef = aw4.get()
check viaId.len == viaRef.len
for i in 0 ..< viaId.len:
check viaId[i].key == viaRef[i].key
check viaId[i].payload == viaRef[i].payload

10
tests/waku_relay/utils.nim
View File

@ -8,17 +8,13 @@ import
libp2p/switch,
libp2p/protocols/pubsub/pubsub
import brokers/broker_context
from std/times import epochTime
import
waku/[
waku_relay,
node/waku_node,
node/peer_manager,
waku_core,
waku_node,
waku_rln_relay,
common/broker/broker_context,
waku_relay, node/waku_node, node/peer_manager, waku_core, waku_node, waku_rln_relay
],
../waku_store/store_utils,
../waku_archive/archive_utils,

4
tests/waku_rln_relay/test_waku_rln_relay.nim
View File

@ -8,6 +8,9 @@ import
chronicles,
stint,
libp2p/crypto/crypto
import brokers/broker_context
import
waku/[
waku_core,
@ -15,7 +18,6 @@ import
waku_rln_relay/rln,
waku_rln_relay/protocol_metrics,
waku_keystore,
common/broker/broker_context,
],
./rln/waku_rln_relay_utils,
./utils_onchain,

7
tests/waku_rln_relay/test_wakunode_rln_relay.nim
View File

@ -7,13 +7,14 @@ import
chronicles,
chronos,
libp2p/switch,
libp2p/protocols/pubsub/pubsub
libp2p/protocols/pubsub/pubsub,
brokers/broker_context
import
waku/[waku_core, waku_node, waku_rln_relay],
../testlib/[wakucore, futures, wakunode, testutils],
./utils_onchain,
./rln/waku_rln_relay_utils,
waku/common/broker/broker_context
./rln/waku_rln_relay_utils
from std/times import epochTime

2
tests/wakunode_rest/test_rest_relay.nim
View File

@ -7,6 +7,7 @@ import
presto,
presto/client as presto_client,
libp2p/crypto/crypto
import brokers/broker_context
import
waku/[
common/base64,
@ -21,7 +22,6 @@ import
rest_api/endpoint/relay/client as relay_rest_client,
waku_relay,
waku_rln_relay,
common/broker/broker_context,
],
../testlib/wakucore,
../testlib/wakunode,

8
tools/confutils/cli_args.nim
View File

@ -698,6 +698,12 @@ with the drawback of consuming some more bandwidth.""",
name: "rate-limit"
.}: seq[string]
localStoragePath* {.
desc: "Path to store local data.",
defaultValue: "./data",
name: "local-storage-path"
.}: string
## Parsing
# NOTE: Keys are different in nim-libp2p
@ -1119,6 +1125,8 @@ proc toWakuConf*(n: WakuNodeConf): ConfResult[WakuConf] =
if n.rateLimits.len > 0:
b.rateLimitConf.withRateLimits(n.rateLimits)
b.withLocalStoragePath(n.localStoragePath)
b.kademliaDiscoveryConf.withEnabled(n.enableKadDiscovery)
b.kademliaDiscoveryConf.withBootstrapNodes(n.kadBootstrapNodes)

10
tools/gen-nix-deps.sh
View File

@ -36,7 +36,10 @@ fi
echo "[*] Generating $OUTFILE from $LOCKFILE"
mkdir -p "$(dirname "$OUTFILE")"
cat > "$OUTFILE" <<'EOF'
TMPFILE=$(mktemp)
trap 'rm -f "$TMPFILE"' EXIT
cat > "$TMPFILE" <<'EOF'
# AUTOGENERATED from nimble.lock — do not edit manually.
# Regenerate with: ./tools/gen-nix-deps.sh nimble.lock nix/deps.nix
{ pkgs }:
@ -62,7 +65,7 @@ jq -c '
--fetch-submodules \
| jq -r '.sha256')
cat >> "$OUTFILE" <<EOF
cat >> "$TMPFILE" <<EOF
${name} = pkgs.fetchgit {
url = "${url}";
rev = "${rev}";
@ -73,8 +76,9 @@ jq -c '
EOF
done
cat >> "$OUTFILE" <<'EOF'
cat >> "$TMPFILE" <<'EOF'
}
EOF
mv "$TMPFILE" "$OUTFILE"
echo "[✓] Wrote $OUTFILE"

10
waku.nimble
View File

@ -63,6 +63,16 @@ requires "https://github.com/logos-messaging/nim-ffi"
requires "https://github.com/logos-messaging/nim-sds.git#2e9a7683f0e180bf112135fae3a3803eed8490d4"
# brokers: pinned by URL+commit rather than the bare `brokers >= 2.0.1`
# form because the nim-lang/packages registry entry for `brokers` only
# carries metadata for the original v0.1.0 publication. Until that
# registry entry is refreshed, the local SAT solver enumerates "0.1.0"
# as the only available version and cannot satisfy `>= 2.0.1`. The URL
# pin below bypasses the registry and locks the exact commit of the
# v2.0.1 tag. Revert to the bare form once nim-lang/packages is
# updated.
requires "https://github.com/NagyZoltanPeter/nim-brokers.git#v2.0.1"
requires "https://github.com/vacp2p/nim-lsquic"
requires "https://github.com/vacp2p/nim-jwt.git#057ec95eb5af0eea9c49bfe9025b3312c95dc5f2"

68
waku/common/broker/broker_context.nim
View File

@ -1,68 +0,0 @@
{.push raises: [].}
import std/[strutils, concurrency/atomics], chronos
type BrokerContext* = distinct uint32
func `==`*(a, b: BrokerContext): bool =
uint32(a) == uint32(b)
func `!=`*(a, b: BrokerContext): bool =
uint32(a) != uint32(b)
func `$`*(bc: BrokerContext): string =
toHex(uint32(bc), 8)
const DefaultBrokerContext* = BrokerContext(0xCAFFE14E'u32)
# Global broker context accessor.
#
# NOTE: This intentionally creates a *single* active BrokerContext per process
# (per event loop thread). Use only if you accept serialization of all broker
# context usage through the lock.
var globalBrokerContextLock {.threadvar.}: AsyncLock
globalBrokerContextLock = newAsyncLock()
var globalBrokerContextValue {.threadvar.}: BrokerContext
globalBrokerContextValue = DefaultBrokerContext
proc globalBrokerContext*(): BrokerContext =
## Returns the currently active global broker context.
##
## This is intentionally lock-free; callers should use it inside
## `withNewGlobalBrokerContext` / `withGlobalBrokerContext`.
globalBrokerContextValue
var gContextCounter: Atomic[uint32]
proc NewBrokerContext*(): BrokerContext =
var nextId = gContextCounter.fetchAdd(1, moRelaxed)
if nextId == uint32(DefaultBrokerContext):
nextId = gContextCounter.fetchAdd(1, moRelaxed)
return BrokerContext(nextId)
template lockGlobalBrokerContext*(brokerCtx: BrokerContext, body: untyped): untyped =
## Runs `body` while holding the global broker context lock with the provided
## `brokerCtx` installed as the globally accessible context.
##
## This template is intended for use from within `chronos` async procs.
block:
await noCancel(globalBrokerContextLock.acquire())
let previousBrokerCtx = globalBrokerContextValue
globalBrokerContextValue = brokerCtx
try:
body
finally:
globalBrokerContextValue = previousBrokerCtx
try:
globalBrokerContextLock.release()
except AsyncLockError:
doAssert false, "globalBrokerContextLock.release(): lock not held"
template lockNewGlobalBrokerContext*(body: untyped): untyped =
## Runs `body` while holding the global broker context lock with a freshly
## generated broker context installed as the global accessor.
##
## The previous global broker context (if any) is restored on exit.
lockGlobalBrokerContext(NewBrokerContext()):
body
{.pop.}

411
waku/common/broker/event_broker.nim
View File

@ -1,411 +0,0 @@
## EventBroker
## -------------------
## EventBroker represents a reactive decoupling pattern, that
## allows event-driven development without
## need for direct dependencies in between emitters and listeners.
## Worth considering using it in a single or many emitters to many listeners scenario.
##
## Generates a standalone, type-safe event broker for the declared type.
## The macro exports the value type itself plus a broker companion that manages
## listeners via thread-local storage.
##
## Type definitions:
## - Inline `object` / `ref object` definitions are supported.
## - Native types, aliases, and externally-defined types are also supported.
## In that case, EventBroker will automatically wrap the declared RHS type in
## `distinct` unless you already used `distinct`.
## This keeps event types unique even when multiple brokers share the same
## underlying base type.
##
## Default vs. context aware use:
## Every generated broker is a thread-local global instance. This means EventBroker
## enables decoupled event exchange threadwise.
##
## Sometimes we use brokers inside a context (e.g. within a component that has many
## modules or subsystems). If you instantiate multiple such components in a single
## thread, and each component must have its own listener set for the same EventBroker
## type, you can use context-aware EventBroker.
##
## Context awareness is supported through the `BrokerContext` argument for
## `listen`, `emit`, `dropListener`, and `dropAllListeners`.
## Listener stores are kept separate per broker context.
##
## Default broker context is defined as `DefaultBrokerContext`. If you don't need
## context awareness, you can keep using the interfaces without the context
## argument, which operate on `DefaultBrokerContext`.
##
## Usage:
## Declare your desired event type inside an `EventBroker` macro, add any number of fields.:
## ```nim
## EventBroker:
## type TypeName = object
## field1*: FieldType
## field2*: AnotherFieldType
## ```
##
## After this, you can register async listeners anywhere in your code with
## `TypeName.listen(...)`, which returns a handle to the registered listener.
## Listeners are async procs or lambdas that take a single argument of the event type.
## Any number of listeners can be registered in different modules.
##
## Events can be emitted from anywhere with no direct dependency on the listeners by
## calling `TypeName.emit(...)` with an instance of the event type.
## This will asynchronously notify all registered listeners with the emitted event.
##
## Whenever you no longer need a listener (or your object instance that listen to the event goes out of scope),
## you can remove it from the broker with the handle returned by `listen`.
## This is done by calling `TypeName.dropListener(handle)`.
## Alternatively, you can remove all registered listeners through `TypeName.dropAllListeners()`.
##
##
## Example:
## ```nim
## EventBroker:
## type GreetingEvent = object
## text*: string
##
## let handle = GreetingEvent.listen(
## proc(evt: GreetingEvent): Future[void] {.async.} =
## echo evt.text
## )
## GreetingEvent.emit(text= "hi")
## GreetingEvent.dropListener(handle)
## ```
## Example (non-object event type):
## ```nim
## EventBroker:
## type CounterEvent = int # exported as: `distinct int`
##
## discard CounterEvent.listen(
## proc(evt: CounterEvent): Future[void] {.async.} =
## echo int(evt)
## )
## CounterEvent.emit(CounterEvent(42))
## ```
import std/[macros, tables]
import chronos, chronicles, results
import ./helper/broker_utils, broker_context
export chronicles, results, chronos, broker_context
macro EventBroker*(body: untyped): untyped =
when defined(eventBrokerDebug):
echo body.treeRepr
let parsed = parseSingleTypeDef(body, "EventBroker", collectFieldInfo = true)
let typeIdent = parsed.typeIdent
let objectDef = parsed.objectDef
let fieldNames = parsed.fieldNames
let fieldTypes = parsed.fieldTypes
let hasInlineFields = parsed.hasInlineFields
let exportedTypeIdent = postfix(copyNimTree(typeIdent), "*")
let sanitized = sanitizeIdentName(typeIdent)
let typeNameLit = newLit($typeIdent)
let handlerProcIdent = ident(sanitized & "ListenerProc")
let listenerHandleIdent = ident(sanitized & "Listener")
let brokerTypeIdent = ident(sanitized & "Broker")
let exportedHandlerProcIdent = postfix(copyNimTree(handlerProcIdent), "*")
let exportedListenerHandleIdent = postfix(copyNimTree(listenerHandleIdent), "*")
let exportedBrokerTypeIdent = postfix(copyNimTree(brokerTypeIdent), "*")
let bucketTypeIdent = ident(sanitized & "CtxBucket")
let findBucketIdxIdent = ident(sanitized & "FindBucketIdx")
let getOrCreateBucketIdxIdent = ident(sanitized & "GetOrCreateBucketIdx")
let accessProcIdent = ident("access" & sanitized & "Broker")
let globalVarIdent = ident("g" & sanitized & "Broker")
let listenImplIdent = ident("register" & sanitized & "Listener")
let dropListenerImplIdent = ident("drop" & sanitized & "Listener")
let dropAllListenersImplIdent = ident("dropAll" & sanitized & "Listeners")
let emitImplIdent = ident("emit" & sanitized & "Value")
let listenerTaskIdent = ident("notify" & sanitized & "Listener")
result = newStmtList()
result.add(
quote do:
type
`exportedTypeIdent` = `objectDef`
`exportedListenerHandleIdent` = object
id*: uint64
`exportedHandlerProcIdent` =
proc(event: `typeIdent`): Future[void] {.async: (raises: []), gcsafe.}
`bucketTypeIdent` = object
brokerCtx: BrokerContext
listeners: Table[uint64, `handlerProcIdent`]
nextId: uint64
`exportedBrokerTypeIdent` = ref object
buckets: seq[`bucketTypeIdent`]
)
result.add(
quote do:
var `globalVarIdent` {.threadvar.}: `brokerTypeIdent`
)
result.add(
quote do:
proc `accessProcIdent`(): `brokerTypeIdent` =
if `globalVarIdent`.isNil():
new(`globalVarIdent`)
`globalVarIdent`.buckets = @[
`bucketTypeIdent`(
brokerCtx: DefaultBrokerContext,
listeners: initTable[uint64, `handlerProcIdent`](),
nextId: 1'u64,
)
]
`globalVarIdent`
)
result.add(
quote do:
proc `findBucketIdxIdent`(
broker: `brokerTypeIdent`, brokerCtx: BrokerContext
): int =
if brokerCtx == DefaultBrokerContext:
return 0
for i in 1 ..< broker.buckets.len:
if broker.buckets[i].brokerCtx == brokerCtx:
return i
return -1
proc `getOrCreateBucketIdxIdent`(
broker: `brokerTypeIdent`, brokerCtx: BrokerContext
): int =
let idx = `findBucketIdxIdent`(broker, brokerCtx)
if idx >= 0:
return idx
broker.buckets.add(
`bucketTypeIdent`(
brokerCtx: brokerCtx,
listeners: initTable[uint64, `handlerProcIdent`](),
nextId: 1'u64,
)
)
return broker.buckets.high
proc `listenImplIdent`(
brokerCtx: BrokerContext, handler: `handlerProcIdent`
): Result[`listenerHandleIdent`, string] =
if handler.isNil():
return err("Must provide a non-nil event handler")
var broker = `accessProcIdent`()
let bucketIdx = `getOrCreateBucketIdxIdent`(broker, brokerCtx)
if broker.buckets[bucketIdx].nextId == 0'u64:
broker.buckets[bucketIdx].nextId = 1'u64
if broker.buckets[bucketIdx].nextId == high(uint64):
error "Cannot add more listeners: ID space exhausted",
nextId = $broker.buckets[bucketIdx].nextId
return err("Cannot add more listeners, listener ID space exhausted")
let newId = broker.buckets[bucketIdx].nextId
inc broker.buckets[bucketIdx].nextId
broker.buckets[bucketIdx].listeners[newId] = handler
return ok(`listenerHandleIdent`(id: newId))
)
result.add(
quote do:
proc `dropListenerImplIdent`(
brokerCtx: BrokerContext, handle: `listenerHandleIdent`
) =
if handle.id == 0'u64:
return
var broker = `accessProcIdent`()
let bucketIdx = `findBucketIdxIdent`(broker, brokerCtx)
if bucketIdx < 0:
return
if broker.buckets[bucketIdx].listeners.len == 0:
return
broker.buckets[bucketIdx].listeners.del(handle.id)
if brokerCtx != DefaultBrokerContext and
broker.buckets[bucketIdx].listeners.len == 0:
broker.buckets.delete(bucketIdx)
)
result.add(
quote do:
proc `dropAllListenersImplIdent`(brokerCtx: BrokerContext) =
var broker = `accessProcIdent`()
let bucketIdx = `findBucketIdxIdent`(broker, brokerCtx)
if bucketIdx < 0:
return
if broker.buckets[bucketIdx].listeners.len > 0:
broker.buckets[bucketIdx].listeners.clear()
if brokerCtx != DefaultBrokerContext:
broker.buckets.delete(bucketIdx)
)
result.add(
quote do:
proc listen*(
_: typedesc[`typeIdent`], handler: `handlerProcIdent`
): Result[`listenerHandleIdent`, string] =
return `listenImplIdent`(DefaultBrokerContext, handler)
proc listen*(
_: typedesc[`typeIdent`],
brokerCtx: BrokerContext,
handler: `handlerProcIdent`,
): Result[`listenerHandleIdent`, string] =
return `listenImplIdent`(brokerCtx, handler)
)
result.add(
quote do:
proc dropListener*(_: typedesc[`typeIdent`], handle: `listenerHandleIdent`) =
`dropListenerImplIdent`(DefaultBrokerContext, handle)
proc dropListener*(
_: typedesc[`typeIdent`],
brokerCtx: BrokerContext,
handle: `listenerHandleIdent`,
) =
`dropListenerImplIdent`(brokerCtx, handle)
proc dropAllListeners*(_: typedesc[`typeIdent`]) =
`dropAllListenersImplIdent`(DefaultBrokerContext)
proc dropAllListeners*(_: typedesc[`typeIdent`], brokerCtx: BrokerContext) =
`dropAllListenersImplIdent`(brokerCtx)
)
result.add(
quote do:
proc `listenerTaskIdent`(
callback: `handlerProcIdent`, event: `typeIdent`
) {.async: (raises: []), gcsafe.} =
if callback.isNil():
return
try:
await callback(event)
except Exception:
error "Failed to execute event listener", error = getCurrentExceptionMsg()
proc `emitImplIdent`(
brokerCtx: BrokerContext, event: `typeIdent`
): Future[void] {.async: (raises: []), gcsafe.} =
when compiles(event.isNil()):
if event.isNil():
error "Cannot emit uninitialized event object", eventType = `typeNameLit`
return
let broker = `accessProcIdent`()
let bucketIdx = `findBucketIdxIdent`(broker, brokerCtx)
if bucketIdx < 0:
# nothing to do as nobody is listening
return
if broker.buckets[bucketIdx].listeners.len == 0:
return
var callbacks: seq[`handlerProcIdent`] = @[]
for cb in broker.buckets[bucketIdx].listeners.values:
callbacks.add(cb)
for cb in callbacks:
asyncSpawn `listenerTaskIdent`(cb, event)
proc emit*(event: `typeIdent`) =
asyncSpawn `emitImplIdent`(DefaultBrokerContext, event)
proc emit*(_: typedesc[`typeIdent`], event: `typeIdent`) =
asyncSpawn `emitImplIdent`(DefaultBrokerContext, event)
proc emit*(
_: typedesc[`typeIdent`], brokerCtx: BrokerContext, event: `typeIdent`
) =
asyncSpawn `emitImplIdent`(brokerCtx, event)
)
if hasInlineFields:
# Typedesc emit constructor overloads for inline object/ref object types.
var emitCtorParams = newTree(nnkFormalParams, newEmptyNode())
let typedescParamType =
newTree(nnkBracketExpr, ident("typedesc"), copyNimTree(typeIdent))
emitCtorParams.add(
newTree(nnkIdentDefs, ident("_"), typedescParamType, newEmptyNode())
)
for i in 0 ..< fieldNames.len:
emitCtorParams.add(
newTree(
nnkIdentDefs,
copyNimTree(fieldNames[i]),
copyNimTree(fieldTypes[i]),
newEmptyNode(),
)
)
var emitCtorExpr = newTree(nnkObjConstr, copyNimTree(typeIdent))
for i in 0 ..< fieldNames.len:
emitCtorExpr.add(
newTree(
nnkExprColonExpr, copyNimTree(fieldNames[i]), copyNimTree(fieldNames[i])
)
)
let emitCtorCallDefault =
newCall(copyNimTree(emitImplIdent), ident("DefaultBrokerContext"), emitCtorExpr)
let emitCtorBodyDefault = quote:
asyncSpawn `emitCtorCallDefault`
let typedescEmitProcDefault = newTree(
nnkProcDef,
postfix(ident("emit"), "*"),
newEmptyNode(),
newEmptyNode(),
emitCtorParams,
newEmptyNode(),
newEmptyNode(),
emitCtorBodyDefault,
)
result.add(typedescEmitProcDefault)
var emitCtorParamsCtx = newTree(nnkFormalParams, newEmptyNode())
emitCtorParamsCtx.add(
newTree(nnkIdentDefs, ident("_"), typedescParamType, newEmptyNode())
)
emitCtorParamsCtx.add(
newTree(nnkIdentDefs, ident("brokerCtx"), ident("BrokerContext"), newEmptyNode())
)
for i in 0 ..< fieldNames.len:
emitCtorParamsCtx.add(
newTree(
nnkIdentDefs,
copyNimTree(fieldNames[i]),
copyNimTree(fieldTypes[i]),
newEmptyNode(),
)
)
let emitCtorCallCtx =
newCall(copyNimTree(emitImplIdent), ident("brokerCtx"), copyNimTree(emitCtorExpr))
let emitCtorBodyCtx = quote:
asyncSpawn `emitCtorCallCtx`
let typedescEmitProcCtx = newTree(
nnkProcDef,
postfix(ident("emit"), "*"),
newEmptyNode(),
newEmptyNode(),
emitCtorParamsCtx,
newEmptyNode(),
newEmptyNode(),
emitCtorBodyCtx,
)
result.add(typedescEmitProcCtx)
when defined(eventBrokerDebug):
echo result.repr

206
waku/common/broker/helper/broker_utils.nim
View File

@ -1,206 +0,0 @@
import std/macros
type ParsedBrokerType* = object
## Result of parsing the single `type` definition inside a broker macro body.
##
## - `typeIdent`: base identifier for the declared type name
## - `objectDef`: exported type definition RHS (inline object fields exported;
## non-object types wrapped in `distinct` unless already distinct)
## - `isRefObject`: true only for inline `ref object` definitions
## - `hasInlineFields`: true for inline `object` / `ref object`
## - `fieldNames`/`fieldTypes`: populated only when `collectFieldInfo = true`
typeIdent*: NimNode
objectDef*: NimNode
isRefObject*: bool
hasInlineFields*: bool
fieldNames*: seq[NimNode]
fieldTypes*: seq[NimNode]
proc sanitizeIdentName*(node: NimNode): string =
var raw = $node
var sanitizedName = newStringOfCap(raw.len)
for ch in raw:
case ch
of 'A' .. 'Z', 'a' .. 'z', '0' .. '9', '_':
sanitizedName.add(ch)
else:
sanitizedName.add('_')
sanitizedName
proc ensureFieldDef*(node: NimNode) =
if node.kind != nnkIdentDefs or node.len < 3:
error("Expected field definition of the form `name: Type`", node)
let typeSlot = node.len - 2
if node[typeSlot].kind == nnkEmpty:
error("Field `" & $node[0] & "` must declare a type", node)
proc exportIdentNode*(node: NimNode): NimNode =
case node.kind
of nnkIdent:
postfix(copyNimTree(node), "*")
of nnkPostfix:
node
else:
error("Unsupported identifier form in field definition", node)
proc baseTypeIdent*(defName: NimNode): NimNode =
case defName.kind
of nnkIdent:
defName
of nnkAccQuoted:
if defName.len != 1:
error("Unsupported quoted identifier", defName)
defName[0]
of nnkPostfix:
baseTypeIdent(defName[1])
of nnkPragmaExpr:
baseTypeIdent(defName[0])
else:
error("Unsupported type name in broker definition", defName)
proc ensureDistinctType*(rhs: NimNode): NimNode =
## For PODs / aliases / externally-defined types, wrap in `distinct` unless
## it's already distinct.
if rhs.kind == nnkDistinctTy:
return copyNimTree(rhs)
newTree(nnkDistinctTy, copyNimTree(rhs))
proc cloneParams*(params: seq[NimNode]): seq[NimNode] =
## Deep copy parameter definitions so they can be inserted in multiple places.
result = @[]
for param in params:
result.add(copyNimTree(param))
proc collectParamNames*(params: seq[NimNode]): seq[NimNode] =
## Extract all identifier symbols declared across IdentDefs nodes.
result = @[]
for param in params:
assert param.kind == nnkIdentDefs
for i in 0 ..< param.len - 2:
let nameNode = param[i]
if nameNode.kind == nnkEmpty:
continue
result.add(ident($nameNode))
proc parseSingleTypeDef*(
body: NimNode,
macroName: string,
allowRefToNonObject = false,
collectFieldInfo = false,
): ParsedBrokerType =
## Parses exactly one `type` definition from a broker macro body.
##
## Supported RHS:
## - inline `object` / `ref object` (fields are auto-exported)
## - non-object types / aliases / externally-defined types (wrapped in `distinct`)
## - optionally: `ref SomeType` when `allowRefToNonObject = true`
var typeIdent: NimNode = nil
var objectDef: NimNode = nil
var isRefObject = false
var hasInlineFields = false
var fieldNames: seq[NimNode] = @[]
var fieldTypes: seq[NimNode] = @[]
for stmt in body:
if stmt.kind != nnkTypeSection:
continue
for def in stmt:
if def.kind != nnkTypeDef:
continue
if not typeIdent.isNil():
error("Only one type may be declared inside " & macroName, def)
typeIdent = baseTypeIdent(def[0])
let rhs = def[2]
case rhs.kind
of nnkObjectTy:
let recList = rhs[2]
if recList.kind != nnkRecList:
error(macroName & " object must declare a standard field list", rhs)
var exportedRecList = newTree(nnkRecList)
for field in recList:
case field.kind
of nnkIdentDefs:
ensureFieldDef(field)
if collectFieldInfo:
let fieldTypeNode = field[field.len - 2]
for i in 0 ..< field.len - 2:
let baseFieldIdent = baseTypeIdent(field[i])
fieldNames.add(copyNimTree(baseFieldIdent))
fieldTypes.add(copyNimTree(fieldTypeNode))
var cloned = copyNimTree(field)
for i in 0 ..< cloned.len - 2:
cloned[i] = exportIdentNode(cloned[i])
exportedRecList.add(cloned)
of nnkEmpty:
discard
else:
error(
macroName & " object definition only supports simple field declarations",
field,
)
objectDef = newTree(
nnkObjectTy, copyNimTree(rhs[0]), copyNimTree(rhs[1]), exportedRecList
)
isRefObject = false
hasInlineFields = true
of nnkRefTy:
if rhs.len != 1:
error(macroName & " ref type must have a single base", rhs)
if rhs[0].kind == nnkObjectTy:
let obj = rhs[0]
let recList = obj[2]
if recList.kind != nnkRecList:
error(macroName & " object must declare a standard field list", obj)
var exportedRecList = newTree(nnkRecList)
for field in recList:
case field.kind
of nnkIdentDefs:
ensureFieldDef(field)
if collectFieldInfo:
let fieldTypeNode = field[field.len - 2]
for i in 0 ..< field.len - 2:
let baseFieldIdent = baseTypeIdent(field[i])
fieldNames.add(copyNimTree(baseFieldIdent))
fieldTypes.add(copyNimTree(fieldTypeNode))
var cloned = copyNimTree(field)
for i in 0 ..< cloned.len - 2:
cloned[i] = exportIdentNode(cloned[i])
exportedRecList.add(cloned)
of nnkEmpty:
discard
else:
error(
macroName & " object definition only supports simple field declarations",
field,
)
let exportedObjectType = newTree(
nnkObjectTy, copyNimTree(obj[0]), copyNimTree(obj[1]), exportedRecList
)
objectDef = newTree(nnkRefTy, exportedObjectType)
isRefObject = true
hasInlineFields = true
elif allowRefToNonObject:
## `ref SomeType` (SomeType can be defined elsewhere)
objectDef = ensureDistinctType(rhs)
isRefObject = false
hasInlineFields = false
else:
error(macroName & " ref object must wrap a concrete object definition", rhs)
else:
## Non-object type / alias.
objectDef = ensureDistinctType(rhs)
isRefObject = false
hasInlineFields = false
if typeIdent.isNil():
error(macroName & " body must declare exactly one type", body)
result = ParsedBrokerType(
typeIdent: typeIdent,
objectDef: objectDef,
isRefObject: isRefObject,
hasInlineFields: hasInlineFields,
fieldNames: fieldNames,
fieldTypes: fieldTypes,
)

743
waku/common/broker/multi_request_broker.nim
View File

@ -1,743 +0,0 @@
## MultiRequestBroker
## --------------------
## MultiRequestBroker represents a proactive decoupling pattern, that
## allows defining request-response style interactions between modules without
## need for direct dependencies in between.
## Worth considering using it for use cases where you need to collect data from multiple providers.
##
## Generates a standalone, type-safe request broker for the declared type.
## The macro exports the value type itself plus a broker companion that manages
## providers via thread-local storage.
##
## Unlike `RequestBroker`, every call to `request` fan-outs to every registered
## provider and returns all collected responses.
## The request succeeds only if all providers succeed, otherwise it fails.
##
## Type definitions:
## - Inline `object` / `ref object` definitions are supported.
## - Native types, aliases, and externally-defined types are also supported.
## In that case, MultiRequestBroker will automatically wrap the declared RHS
## type in `distinct` unless you already used `distinct`.
## This keeps request types unique even when multiple brokers share the same
## underlying base type.
##
## Default vs. context aware use:
## Every generated broker is a thread-local global instance.
## Sometimes you want multiple independent provider sets for the same request
## type within the same thread (e.g. multiple components). For that, you can use
## context-aware MultiRequestBroker.
##
## Context awareness is supported through the `BrokerContext` argument for
## `setProvider`, `request`, `removeProvider`, and `clearProviders`.
## Provider stores are kept separate per broker context.
##
## Default broker context is defined as `DefaultBrokerContext`. If you don't
## need context awareness, you can keep using the interfaces without the context
## argument, which operate on `DefaultBrokerContext`.
##
## Usage:
##
## Declare collectable request data type inside a `MultiRequestBroker` macro, add any number of fields:
## ```nim
## MultiRequestBroker:
## type TypeName = object
## field1*: Type1
## field2*: Type2
##
## ## Define the request and provider signature, that is enforced at compile time.
## proc signature*(): Future[Result[TypeName, string]] {.async: (raises: []).}
##
## ## Also possible to define signature with arbitrary input arguments.
## proc signature*(arg1: ArgType, arg2: AnotherArgType): Future[Result[TypeName, string]] {.async: (raises: []).}
##
## ```
##
## You can register a request processor (provider) anywhere without the need to
## know who will request.
## Register provider functions with `TypeName.setProvider(...)`.
## Providers are async procs or lambdas that return `Future[Result[TypeName, string]]`.
## `setProvider` returns a handle (or an error) that can later be used to remove
## the provider.
## Requests can be made from anywhere with no direct dependency on the provider(s)
## by calling `TypeName.request()` (with arguments respecting the declared signature).
## This will asynchronously call all registered providers and return the collected
## responses as `Future[Result[seq[TypeName], string]]`.
##
## Whenever you don't want to process requests anymore (or your object instance that provides the request goes out of scope),
## you can remove it from the broker with `TypeName.removeProvider(handle)`.
## Alternatively, you can remove all registered providers through `TypeName.clearProviders()`.
##
## Example:
## ```nim
## MultiRequestBroker:
## type Greeting = object
## text*: string
##
## ## Define the request and provider signature, that is enforced at compile time.
## proc signature*(): Future[Result[Greeting, string]] {.async: (raises: []).}
##
## ## Also possible to define signature with arbitrary input arguments.
## proc signature*(lang: string): Future[Result[Greeting, string]] {.async: (raises: []).}
##
## ...
## let handle = Greeting.setProvider(
## proc(): Future[Result[Greeting, string]] {.async: (raises: []).} =
## ok(Greeting(text: "hello"))
## )
##
## let anotherHandle = Greeting.setProvider(
## proc(): Future[Result[Greeting, string]] {.async: (raises: []).} =
## ok(Greeting(text: "szia"))
## )
##
## let responses = (await Greeting.request()).valueOr(@[Greeting(text: "default")])
##
## echo responses.len
## Greeting.clearProviders()
## ```
## If no `signature` proc is declared, a zero-argument form is generated
## automatically, so the caller only needs to provide the type definition.
import std/[macros, strutils, tables, sugar]
import chronos
import results
import ./helper/broker_utils
import ./broker_context
export results, chronos, broker_context
proc isReturnTypeValid(returnType, typeIdent: NimNode): bool =
## Accept Future[Result[TypeIdent, string]] as the contract.
if returnType.kind != nnkBracketExpr or returnType.len != 2:
return false
if returnType[0].kind != nnkIdent or not returnType[0].eqIdent("Future"):
return false
let inner = returnType[1]
if inner.kind != nnkBracketExpr or inner.len != 3:
return false
if inner[0].kind != nnkIdent or not inner[0].eqIdent("Result"):
return false
if inner[1].kind != nnkIdent or not inner[1].eqIdent($typeIdent):
return false
inner[2].kind == nnkIdent and inner[2].eqIdent("string")
proc makeProcType(returnType: NimNode, params: seq[NimNode]): NimNode =
var formal = newTree(nnkFormalParams)
formal.add(returnType)
for param in params:
formal.add(param)
let pragmas = quote:
{.async.}
newTree(nnkProcTy, formal, pragmas)
macro MultiRequestBroker*(body: untyped): untyped =
when defined(requestBrokerDebug):
echo body.treeRepr
let parsed = parseSingleTypeDef(body, "MultiRequestBroker")
let typeIdent = parsed.typeIdent
let objectDef = parsed.objectDef
let isRefObject = parsed.isRefObject
when defined(requestBrokerDebug):
echo "MultiRequestBroker generating type: ", $typeIdent
let exportedTypeIdent = postfix(copyNimTree(typeIdent), "*")
let sanitized = sanitizeIdentName(typeIdent)
let typeNameLit = newLit($typeIdent)
let isRefObjectLit = newLit(isRefObject)
let uint64Ident = ident("uint64")
let providerKindIdent = ident(sanitized & "ProviderKind")
let providerHandleIdent = ident(sanitized & "ProviderHandle")
let exportedProviderHandleIdent = postfix(copyNimTree(providerHandleIdent), "*")
let bucketTypeIdent = ident(sanitized & "CtxBucket")
let findBucketIdxIdent = ident(sanitized & "FindBucketIdx")
let getOrCreateBucketIdxIdent = ident(sanitized & "GetOrCreateBucketIdx")
let zeroKindIdent = ident("pk" & sanitized & "NoArgs")
let argKindIdent = ident("pk" & sanitized & "WithArgs")
var zeroArgSig: NimNode = nil
var zeroArgProviderName: NimNode = nil
var zeroArgFieldName: NimNode = nil
var argSig: NimNode = nil
var argParams: seq[NimNode] = @[]
var argProviderName: NimNode = nil
var argFieldName: NimNode = nil
for stmt in body:
case stmt.kind
of nnkProcDef:
let procName = stmt[0]
let procNameIdent =
case procName.kind
of nnkIdent:
procName
of nnkPostfix:
procName[1]
else:
procName
let procNameStr = $procNameIdent
if not procNameStr.startsWith("signature"):
error("Signature proc names must start with `signature`", procName)
let params = stmt.params
if params.len == 0:
error("Signature must declare a return type", stmt)
let returnType = params[0]
if not isReturnTypeValid(returnType, typeIdent):
error(
"Signature must return Future[Result[`" & $typeIdent & "`, string]]", stmt
)
let paramCount = params.len - 1
if paramCount == 0:
if zeroArgSig != nil:
error("Only one zero-argument signature is allowed", stmt)
zeroArgSig = stmt
zeroArgProviderName = ident(sanitizeIdentName(typeIdent) & "ProviderNoArgs")
zeroArgFieldName = ident("providerNoArgs")
elif paramCount >= 1:
if argSig != nil:
error("Only one argument-based signature is allowed", stmt)
argSig = stmt
argParams = @[]
for idx in 1 ..< params.len:
let paramDef = params[idx]
if paramDef.kind != nnkIdentDefs:
error(
"Signature parameter must be a standard identifier declaration", paramDef
)
let paramTypeNode = paramDef[paramDef.len - 2]
if paramTypeNode.kind == nnkEmpty:
error("Signature parameter must declare a type", paramDef)
var hasName = false
for i in 0 ..< paramDef.len - 2:
if paramDef[i].kind != nnkEmpty:
hasName = true
if not hasName:
error("Signature parameter must declare a name", paramDef)
argParams.add(copyNimTree(paramDef))
argProviderName = ident(sanitizeIdentName(typeIdent) & "ProviderWithArgs")
argFieldName = ident("providerWithArgs")
of nnkTypeSection, nnkEmpty:
discard
else:
error("Unsupported statement inside MultiRequestBroker definition", stmt)
if zeroArgSig.isNil() and argSig.isNil():
zeroArgSig = newEmptyNode()
zeroArgProviderName = ident(sanitizeIdentName(typeIdent) & "ProviderNoArgs")
zeroArgFieldName = ident("providerNoArgs")
var typeSection = newTree(nnkTypeSection)
typeSection.add(newTree(nnkTypeDef, exportedTypeIdent, newEmptyNode(), objectDef))
var kindEnum = newTree(nnkEnumTy, newEmptyNode())
if not zeroArgSig.isNil():
kindEnum.add(zeroKindIdent)
if not argSig.isNil():
kindEnum.add(argKindIdent)
typeSection.add(newTree(nnkTypeDef, providerKindIdent, newEmptyNode(), kindEnum))
var handleRecList = newTree(nnkRecList)
handleRecList.add(newTree(nnkIdentDefs, ident("id"), uint64Ident, newEmptyNode()))
handleRecList.add(
newTree(nnkIdentDefs, ident("kind"), providerKindIdent, newEmptyNode())
)
typeSection.add(
newTree(
nnkTypeDef,
exportedProviderHandleIdent,
newEmptyNode(),
newTree(nnkObjectTy, newEmptyNode(), newEmptyNode(), handleRecList),
)
)
let returnType = quote:
Future[Result[`typeIdent`, string]]
if not zeroArgSig.isNil():
let procType = makeProcType(returnType, @[])
typeSection.add(newTree(nnkTypeDef, zeroArgProviderName, newEmptyNode(), procType))
if not argSig.isNil():
let procType = makeProcType(returnType, cloneParams(argParams))
typeSection.add(newTree(nnkTypeDef, argProviderName, newEmptyNode(), procType))
var bucketRecList = newTree(nnkRecList)
bucketRecList.add(
newTree(nnkIdentDefs, ident("brokerCtx"), ident("BrokerContext"), newEmptyNode())
)
if not zeroArgSig.isNil():
bucketRecList.add(
newTree(
nnkIdentDefs,
zeroArgFieldName,
newTree(nnkBracketExpr, ident("seq"), zeroArgProviderName),
newEmptyNode(),
)
)
if not argSig.isNil():
bucketRecList.add(
newTree(
nnkIdentDefs,
argFieldName,
newTree(nnkBracketExpr, ident("seq"), argProviderName),
newEmptyNode(),
)
)
typeSection.add(
newTree(
nnkTypeDef,
bucketTypeIdent,
newEmptyNode(),
newTree(nnkObjectTy, newEmptyNode(), newEmptyNode(), bucketRecList),
)
)
var brokerRecList = newTree(nnkRecList)
brokerRecList.add(
newTree(
nnkIdentDefs,
ident("buckets"),
newTree(nnkBracketExpr, ident("seq"), bucketTypeIdent),
newEmptyNode(),
)
)
let brokerTypeIdent = ident(sanitizeIdentName(typeIdent) & "Broker")
typeSection.add(
newTree(
nnkTypeDef,
brokerTypeIdent,
newEmptyNode(),
newTree(
nnkRefTy, newTree(nnkObjectTy, newEmptyNode(), newEmptyNode(), brokerRecList)
),
)
)
result = newStmtList()
result.add(typeSection)
let globalVarIdent = ident("g" & sanitizeIdentName(typeIdent) & "Broker")
let accessProcIdent = ident("access" & sanitizeIdentName(typeIdent) & "Broker")
result.add(
quote do:
var `globalVarIdent` {.threadvar.}: `brokerTypeIdent`
proc `findBucketIdxIdent`(
broker: `brokerTypeIdent`, brokerCtx: BrokerContext
): int =
if brokerCtx == DefaultBrokerContext:
return 0
for i in 1 ..< broker.buckets.len:
if broker.buckets[i].brokerCtx == brokerCtx:
return i
return -1
proc `getOrCreateBucketIdxIdent`(
broker: `brokerTypeIdent`, brokerCtx: BrokerContext
): int =
let idx = `findBucketIdxIdent`(broker, brokerCtx)
if idx >= 0:
return idx
broker.buckets.add(`bucketTypeIdent`(brokerCtx: brokerCtx))
return broker.buckets.high
proc `accessProcIdent`(): `brokerTypeIdent` =
if `globalVarIdent`.isNil():
new(`globalVarIdent`)
`globalVarIdent`.buckets =
@[`bucketTypeIdent`(brokerCtx: DefaultBrokerContext)]
return `globalVarIdent`
)
var clearBody = newStmtList()
if not zeroArgSig.isNil():
result.add(
quote do:
proc setProvider*(
_: typedesc[`typeIdent`],
brokerCtx: BrokerContext,
handler: `zeroArgProviderName`,
): Result[`providerHandleIdent`, string] =
if handler.isNil():
return err("Provider handler must be provided")
let broker = `accessProcIdent`()
let bucketIdx = `getOrCreateBucketIdxIdent`(broker, brokerCtx)
for i, existing in broker.buckets[bucketIdx].`zeroArgFieldName`:
if not existing.isNil() and existing == handler:
return ok(`providerHandleIdent`(id: uint64(i + 1), kind: `zeroKindIdent`))
broker.buckets[bucketIdx].`zeroArgFieldName`.add(handler)
return ok(
`providerHandleIdent`(
id: uint64(broker.buckets[bucketIdx].`zeroArgFieldName`.len),
kind: `zeroKindIdent`,
)
)
proc setProvider*(
_: typedesc[`typeIdent`], handler: `zeroArgProviderName`
): Result[`providerHandleIdent`, string] =
return setProvider(`typeIdent`, DefaultBrokerContext, handler)
)
result.add(
quote do:
proc request*(
_: typedesc[`typeIdent`], brokerCtx: BrokerContext
): Future[Result[seq[`typeIdent`], string]] {.async: (raises: []), gcsafe.} =
var aggregated: seq[`typeIdent`] = @[]
let broker = `accessProcIdent`()
let bucketIdx = `findBucketIdxIdent`(broker, brokerCtx)
if bucketIdx < 0:
return ok(aggregated)
let providers = broker.buckets[bucketIdx].`zeroArgFieldName`
if providers.len == 0:
return ok(aggregated)
# var providersFut: seq[Future[Result[`typeIdent`, string]]] = collect:
var providersFut = collect(newSeq):
for provider in providers:
if provider.isNil():
continue
provider()
let catchable = catch:
await allFinished(providersFut)
catchable.isOkOr:
return err("Some provider(s) failed:" & error.msg)
for fut in catchable.get():
if fut.failed():
return err("Some provider(s) failed:" & fut.error.msg)
elif fut.finished():
let providerResult = fut.value()
if providerResult.isOk:
let providerValue = providerResult.get()
when `isRefObjectLit`:
if providerValue.isNil():
return err(
"MultiRequestBroker(" & `typeNameLit` &
"): provider returned nil result"
)
aggregated.add(providerValue)
else:
return err("Some provider(s) failed:" & providerResult.error)
return ok(aggregated)
proc request*(
_: typedesc[`typeIdent`]
): Future[Result[seq[`typeIdent`], string]] =
return request(`typeIdent`, DefaultBrokerContext)
)
if not argSig.isNil():
result.add(
quote do:
proc setProvider*(
_: typedesc[`typeIdent`],
brokerCtx: BrokerContext,
handler: `argProviderName`,
): Result[`providerHandleIdent`, string] =
if handler.isNil():
return err("Provider handler must be provided")
let broker = `accessProcIdent`()
let bucketIdx = `getOrCreateBucketIdxIdent`(broker, brokerCtx)
for i, existing in broker.buckets[bucketIdx].`argFieldName`:
if not existing.isNil() and existing == handler:
return ok(`providerHandleIdent`(id: uint64(i + 1), kind: `argKindIdent`))
broker.buckets[bucketIdx].`argFieldName`.add(handler)
return ok(
`providerHandleIdent`(
id: uint64(broker.buckets[bucketIdx].`argFieldName`.len),
kind: `argKindIdent`,
)
)
proc setProvider*(
_: typedesc[`typeIdent`], handler: `argProviderName`
): Result[`providerHandleIdent`, string] =
return setProvider(`typeIdent`, DefaultBrokerContext, handler)
)
let requestParamDefs = cloneParams(argParams)
let argNameIdents = collectParamNames(requestParamDefs)
let providerSym = genSym(nskLet, "providerVal")
var providerCall = newCall(providerSym)
for argName in argNameIdents:
providerCall.add(argName)
var formalParams = newTree(nnkFormalParams)
formalParams.add(
quote do:
Future[Result[seq[`typeIdent`], string]]
)
formalParams.add(
newTree(
nnkIdentDefs,
ident("_"),
newTree(nnkBracketExpr, ident("typedesc"), copyNimTree(typeIdent)),
newEmptyNode(),
)
)
formalParams.add(
newTree(nnkIdentDefs, ident("brokerCtx"), ident("BrokerContext"), newEmptyNode())
)
for paramDef in requestParamDefs:
formalParams.add(paramDef)
let requestPragmas = quote:
{.async: (raises: []), gcsafe.}
let requestBody = quote:
var aggregated: seq[`typeIdent`] = @[]
let broker = `accessProcIdent`()
let bucketIdx = `findBucketIdxIdent`(broker, brokerCtx)
if bucketIdx < 0:
return ok(aggregated)
let providers = broker.buckets[bucketIdx].`argFieldName`
if providers.len == 0:
return ok(aggregated)
var providersFut = collect(newSeq):
for provider in providers:
if provider.isNil():
continue
let `providerSym` = provider
`providerCall`
let catchable = catch:
await allFinished(providersFut)
catchable.isOkOr:
return err("Some provider(s) failed:" & error.msg)
for fut in catchable.get():
if fut.failed():
return err("Some provider(s) failed:" & fut.error.msg)
elif fut.finished():
let providerResult = fut.value()
if providerResult.isOk:
let providerValue = providerResult.get()
when `isRefObjectLit`:
if providerValue.isNil():
return err(
"MultiRequestBroker(" & `typeNameLit` &
"): provider returned nil result"
)
aggregated.add(providerValue)
else:
return err("Some provider(s) failed:" & providerResult.error)
return ok(aggregated)
result.add(
newTree(
nnkProcDef,
postfix(ident("request"), "*"),
newEmptyNode(),
newEmptyNode(),
formalParams,
requestPragmas,
newEmptyNode(),
requestBody,
)
)
# Backward-compatible default-context overload (no brokerCtx parameter).
var formalParamsDefault = newTree(nnkFormalParams)
formalParamsDefault.add(
quote do:
Future[Result[seq[`typeIdent`], string]]
)
formalParamsDefault.add(
newTree(
nnkIdentDefs,
ident("_"),
newTree(nnkBracketExpr, ident("typedesc"), copyNimTree(typeIdent)),
newEmptyNode(),
)
)
for paramDef in requestParamDefs:
formalParamsDefault.add(copyNimTree(paramDef))
var wrapperCall = newCall(ident("request"))
wrapperCall.add(copyNimTree(typeIdent))
wrapperCall.add(ident("DefaultBrokerContext"))
for argName in argNameIdents:
wrapperCall.add(copyNimTree(argName))
result.add(
newTree(
nnkProcDef,
postfix(ident("request"), "*"),
newEmptyNode(),
newEmptyNode(),
formalParamsDefault,
newEmptyNode(),
newEmptyNode(),
newStmtList(newTree(nnkReturnStmt, wrapperCall)),
)
)
let removeHandleCtxSym = genSym(nskParam, "handle")
let removeHandleDefaultSym = genSym(nskParam, "handle")
when true:
# Generate clearProviders / removeProvider with macro-time knowledge about which
# provider lists exist (zero-arg and/or arg providers).
if not zeroArgSig.isNil() and not argSig.isNil():
result.add(
quote do:
proc clearProviders*(_: typedesc[`typeIdent`], brokerCtx: BrokerContext) =
let broker = `accessProcIdent`()
if broker.isNil():
return
let bucketIdx = `findBucketIdxIdent`(broker, brokerCtx)
if bucketIdx < 0:
return
broker.buckets[bucketIdx].`zeroArgFieldName`.setLen(0)
broker.buckets[bucketIdx].`argFieldName`.setLen(0)
if brokerCtx != DefaultBrokerContext:
broker.buckets.delete(bucketIdx)
proc clearProviders*(_: typedesc[`typeIdent`]) =
clearProviders(`typeIdent`, DefaultBrokerContext)
proc removeProvider*(
_: typedesc[`typeIdent`],
brokerCtx: BrokerContext,
`removeHandleCtxSym`: `providerHandleIdent`,
) =
if `removeHandleCtxSym`.id == 0'u64:
return
let broker = `accessProcIdent`()
if broker.isNil():
return
let bucketIdx = `findBucketIdxIdent`(broker, brokerCtx)
if bucketIdx < 0:
return
if `removeHandleCtxSym`.kind == `zeroKindIdent`:
let idx = int(`removeHandleCtxSym`.id) - 1
if idx >= 0 and idx < broker.buckets[bucketIdx].`zeroArgFieldName`.len:
broker.buckets[bucketIdx].`zeroArgFieldName`[idx] = nil
elif `removeHandleCtxSym`.kind == `argKindIdent`:
let idx = int(`removeHandleCtxSym`.id) - 1
if idx >= 0 and idx < broker.buckets[bucketIdx].`argFieldName`.len:
broker.buckets[bucketIdx].`argFieldName`[idx] = nil
if brokerCtx != DefaultBrokerContext:
var hasAny = false
for p in broker.buckets[bucketIdx].`zeroArgFieldName`:
if not p.isNil():
hasAny = true
break
if not hasAny:
for p in broker.buckets[bucketIdx].`argFieldName`:
if not p.isNil():
hasAny = true
break
if not hasAny:
broker.buckets.delete(bucketIdx)
proc removeProvider*(
_: typedesc[`typeIdent`], `removeHandleDefaultSym`: `providerHandleIdent`
) =
removeProvider(`typeIdent`, DefaultBrokerContext, `removeHandleDefaultSym`)
)
elif not zeroArgSig.isNil():
result.add(
quote do:
proc clearProviders*(_: typedesc[`typeIdent`], brokerCtx: BrokerContext) =
let broker = `accessProcIdent`()
if broker.isNil():
return
let bucketIdx = `findBucketIdxIdent`(broker, brokerCtx)
if bucketIdx < 0:
return
broker.buckets[bucketIdx].`zeroArgFieldName`.setLen(0)
if brokerCtx != DefaultBrokerContext:
broker.buckets.delete(bucketIdx)
proc clearProviders*(_: typedesc[`typeIdent`]) =
clearProviders(`typeIdent`, DefaultBrokerContext)
proc removeProvider*(
_: typedesc[`typeIdent`],
brokerCtx: BrokerContext,
`removeHandleCtxSym`: `providerHandleIdent`,
) =
if `removeHandleCtxSym`.id == 0'u64:
return
let broker = `accessProcIdent`()
if broker.isNil():
return
let bucketIdx = `findBucketIdxIdent`(broker, brokerCtx)
if bucketIdx < 0:
return
if `removeHandleCtxSym`.kind != `zeroKindIdent`:
return
let idx = int(`removeHandleCtxSym`.id) - 1
if idx >= 0 and idx < broker.buckets[bucketIdx].`zeroArgFieldName`.len:
broker.buckets[bucketIdx].`zeroArgFieldName`[idx] = nil
if brokerCtx != DefaultBrokerContext:
var hasAny = false
for p in broker.buckets[bucketIdx].`zeroArgFieldName`:
if not p.isNil():
hasAny = true
break
if not hasAny:
broker.buckets.delete(bucketIdx)
proc removeProvider*(
_: typedesc[`typeIdent`], `removeHandleDefaultSym`: `providerHandleIdent`
) =
removeProvider(`typeIdent`, DefaultBrokerContext, `removeHandleDefaultSym`)
)
else:
result.add(
quote do:
proc clearProviders*(_: typedesc[`typeIdent`], brokerCtx: BrokerContext) =
let broker = `accessProcIdent`()
if broker.isNil():
return
let bucketIdx = `findBucketIdxIdent`(broker, brokerCtx)
if bucketIdx < 0:
return
broker.buckets[bucketIdx].`argFieldName`.setLen(0)
if brokerCtx != DefaultBrokerContext:
broker.buckets.delete(bucketIdx)
proc clearProviders*(_: typedesc[`typeIdent`]) =
clearProviders(`typeIdent`, DefaultBrokerContext)
proc removeProvider*(
_: typedesc[`typeIdent`],
brokerCtx: BrokerContext,
`removeHandleCtxSym`: `providerHandleIdent`,
) =
if `removeHandleCtxSym`.id == 0'u64:
return
let broker = `accessProcIdent`()
if broker.isNil():
return
let bucketIdx = `findBucketIdxIdent`(broker, brokerCtx)
if bucketIdx < 0:
return
if `removeHandleCtxSym`.kind != `argKindIdent`:
return
let idx = int(`removeHandleCtxSym`.id) - 1
if idx >= 0 and idx < broker.buckets[bucketIdx].`argFieldName`.len:
broker.buckets[bucketIdx].`argFieldName`[idx] = nil
if brokerCtx != DefaultBrokerContext:
var hasAny = false
for p in broker.buckets[bucketIdx].`argFieldName`:
if not p.isNil():
hasAny = true
break
if not hasAny:
broker.buckets.delete(bucketIdx)
proc removeProvider*(
_: typedesc[`typeIdent`], `removeHandleDefaultSym`: `providerHandleIdent`
) =
removeProvider(`typeIdent`, DefaultBrokerContext, `removeHandleDefaultSym`)
)
when defined(requestBrokerDebug):
echo result.repr

841
waku/common/broker/request_broker.nim
View File

@ -1,841 +0,0 @@
## RequestBroker
## --------------------
## RequestBroker represents a proactive decoupling pattern, that
## allows defining request-response style interactions between modules without
## need for direct dependencies in between.
## Worth considering using it in a single provider, many requester scenario.
##
## Provides a declarative way to define an immutable value type together with a
## thread-local broker that can register an asynchronous or synchronous provider,
## dispatch typed requests and clear provider.
##
## For consideration use `sync` mode RequestBroker when you need to provide simple value(s)
## where there is no long-running async operation involved.
## Typically it act as a accessor for the local state of generic setting.
##
## `async` mode is better to be used when you request date that may involve some long IO operation
## or action.
##
## Default vs. context aware use:
## Every generated broker is a thread-local global instance. This means each RequestBroker enables decoupled
## data exchange threadwise. Sometimes we use brokers inside a context - like inside a component that has many modules or subsystems.
## In case you would instantiate multiple such components in a single thread, and each component must has its own provider for the same RequestBroker type,
## in order to avoid provider collision, you can use context aware RequestBroker.
## Context awareness is supported through the `BrokerContext` argument for `setProvider`, `request`, `clearProvider` interfaces.
## Suce use requires generating a new unique `BrokerContext` value per component instance, and spread it to all modules using the brokers.
## Example, store the `BrokerContext` as a field inside the top level component instance, and spread around at initialization of the subcomponents..
##
## Default broker context is defined as `DefaultBrokerContext` constant. But if you don't need context awareness, you can use the
## interfaces without context argument.
##
## Usage:
## Declare your desired request type inside a `RequestBroker` macro, add any number of fields.
## Define the provider signature, that is enforced at compile time.
##
## ```nim
## RequestBroker:
## type TypeName = object
## field1*: FieldType
## field2*: AnotherFieldType
##
## proc signature*(): Future[Result[TypeName, string]]
## ## Also possible to define signature with arbitrary input arguments.
## proc signature*(arg1: ArgType, arg2: AnotherArgType): Future[Result[TypeName, string]]
##
## ```
##
## Sync mode (no `async` / `Future`) can be generated with:
##
## ```nim
## RequestBroker(sync):
## type TypeName = object
## field1*: FieldType
##
## proc signature*(): Result[TypeName, string]
## proc signature*(arg1: ArgType): Result[TypeName, string]
## ```
##
## Note: When the request type is declared as a native type / alias / externally-defined
## type (i.e. not an inline `object` / `ref object` definition), RequestBroker
## will wrap it in `distinct` automatically unless you already used `distinct`.
## This avoids overload ambiguity when multiple brokers share the same
## underlying base type (Nim overload resolution does not consider return type).
##
## This means that for non-object request types you typically:
## - construct values with an explicit cast/constructor, e.g. `MyType("x")`
## - unwrap with a cast when needed, e.g. `string(myVal)` or `BaseType(myVal)`
##
## Example (native response type):
## ```nim
## RequestBroker(sync):
## type MyCount = int # exported as: `distinct int`
##
## MyCount.setProvider(proc(): Result[MyCount, string] = ok(MyCount(42)))
## let res = MyCount.request()
## if res.isOk():
## let raw = int(res.get())
## ```
##
## Example (externally-defined type):
## ```nim
## type External = object
## label*: string
##
## RequestBroker:
## type MyExternal = External # exported as: `distinct External`
##
## MyExternal.setProvider(
## proc(): Future[Result[MyExternal, string]] {.async.} =
## ok(MyExternal(External(label: "hi")))
## )
## let res = await MyExternal.request()
## if res.isOk():
## let base = External(res.get())
## echo base.label
## ```
## The 'TypeName' object defines the requestable data (but also can be seen as request for action with return value).
## The 'signature' proc defines the provider(s) signature, that is enforced at compile time.
## One signature can be with no arguments, another with any number of arguments - where the input arguments are
## not related to the request type - but alternative inputs for the request to be processed.
##
## After this, you can register a provider anywhere in your code with
## `TypeName.setProvider(...)`, which returns error if already having a provider.
## Providers are async procs/lambdas in default mode and sync procs in sync mode.
##
## Providers are stored as a broker-context keyed list:
## - the default provider is always stored at index 0 (reserved broker context: 0)
## - additional providers can be registered under arbitrary non-zero broker contexts
##
## The original `setProvider(handler)` / `request(...)` APIs continue to operate
## on the default provider (broker context 0) for backward compatibility.
##
## Requests can be made from anywhere with no direct dependency on the provider by
## calling `TypeName.request()` - with arguments respecting the signature(s).
## In async mode, this returns a Future[Result[TypeName, string]]. In sync mode, it returns Result[TypeName, string].
##
## Whenever you no want to process requests (or your object instance that provides the request goes out of scope),
## you can remove it from the broker with `TypeName.clearProvider()`.
##
##
## Example:
## ```nim
## RequestBroker:
## type Greeting = object
## text*: string
##
## ## Define the request and provider signature, that is enforced at compile time.
## proc signature*(): Future[Result[Greeting, string]] {.async.}
##
## ## Also possible to define signature with arbitrary input arguments.
## proc signature*(lang: string): Future[Result[Greeting, string]] {.async.}
##
## ...
## Greeting.setProvider(
## proc(): Future[Result[Greeting, string]] {.async.} =
## ok(Greeting(text: "hello"))
## )
## let res = await Greeting.request()
##
##
## ...
## # using native type as response for a synchronous request.
## RequestBroker(sync):
## type NeedThatInfo = string
##
##...
## NeedThatInfo.setProvider(
## proc(): Result[NeedThatInfo, string] =
## ok("this is the info you wanted")
## )
## let res = NeedThatInfo.request().valueOr:
## echo "not ok due to: " & error
## NeedThatInfo(":-(")
##
## echo string(res)
## ```
## If no `signature` proc is declared, a zero-argument form is generated
## automatically, so the caller only needs to provide the type definition.
import std/[macros, strutils]
from std/sequtils import keepItIf
import chronos
import results
import ./helper/broker_utils, broker_context
export results, chronos, keepItIf, broker_context
proc errorFuture[T](message: string): Future[Result[T, string]] {.inline.} =
## Build a future that is already completed with an error result.
let fut = newFuture[Result[T, string]]("request_broker.errorFuture")
fut.complete(err(Result[T, string], message))
fut
type RequestBrokerMode = enum
rbAsync
rbSync
proc isAsyncReturnTypeValid(returnType, typeIdent: NimNode): bool =
## Accept Future[Result[TypeIdent, string]] as the contract.
if returnType.kind != nnkBracketExpr or returnType.len != 2:
return false
if returnType[0].kind != nnkIdent or not returnType[0].eqIdent("Future"):
return false
let inner = returnType[1]
if inner.kind != nnkBracketExpr or inner.len != 3:
return false
if inner[0].kind != nnkIdent or not inner[0].eqIdent("Result"):
return false
if inner[1].kind != nnkIdent or not inner[1].eqIdent($typeIdent):
return false
inner[2].kind == nnkIdent and inner[2].eqIdent("string")
proc isSyncReturnTypeValid(returnType, typeIdent: NimNode): bool =
## Accept Result[TypeIdent, string] as the contract.
if returnType.kind != nnkBracketExpr or returnType.len != 3:
return false
if returnType[0].kind != nnkIdent or not returnType[0].eqIdent("Result"):
return false
if returnType[1].kind != nnkIdent or not returnType[1].eqIdent($typeIdent):
return false
returnType[2].kind == nnkIdent and returnType[2].eqIdent("string")
proc isReturnTypeValid(returnType, typeIdent: NimNode, mode: RequestBrokerMode): bool =
case mode
of rbAsync:
isAsyncReturnTypeValid(returnType, typeIdent)
of rbSync:
isSyncReturnTypeValid(returnType, typeIdent)
proc makeProcType(
returnType: NimNode, params: seq[NimNode], mode: RequestBrokerMode
): NimNode =
var formal = newTree(nnkFormalParams)
formal.add(returnType)
for param in params:
formal.add(param)
case mode
of rbAsync:
let pragmas = newTree(nnkPragma, ident("async"))
newTree(nnkProcTy, formal, pragmas)
of rbSync:
let raisesPragma = newTree(
nnkExprColonExpr, ident("raises"), newTree(nnkBracket, ident("CatchableError"))
)
let pragmas = newTree(nnkPragma, raisesPragma, ident("gcsafe"))
newTree(nnkProcTy, formal, pragmas)
proc parseMode(modeNode: NimNode): RequestBrokerMode =
## Parses the mode selector for the 2-argument macro overload.
## Supported spellings: `sync` / `async` (case-insensitive).
let raw = ($modeNode).strip().toLowerAscii()
case raw
of "sync":
rbSync
of "async":
rbAsync
else:
error("RequestBroker mode must be `sync` or `async` (default is async)", modeNode)
proc generateRequestBroker(body: NimNode, mode: RequestBrokerMode): NimNode =
when defined(requestBrokerDebug):
echo body.treeRepr
echo "RequestBroker mode: ", $mode
let parsed = parseSingleTypeDef(body, "RequestBroker", allowRefToNonObject = true)
let typeIdent = parsed.typeIdent
let objectDef = parsed.objectDef
when defined(requestBrokerDebug):
echo "RequestBroker generating type: ", $typeIdent
let exportedTypeIdent = postfix(copyNimTree(typeIdent), "*")
let typeDisplayName = sanitizeIdentName(typeIdent)
let typeNameLit = newLit(typeDisplayName)
var zeroArgSig: NimNode = nil
var zeroArgProviderName: NimNode = nil
var argSig: NimNode = nil
var argParams: seq[NimNode] = @[]
var argProviderName: NimNode = nil
for stmt in body:
case stmt.kind
of nnkProcDef:
let procName = stmt[0]
let procNameIdent =
case procName.kind
of nnkIdent:
procName
of nnkPostfix:
procName[1]
else:
procName
let procNameStr = $procNameIdent
if not procNameStr.startsWith("signature"):
error("Signature proc names must start with `signature`", procName)
let params = stmt.params
if params.len == 0:
error("Signature must declare a return type", stmt)
let returnType = params[0]
if not isReturnTypeValid(returnType, typeIdent, mode):
case mode
of rbAsync:
error(
"Signature must return Future[Result[`" & $typeIdent & "`, string]]", stmt
)
of rbSync:
error("Signature must return Result[`" & $typeIdent & "`, string]", stmt)
let paramCount = params.len - 1
if paramCount == 0:
if zeroArgSig != nil:
error("Only one zero-argument signature is allowed", stmt)
zeroArgSig = stmt
zeroArgProviderName = ident(sanitizeIdentName(typeIdent) & "ProviderNoArgs")
elif paramCount >= 1:
if argSig != nil:
error("Only one argument-based signature is allowed", stmt)
argSig = stmt
argParams = @[]
for idx in 1 ..< params.len:
let paramDef = params[idx]
if paramDef.kind != nnkIdentDefs:
error(
"Signature parameter must be a standard identifier declaration", paramDef
)
let paramTypeNode = paramDef[paramDef.len - 2]
if paramTypeNode.kind == nnkEmpty:
error("Signature parameter must declare a type", paramDef)
var hasName = false
for i in 0 ..< paramDef.len - 2:
if paramDef[i].kind != nnkEmpty:
hasName = true
if not hasName:
error("Signature parameter must declare a name", paramDef)
argParams.add(copyNimTree(paramDef))
argProviderName = ident(sanitizeIdentName(typeIdent) & "ProviderWithArgs")
of nnkTypeSection, nnkEmpty:
discard
else:
error("Unsupported statement inside RequestBroker definition", stmt)
if zeroArgSig.isNil() and argSig.isNil():
zeroArgSig = newEmptyNode()
zeroArgProviderName = ident(sanitizeIdentName(typeIdent) & "ProviderNoArgs")
var typeSection = newTree(nnkTypeSection)
typeSection.add(newTree(nnkTypeDef, exportedTypeIdent, newEmptyNode(), objectDef))
let returnType =
case mode
of rbAsync:
quote:
Future[Result[`typeIdent`, string]]
of rbSync:
quote:
Result[`typeIdent`, string]
if not zeroArgSig.isNil():
let procType = makeProcType(returnType, @[], mode)
typeSection.add(newTree(nnkTypeDef, zeroArgProviderName, newEmptyNode(), procType))
if not argSig.isNil():
let procType = makeProcType(returnType, cloneParams(argParams), mode)
typeSection.add(newTree(nnkTypeDef, argProviderName, newEmptyNode(), procType))
var brokerRecList = newTree(nnkRecList)
if not zeroArgSig.isNil():
let zeroArgProvidersFieldName = ident("providersNoArgs")
let zeroArgProvidersTupleTy = newTree(
nnkTupleTy,
newTree(nnkIdentDefs, ident("brokerCtx"), ident("BrokerContext"), newEmptyNode()),
newTree(nnkIdentDefs, ident("handler"), zeroArgProviderName, newEmptyNode()),
)
let zeroArgProvidersSeqTy =
newTree(nnkBracketExpr, ident("seq"), zeroArgProvidersTupleTy)
brokerRecList.add(
newTree(
nnkIdentDefs, zeroArgProvidersFieldName, zeroArgProvidersSeqTy, newEmptyNode()
)
)
if not argSig.isNil():
let argProvidersFieldName = ident("providersWithArgs")
let argProvidersTupleTy = newTree(
nnkTupleTy,
newTree(nnkIdentDefs, ident("brokerCtx"), ident("BrokerContext"), newEmptyNode()),
newTree(nnkIdentDefs, ident("handler"), argProviderName, newEmptyNode()),
)
let argProvidersSeqTy = newTree(nnkBracketExpr, ident("seq"), argProvidersTupleTy)
brokerRecList.add(
newTree(nnkIdentDefs, argProvidersFieldName, argProvidersSeqTy, newEmptyNode())
)
let brokerTypeIdent = ident(sanitizeIdentName(typeIdent) & "Broker")
let brokerTypeDef = newTree(
nnkTypeDef,
brokerTypeIdent,
newEmptyNode(),
newTree(nnkObjectTy, newEmptyNode(), newEmptyNode(), brokerRecList),
)
typeSection.add(brokerTypeDef)
result = newStmtList()
result.add(typeSection)
let globalVarIdent = ident("g" & sanitizeIdentName(typeIdent) & "Broker")
let accessProcIdent = ident("access" & sanitizeIdentName(typeIdent) & "Broker")
var brokerNewBody = newStmtList()
if not zeroArgSig.isNil():
brokerNewBody.add(
quote do:
result.providersNoArgs =
@[(brokerCtx: DefaultBrokerContext, handler: default(`zeroArgProviderName`))]
)
if not argSig.isNil():
brokerNewBody.add(
quote do:
result.providersWithArgs =
@[(brokerCtx: DefaultBrokerContext, handler: default(`argProviderName`))]
)
var brokerInitChecks = newStmtList()
if not zeroArgSig.isNil():
brokerInitChecks.add(
quote do:
if `globalVarIdent`.providersNoArgs.len == 0:
`globalVarIdent` = `brokerTypeIdent`.new()
)
if not argSig.isNil():
brokerInitChecks.add(
quote do:
if `globalVarIdent`.providersWithArgs.len == 0:
`globalVarIdent` = `brokerTypeIdent`.new()
)
result.add(
quote do:
var `globalVarIdent` {.threadvar.}: `brokerTypeIdent`
proc new(_: type `brokerTypeIdent`): `brokerTypeIdent` =
result = `brokerTypeIdent`()
`brokerNewBody`
proc `accessProcIdent`(): var `brokerTypeIdent` =
`brokerInitChecks`
`globalVarIdent`
)
var clearBodyKeyed = newStmtList()
let brokerCtxParamIdent = ident("brokerCtx")
if not zeroArgSig.isNil():
let zeroArgProvidersFieldName = ident("providersNoArgs")
result.add(
quote do:
proc setProvider*(
_: typedesc[`typeIdent`], handler: `zeroArgProviderName`
): Result[void, string] =
if not `accessProcIdent`().`zeroArgProvidersFieldName`[0].handler.isNil():
return err("Zero-arg provider already set")
`accessProcIdent`().`zeroArgProvidersFieldName`[0].handler = handler
return ok()
)
result.add(
quote do:
proc setProvider*(
_: typedesc[`typeIdent`],
brokerCtx: BrokerContext,
handler: `zeroArgProviderName`,
): Result[void, string] =
if brokerCtx == DefaultBrokerContext:
return setProvider(`typeIdent`, handler)
for entry in `accessProcIdent`().`zeroArgProvidersFieldName`:
if entry.brokerCtx == brokerCtx:
return err(
"RequestBroker(" & `typeNameLit` &
"): provider already set for broker context " & $brokerCtx
)
`accessProcIdent`().`zeroArgProvidersFieldName`.add(
(brokerCtx: brokerCtx, handler: handler)
)
return ok()
)
clearBodyKeyed.add(
quote do:
if `brokerCtxParamIdent` == DefaultBrokerContext:
`accessProcIdent`().`zeroArgProvidersFieldName`[0].handler =
default(`zeroArgProviderName`)
else:
`accessProcIdent`().`zeroArgProvidersFieldName`.keepItIf(
it.brokerCtx != `brokerCtxParamIdent`
)
)
case mode
of rbAsync:
result.add(
quote do:
proc request*(
_: typedesc[`typeIdent`]
): Future[Result[`typeIdent`, string]] {.async: (raises: []).} =
return await request(`typeIdent`, DefaultBrokerContext)
)
result.add(
quote do:
proc request*(
_: typedesc[`typeIdent`], brokerCtx: BrokerContext
): Future[Result[`typeIdent`, string]] {.async: (raises: []).} =
var provider: `zeroArgProviderName`
if brokerCtx == DefaultBrokerContext:
provider = `accessProcIdent`().`zeroArgProvidersFieldName`[0].handler
else:
for entry in `accessProcIdent`().`zeroArgProvidersFieldName`:
if entry.brokerCtx == brokerCtx:
provider = entry.handler
break
if provider.isNil():
if brokerCtx == DefaultBrokerContext:
return err(
"RequestBroker(" & `typeNameLit` & "): no zero-arg provider registered"
)
return err(
"RequestBroker(" & `typeNameLit` &
"): no provider registered for broker context " & $brokerCtx
)
let catchedRes = catch:
await provider()
if catchedRes.isErr():
return err(
"RequestBroker(" & `typeNameLit` & "): provider threw exception: " &
catchedRes.error.msg
)
let providerRes = catchedRes.get()
if providerRes.isOk():
let resultValue = providerRes.get()
when compiles(resultValue.isNil()):
if resultValue.isNil():
return err(
"RequestBroker(" & `typeNameLit` & "): provider returned nil result"
)
return providerRes
)
of rbSync:
result.add(
quote do:
proc request*(
_: typedesc[`typeIdent`]
): Result[`typeIdent`, string] {.gcsafe, raises: [].} =
return request(`typeIdent`, DefaultBrokerContext)
)
result.add(
quote do:
proc request*(
_: typedesc[`typeIdent`], brokerCtx: BrokerContext
): Result[`typeIdent`, string] {.gcsafe, raises: [].} =
var provider: `zeroArgProviderName`
if brokerCtx == DefaultBrokerContext:
provider = `accessProcIdent`().`zeroArgProvidersFieldName`[0].handler
else:
for entry in `accessProcIdent`().`zeroArgProvidersFieldName`:
if entry.brokerCtx == brokerCtx:
provider = entry.handler
break
if provider.isNil():
if brokerCtx == DefaultBrokerContext:
return err(
"RequestBroker(" & `typeNameLit` & "): no zero-arg provider registered"
)
return err(
"RequestBroker(" & `typeNameLit` &
"): no provider registered for broker context " & $brokerCtx
)
var providerRes: Result[`typeIdent`, string]
try:
providerRes = provider()
except CatchableError as e:
return err(
"RequestBroker(" & `typeNameLit` & "): provider threw exception: " &
e.msg
)
if providerRes.isOk():
let resultValue = providerRes.get()
when compiles(resultValue.isNil()):
if resultValue.isNil():
return err(
"RequestBroker(" & `typeNameLit` & "): provider returned nil result"
)
return providerRes
)
if not argSig.isNil():
let argProvidersFieldName = ident("providersWithArgs")
result.add(
quote do:
proc setProvider*(
_: typedesc[`typeIdent`], handler: `argProviderName`
): Result[void, string] =
if not `accessProcIdent`().`argProvidersFieldName`[0].handler.isNil():
return err("Provider already set")
`accessProcIdent`().`argProvidersFieldName`[0].handler = handler
return ok()
)
result.add(
quote do:
proc setProvider*(
_: typedesc[`typeIdent`],
brokerCtx: BrokerContext,
handler: `argProviderName`,
): Result[void, string] =
if brokerCtx == DefaultBrokerContext:
return setProvider(`typeIdent`, handler)
for entry in `accessProcIdent`().`argProvidersFieldName`:
if entry.brokerCtx == brokerCtx:
return err(
"RequestBroker(" & `typeNameLit` &
"): provider already set for broker context " & $brokerCtx
)
`accessProcIdent`().`argProvidersFieldName`.add(
(brokerCtx: brokerCtx, handler: handler)
)
return ok()
)
clearBodyKeyed.add(
quote do:
if `brokerCtxParamIdent` == DefaultBrokerContext:
`accessProcIdent`().`argProvidersFieldName`[0].handler =
default(`argProviderName`)
else:
`accessProcIdent`().`argProvidersFieldName`.keepItIf(
it.brokerCtx != `brokerCtxParamIdent`
)
)
let requestParamDefs = cloneParams(argParams)
let argNameIdents = collectParamNames(requestParamDefs)
var formalParams = newTree(nnkFormalParams)
formalParams.add(copyNimTree(returnType))
formalParams.add(
newTree(
nnkIdentDefs,
ident("_"),
newTree(nnkBracketExpr, ident("typedesc"), copyNimTree(typeIdent)),
newEmptyNode(),
)
)
for paramDef in requestParamDefs:
formalParams.add(paramDef)
let requestPragmas =
case mode
of rbAsync:
quote:
{.async: (raises: []).}
of rbSync:
quote:
{.gcsafe, raises: [].}
var forwardCall = newCall(ident("request"))
forwardCall.add(copyNimTree(typeIdent))
forwardCall.add(ident("DefaultBrokerContext"))
for argName in argNameIdents:
forwardCall.add(argName)
var requestBody = newStmtList()
case mode
of rbAsync:
requestBody.add(
quote do:
return await `forwardCall`
)
of rbSync:
requestBody.add(
quote do:
return `forwardCall`
)
result.add(
newTree(
nnkProcDef,
postfix(ident("request"), "*"),
newEmptyNode(),
newEmptyNode(),
formalParams,
requestPragmas,
newEmptyNode(),
requestBody,
)
)
# Keyed request variant for the argument-based signature.
let requestParamDefsKeyed = cloneParams(argParams)
let argNameIdentsKeyed = collectParamNames(requestParamDefsKeyed)
let providerSymKeyed = genSym(nskVar, "provider")
var formalParamsKeyed = newTree(nnkFormalParams)
formalParamsKeyed.add(copyNimTree(returnType))
formalParamsKeyed.add(
newTree(
nnkIdentDefs,
ident("_"),
newTree(nnkBracketExpr, ident("typedesc"), copyNimTree(typeIdent)),
newEmptyNode(),
)
)
formalParamsKeyed.add(
newTree(nnkIdentDefs, ident("brokerCtx"), ident("BrokerContext"), newEmptyNode())
)
for paramDef in requestParamDefsKeyed:
formalParamsKeyed.add(paramDef)
let requestPragmasKeyed = requestPragmas
var providerCallKeyed = newCall(providerSymKeyed)
for argName in argNameIdentsKeyed:
providerCallKeyed.add(argName)
var requestBodyKeyed = newStmtList()
requestBodyKeyed.add(
quote do:
var `providerSymKeyed`: `argProviderName`
if brokerCtx == DefaultBrokerContext:
`providerSymKeyed` = `accessProcIdent`().`argProvidersFieldName`[0].handler
else:
for entry in `accessProcIdent`().`argProvidersFieldName`:
if entry.brokerCtx == brokerCtx:
`providerSymKeyed` = entry.handler
break
)
requestBodyKeyed.add(
quote do:
if `providerSymKeyed`.isNil():
if brokerCtx == DefaultBrokerContext:
return err(
"RequestBroker(" & `typeNameLit` &
"): no provider registered for input signature"
)
return err(
"RequestBroker(" & `typeNameLit` &
"): no provider registered for broker context " & $brokerCtx
)
)
case mode
of rbAsync:
requestBodyKeyed.add(
quote do:
let catchedRes = catch:
await `providerCallKeyed`
if catchedRes.isErr():
return err(
"RequestBroker(" & `typeNameLit` & "): provider threw exception: " &
catchedRes.error.msg
)
let providerRes = catchedRes.get()
if providerRes.isOk():
let resultValue = providerRes.get()
when compiles(resultValue.isNil()):
if resultValue.isNil():
return err(
"RequestBroker(" & `typeNameLit` & "): provider returned nil result"
)
return providerRes
)
of rbSync:
requestBodyKeyed.add(
quote do:
var providerRes: Result[`typeIdent`, string]
try:
providerRes = `providerCallKeyed`
except CatchableError as e:
return err(
"RequestBroker(" & `typeNameLit` & "): provider threw exception: " & e.msg
)
if providerRes.isOk():
let resultValue = providerRes.get()
when compiles(resultValue.isNil()):
if resultValue.isNil():
return err(
"RequestBroker(" & `typeNameLit` & "): provider returned nil result"
)
return providerRes
)
result.add(
newTree(
nnkProcDef,
postfix(ident("request"), "*"),
newEmptyNode(),
newEmptyNode(),
formalParamsKeyed,
requestPragmasKeyed,
newEmptyNode(),
requestBodyKeyed,
)
)
block:
var formalParamsClearKeyed = newTree(nnkFormalParams)
formalParamsClearKeyed.add(newEmptyNode())
formalParamsClearKeyed.add(
newTree(
nnkIdentDefs,
ident("_"),
newTree(nnkBracketExpr, ident("typedesc"), copyNimTree(typeIdent)),
newEmptyNode(),
)
)
formalParamsClearKeyed.add(
newTree(nnkIdentDefs, brokerCtxParamIdent, ident("BrokerContext"), newEmptyNode())
)
result.add(
newTree(
nnkProcDef,
postfix(ident("clearProvider"), "*"),
newEmptyNode(),
newEmptyNode(),
formalParamsClearKeyed,
newEmptyNode(),
newEmptyNode(),
clearBodyKeyed,
)
)
result.add(
quote do:
proc clearProvider*(_: typedesc[`typeIdent`]) =
clearProvider(`typeIdent`, DefaultBrokerContext)
)
when defined(requestBrokerDebug):
echo result.repr
return result
macro RequestBroker*(body: untyped): untyped =
## Default (async) mode.
generateRequestBroker(body, rbAsync)
macro RequestBroker*(mode: untyped, body: untyped): untyped =
## Explicit mode selector.
## Example:
## RequestBroker(sync):
## type Foo = object
## proc signature*(): Result[Foo, string]
generateRequestBroker(body, parseMode(mode))

3
waku/events/delivery_events.nim
View File

@ -1,4 +1,5 @@
import waku/waku_core/[message/message, message/digest], waku/common/broker/event_broker
import brokers/event_broker
import waku/waku_core/[message/message, message/digest]
EventBroker:
type OnFilterSubscribeEvent* = object

4
waku/events/events.nim
View File

@ -1,3 +1,3 @@
import ./[message_events, delivery_events, health_events, peer_events]
import ./[message_events, delivery_events, health_events, peer_events, lifecycle_events]
export message_events, delivery_events, health_events, peer_events
export message_events, delivery_events, health_events, peer_events, lifecycle_events

2
waku/events/health_events.nim
View File

@ -1,4 +1,4 @@
import waku/common/broker/event_broker
import brokers/event_broker
import waku/api/types
import waku/node/health_monitor/[protocol_health, topic_health]

4
waku/events/message_events.nim
View File

@ -1,5 +1,5 @@
import waku/[api/types, waku_core/message, waku_core/topics, common/broker/event_broker]
import brokers/event_broker
import waku/[api/types, waku_core/message, waku_core/topics]
export types
EventBroker:

2
waku/events/peer_events.nim
View File

@ -1,4 +1,4 @@
import waku/common/broker/event_broker
import brokers/event_broker
import libp2p/switch
type WakuPeerEventKind* {.pure.} = enum

7
waku/factory/builder.nim
View File

@ -8,15 +8,16 @@ import
libp2p/builders,
libp2p/nameresolving/nameresolver,
libp2p/transports/wstransport,
libp2p/protocols/connectivity/relay/relay
libp2p/protocols/connectivity/relay/relay,
brokers/broker_context
import
../waku_enr,
../discovery/waku_discv5,
../waku_node,
../node/peer_manager,
../common/rate_limit/setting,
../common/utils/parse_size_units,
../common/broker/broker_context
../common/utils/parse_size_units
type
WakuNodeBuilder* = object # General

7
waku/factory/conf_builder/waku_conf_builder.nim
View File

@ -14,6 +14,7 @@ import
common/logging,
common/utils/parse_size_units,
waku_enr/capabilities,
persistency/persistency,
],
tools/confutils/entry_nodes
@ -136,6 +137,8 @@ type WakuConfBuilder* = object
circuitRelayClient: Option[bool]
p2pReliability: Option[bool]
localStoragePath: Option[string]
proc init*(T: type WakuConfBuilder): WakuConfBuilder =
WakuConfBuilder(
dnsDiscoveryConf: DnsDiscoveryConfBuilder.init(),
@ -272,6 +275,9 @@ proc withRelayShardedPeerManagement*(
proc withP2pReliability*(b: var WakuConfBuilder, p2pReliability: bool) =
b.p2pReliability = some(p2pReliability)
proc withLocalStoragePath*(b: var WakuConfBuilder, localStoragePath: string) =
b.localStoragePath = some(localStoragePath)
proc withExtMultiAddrs*(builder: var WakuConfBuilder, extMultiAddrs: seq[string]) =
builder.extMultiAddrs = concat(builder.extMultiAddrs, extMultiAddrs)
@ -719,6 +725,7 @@ proc build*(
relayShardedPeerManagement: relayShardedPeerManagement,
p2pReliability: builder.p2pReliability.get(false),
wakuFlags: wakuFlags,
localStoragePath: builder.localStoragePath.get(DefaultStoragePath),
)
?wakuConf.validate()

11
waku/factory/waku.nim
View File

@ -18,6 +18,7 @@ import
presto,
metrics,
metrics/chronos_httpserver,
brokers/broker_context,
waku/[
waku_core,
waku_node,
@ -30,7 +31,6 @@ import
waku_enr/multiaddr,
api/types,
common/logging,
common/broker/broker_context,
node/peer_manager,
node/health_monitor,
node/waku_metrics,
@ -46,6 +46,7 @@ import
factory/node_factory,
factory/internal_config,
factory/app_callbacks,
persistency/persistency,
],
./waku_conf,
./waku_state_info
@ -393,6 +394,12 @@ proc startWaku*(waku: ptr Waku): Future[Result[void, string]] {.async: (raises:
else:
waku[].dynamicBootstrapNodes = dynamicBootstrapNodesRes.get()
## Initialize persistency singleton instance - we don't need the instance itself here,
## but this ensures it's initialized before any store job starts.
discard Persistency.instance(conf.localStoragePath).valueOr:
error "Failed to initialize persistency instance", error = $error
return err("Failed to initialize persistency instance: " & $error)
(await startNode(waku.node, waku.conf, waku.dynamicBootstrapNodes)).isOkOr:
return err("error while calling startNode: " & $error)
@ -523,6 +530,8 @@ proc stop*(waku: Waku): Future[Result[void, string]] {.async: (raises: []).} =
try:
waku.healthMonitor.setOverallHealth(HealthStatus.SHUTTING_DOWN)
Persistency.reset()
if not waku.metricsServer.isNil():
await waku.metricsServer.stop()

2
waku/factory/waku_conf.nim
View File

@ -152,6 +152,8 @@ type WakuConf* {.requiresInit.} = ref object
p2pReliability*: bool
localStoragePath*: string
proc logConf*(conf: WakuConf) =
info "Configuration: Enabled protocols",
relay = conf.relay,

4
waku/node/delivery_service/recv_service/recv_service.nim
View File

@ -5,6 +5,7 @@
import std/[tables, sequtils, options, sets]
import chronos, chronicles, libp2p/utility
import ../[subscription_manager]
import brokers/broker_context
import
waku/[
waku_core,
@ -14,7 +15,6 @@ import
waku_core/topics,
events/message_events,
waku_node,
common/broker/broker_context,
]
const StoreCheckPeriod = chronos.minutes(5) ## How often to perform store queries
@ -179,7 +179,7 @@ proc startRecvService*(self: RecvService) =
quit(QuitFailure)
proc stopRecvService*(self: RecvService) {.async.} =
MessageSeenEvent.dropListener(self.brokerCtx, self.seenMsgListener)
await MessageSeenEvent.dropListener(self.brokerCtx, self.seenMsgListener)
if not self.msgCheckerHandler.isNil():
await self.msgCheckerHandler.cancelAndWait()
self.msgCheckerHandler = nil

2
waku/node/delivery_service/send_service/delivery_task.nim
View File

@ -1,6 +1,6 @@
import std/[options, times], chronos
import brokers/broker_context
import waku/waku_core, waku/api/types, waku/requests/node_requests
import waku/common/broker/broker_context
type DeliveryState* {.pure.} = enum
Entry

8
waku/node/delivery_service/send_service/lightpush_processor.nim
View File

@ -1,11 +1,7 @@
import chronicles, chronos, results
import std/options
import
waku/node/peer_manager,
waku/waku_core,
waku/waku_lightpush/[common, client, rpc],
waku/common/broker/broker_context
import brokers/broker_context
import waku/node/peer_manager, waku/waku_core, waku/waku_lightpush/[common, client, rpc]
import ./[delivery_task, send_processor]

2
waku/node/delivery_service/send_service/relay_processor.nim
View File

@ -1,8 +1,8 @@
import std/options
import chronos, chronicles
import brokers/broker_context
import waku/[waku_core], waku/waku_lightpush/[common, rpc]
import waku/requests/health_requests
import waku/common/broker/broker_context
import waku/api/types
import ./[delivery_task, send_processor]

2
waku/node/delivery_service/send_service/send_processor.nim
View File

@ -1,6 +1,6 @@
import chronos
import brokers/broker_context
import ./delivery_task
import waku/common/broker/broker_context
{.push raises: [].}

2
waku/node/delivery_service/send_service/send_service.nim
View File

@ -3,6 +3,7 @@
import std/[sequtils, tables, options]
import chronos, chronicles, libp2p/utility
import brokers/broker_context
import
./[send_processor, relay_processor, lightpush_processor, delivery_task],
../[subscription_manager],
@ -17,7 +18,6 @@ import
waku_lightpush/client,
waku_lightpush/callbacks,
events/message_events,
common/broker/broker_context,
]
logScope:

5
waku/node/delivery_service/subscription_manager.nim
View File

@ -1,5 +1,7 @@
import std/[sequtils, sets, tables, options, strutils], chronos, chronicles, results
import libp2p/[peerid, peerinfo]
import brokers/broker_context
import
waku/[
waku_core,
@ -10,7 +12,6 @@ import
waku_filter_v2/common as filter_common,
waku_filter_v2/client as filter_client,
waku_filter_v2/protocol as filter_protocol,
common/broker/broker_context,
events/health_events,
events/peer_events,
requests/health_requests,
@ -530,7 +531,7 @@ proc stopEdgeFilterLoops(self: SubscriptionManager) {.async: (raises: []).} =
if not fut.finished:
await fut.cancelAndWait()
WakuPeerEvent.dropListener(self.node.brokerCtx, self.peerEventListener)
await WakuPeerEvent.dropListener(self.node.brokerCtx, self.peerEventListener)
# ---------------------------------------------------------------------------
# SubscriptionManager Lifecycle (calls Edge behavior above)

6
waku/node/health_monitor/node_health_monitor.nim
View File

@ -725,8 +725,10 @@ proc stopHealthMonitor*(hm: NodeHealthMonitor) {.async.} =
if not isNil(hm.eventLoopMonitorFut):
await hm.eventLoopMonitorFut.cancelAndWait()
WakuPeerEvent.dropListener(hm.node.brokerCtx, hm.peerEventListener)
EventShardTopicHealthChange.dropListener(hm.node.brokerCtx, hm.shardHealthListener)
await WakuPeerEvent.dropListener(hm.node.brokerCtx, hm.peerEventListener)
await EventShardTopicHealthChange.dropListener(
hm.node.brokerCtx, hm.shardHealthListener
)
if not isNil(hm.node.wakuRelay) and not isNil(hm.relayObserver):
hm.node.wakuRelay.removeObserver(hm.relayObserver)

4
waku/node/kernel_api/relay.nim
View File

@ -16,7 +16,8 @@ import
libp2p/builders,
libp2p/transports/tcptransport,
libp2p/transports/wstransport,
libp2p/utility
libp2p/utility,
brokers/broker_context
import
waku/[
@ -29,7 +30,6 @@ import
waku_rln_relay,
node/waku_node,
node/peer_manager,
common/broker/broker_context,
events/message_events,
]

4
waku/node/peer_manager/peer_manager.nim
View File

@ -8,6 +8,9 @@ import
chronicles,
metrics,
libp2p/[multistream, muxers/muxer, nameresolving/nameresolver, peerstore],
brokers/broker_context
import
waku/[
waku_core,
waku_relay,
@ -21,7 +24,6 @@ import
common/enr,
common/callbacks,
common/utils/parse_size_units,
common/broker/broker_context,
node/health_monitor/online_monitor,
],
./peer_store/peer_storage,

6
waku/node/waku_node.nim
View File

@ -24,7 +24,9 @@ import
libp2p/utility,
libp2p/utils/offsettedseq,
libp2p/protocols/mix,
libp2p/protocols/mix/mix_protocol
libp2p/protocols/mix/mix_protocol,
brokers/broker_context,
brokers/request_broker
import
waku/[
@ -53,8 +55,6 @@ import
common/rate_limit/setting,
common/callbacks,
common/nimchronos,
common/broker/broker_context,
common/broker/request_broker,
waku_mix,
requests/node_requests,
requests/health_requests,

161
waku/persistency/backend_comm.nim Normal file
View File

@ -0,0 +1,161 @@
## Cross-thread broker declarations for the persistency library.
##
## One EventBroker (writes, fire-and-forget) and five RequestBrokers (reads
## + acked delete). All in multi-thread (mt) mode: the listener / provider runs on the
## job's storage thread; callers on any thread reach it via the shared
## BrokerContext owned by the Job.
##
## ## Error type, important
##
## nim-brokers' RequestBroker macro hard-codes the response shape as
## `Future[Result[ResponseType, string]]` — the error channel is `string`,
## not our `PersistencyError`. We honour the broker contract here and lift
## back to `PersistencyError` at the public facade (persistency.nim). The
## convention for the broker-level string is `"<kind>: <msg>"` so the
## facade can reconstruct the `PersistencyErrorKind`.
##
## ## Response shapes
##
## The five Kv* types are *response* objects (the value the provider
## returns). Per-request inputs sit on the `signature` proc parameters.
{.push raises: [].}
import std/[options, strutils]
import chronos, results
import brokers/[event_broker, request_broker, broker_context]
import brokers/internal/mt_codec
import ./types
export broker_context
# ── mt codec overloads for non-POD library types ────────────────────────
#
# brokers 2.0.0's mtMarshalValue / mtUnmarshalValue handle scalars, enums,
# strings, seqs, arrays, and plain object/tuple recursion -- but they do
# not see through `distinct seq[byte]`, nor do they know how to dispatch
# a variant (case) object. We provide explicit overloads for the types
# that appear in our broker payloads.
proc mtMarshalValue*(
buf: ptr UncheckedArray[byte], cap: int, value: Key, pos: var int
): bool {.gcsafe.} =
## Encode a Key as the raw seq[byte] it wraps.
mtMarshalValue(buf, cap, bytes(value), pos)
proc mtUnmarshalValue*(
buf: ptr UncheckedArray[byte], len: int, value: var Key, pos: var int
): bool {.gcsafe.} =
var s: seq[byte]
if not mtUnmarshalValue(buf, len, s, pos):
return false
value = Key(s)
return true
proc mtMarshalValue*(
buf: ptr UncheckedArray[byte], cap: int, value: TxOp, pos: var int
): bool {.gcsafe.} =
## TxOp is a case object: write the discriminator, then only the
## fields that belong to the active branch.
if not mtMarshalValue(buf, cap, value.category, pos):
return false
if not mtMarshalValue(buf, cap, value.key, pos):
return false
let kind = uint8(ord(value.kind))
if not mtMarshalValue(buf, cap, kind, pos):
return false
case value.kind
of txPut:
if not mtMarshalValue(buf, cap, value.payload, pos):
return false
of txDelete:
discard
return true
proc mtUnmarshalValue*(
buf: ptr UncheckedArray[byte], len: int, value: var TxOp, pos: var int
): bool {.gcsafe.} =
var
category: string
key: Key
kindByte: uint8
if not mtUnmarshalValue(buf, len, category, pos):
return false
if not mtUnmarshalValue(buf, len, key, pos):
return false
if not mtUnmarshalValue(buf, len, kindByte, pos):
return false
case TxOpKind(kindByte)
of txPut:
var payload: seq[byte]
if not mtUnmarshalValue(buf, len, payload, pos):
return false
value = TxOp(category: category, key: key, kind: txPut, payload: payload)
of txDelete:
value = TxOp(category: category, key: key, kind: txDelete)
return true
EventBroker(mt):
type PersistEvent* = object
ops*: seq[TxOp]
RequestBroker(mt):
type KvGet* = object
value*: Option[seq[byte]]
proc signature*(category: string, key: Key): Future[Result[KvGet, string]] {.async.}
RequestBroker(mt):
type KvExists* = object
value*: bool
proc signature*(
category: string, key: Key
): Future[Result[KvExists, string]] {.async.}
RequestBroker(mt):
type KvScan* = object
rows*: seq[KvRow]
proc signature*(
category: string, range: KeyRange, reverse: bool
): Future[Result[KvScan, string]] {.async.}
RequestBroker(mt):
type KvCount* = object
n*: int
proc signature*(
category: string, range: KeyRange
): Future[Result[KvCount, string]] {.async.}
RequestBroker(mt):
type KvDelete* = object
existed*: bool
proc signature*(
category: string, key: Key
): Future[Result[KvDelete, string]] {.async.}
# ── string<->PersistencyError boundary helpers ──────────────────────────
const ErrSep = ": "
proc encodeErr*(e: PersistencyError): string =
## Encode a PersistencyError into the broker's string channel. The facade
## decodes via `decodeErr`.
$e.kind & ErrSep & e.msg
proc decodeErr*(s: string): PersistencyError =
## Inverse of encodeErr. Falls back to peBackend if the prefix is missing.
let idx = s.find(ErrSep)
if idx < 0:
return persistencyErr(peBackend, s)
let head = s[0 ..< idx]
let tail = s[idx + ErrSep.len .. ^1]
for k in PersistencyErrorKind:
if $k == head:
return persistencyErr(k, tail)
persistencyErr(peBackend, s)
{.pop.}

247
waku/persistency/backend_sqlite.nim Normal file
View File

@ -0,0 +1,247 @@
## Synchronous SQLite backend for the persistency library.
##
## Plain procs against a SqliteDatabase connection. Phase 3 wraps these in
## per-job storage threads driven by brokers; phase 2 verifies the SQL
## itself against an in-memory database.
import std/options
import results, sqlite3_abi
import ../common/databases/[common, db_sqlite]
import ./[types, schema]
type
KvBackend* = ref object
db*: SqliteDatabase
putStmt: SqliteStmt[(seq[byte], seq[byte], seq[byte]), void]
deleteStmt: SqliteStmt[(seq[byte], seq[byte]), void]
RowHandler = proc(s: ptr sqlite3_stmt) {.gcsafe, raises: [].}
proc toErr(msg: string): PersistencyError {.inline.} =
persistencyErr(peBackend, msg)
proc catBytes(category: string): seq[byte] =
var buf = newSeq[byte](category.len)
for i, c in category:
buf[i] = byte(c)
return buf
proc keyBytes(key: Key): seq[byte] {.inline.} =
bytes(key)
proc readBlob(s: ptr sqlite3_stmt, col: cint): seq[byte] =
let n = sqlite3_column_bytes(s, col)
var buf = newSeq[byte](n)
if n > 0:
let src = cast[ptr UncheckedArray[byte]](sqlite3_column_blob(s, col))
for i in 0 ..< n:
buf[i] = src[i]
return buf
proc bindBlob(s: ptr sqlite3_stmt, n: cint, val: seq[byte]): cint =
if val.len > 0:
sqlite3_bind_blob(s, n, unsafeAddr val[0], val.len.cint, SQLITE_TRANSIENT)
else:
sqlite3_bind_blob(s, n, nil, 0.cint, SQLITE_TRANSIENT)
proc runRead(
db: SqliteDatabase, sql: string, params: openArray[seq[byte]], onRow: RowHandler
): Result[void, PersistencyError] =
var s: ptr sqlite3_stmt
let rc = sqlite3_prepare_v2(db.env, sql.cstring, sql.len.cint, addr s, nil)
if rc != SQLITE_OK:
return err(toErr("prepare: " & $sqlite3_errstr(rc)))
defer:
discard sqlite3_finalize(s)
for i, p in params:
let bc = bindBlob(s, cint(i + 1), p)
if bc != SQLITE_OK:
return err(toErr("bind: " & $sqlite3_errstr(bc)))
while true:
let v = sqlite3_step(s)
case v
of SQLITE_ROW:
onRow(s)
of SQLITE_DONE:
break
else:
return err(toErr("step: " & $sqlite3_errstr(v)))
return ok()
proc prepareStatements(b: KvBackend): DatabaseResult[void] =
b.putStmt = ?b.db.prepareStmt(
"INSERT OR REPLACE INTO kv(category, key, payload) VALUES (?, ?, ?);",
(seq[byte], seq[byte], seq[byte]),
void,
)
b.deleteStmt = ?b.db.prepareStmt(
"DELETE FROM kv WHERE category = ? AND key = ?;", (seq[byte], seq[byte]), void
)
return ok()
proc openBackend*(path: string): Result[KvBackend, PersistencyError] =
let dbRes = SqliteDatabase.new(path)
if dbRes.isErr:
return err(toErr("open " & path & " failed: " & dbRes.error))
let db = dbRes.get()
applyPragmas(db).isOkOr:
return err(toErr(error))
ensureSchema(db).isOkOr:
return err(toErr(error))
let b = KvBackend(db: db)
prepareStatements(b).isOkOr:
return err(toErr(error))
return ok(b)
proc openBackendInMemory*(): Result[KvBackend, PersistencyError] =
## Convenience for tests.
let dbRes = SqliteDatabase.new(":memory:")
if dbRes.isErr:
return err(toErr("open :memory: failed: " & dbRes.error))
let db = dbRes.get()
applyPragmas(db).isOkOr:
return err(toErr(error))
ensureSchema(db).isOkOr:
return err(toErr(error))
let b = KvBackend(db: db)
prepareStatements(b).isOkOr:
return err(toErr(error))
return ok(b)
proc close*(b: KvBackend) =
if b.db != nil:
dispose(b.putStmt)
dispose(b.deleteStmt)
b.db.close()
b.db = nil
proc applyOne(b: KvBackend, op: TxOp): Result[void, PersistencyError] =
case op.kind
of txPut:
let r = b.putStmt.exec((catBytes(op.category), keyBytes(op.key), op.payload))
if r.isErr:
return err(toErr("put failed: " & r.error))
of txDelete:
let r = b.deleteStmt.exec((catBytes(op.category), keyBytes(op.key)))
if r.isErr:
return err(toErr("delete failed: " & r.error))
return ok()
proc execSql(b: KvBackend, sql: string): Result[void, PersistencyError] =
let r = b.db.query(sql, NoopRowHandler)
if r.isErr:
return err(toErr(sql & ": " & r.error))
return ok()
proc applyOps*(b: KvBackend, ops: openArray[TxOp]): Result[void, PersistencyError] =
## Single op = auto-commit. Multiple ops = BEGIN IMMEDIATE / COMMIT, with
## ROLLBACK on first failure. This is the single source of truth for write
## SQL — Phase 3's PersistEvent listener calls straight into here.
if ops.len == 0:
return ok()
if ops.len == 1:
return b.applyOne(ops[0])
?b.execSql("BEGIN IMMEDIATE;")
for op in ops:
let r = b.applyOne(op)
if r.isErr:
discard b.execSql("ROLLBACK;")
return r
?b.execSql("COMMIT;")
return ok()
proc getOne*(
b: KvBackend, category: string, key: Key
): Result[Option[seq[byte]], PersistencyError] =
var found: Option[seq[byte]] = none(seq[byte])
proc onRow(rs: ptr sqlite3_stmt) {.gcsafe, raises: [].} =
found = some(readBlob(rs, 0.cint))
?b.db.runRead(
"SELECT payload FROM kv WHERE category = ? AND key = ? LIMIT 1;",
[catBytes(category), keyBytes(key)],
onRow,
)
return ok(found)
proc existsOne*(
b: KvBackend, category: string, key: Key
): Result[bool, PersistencyError] =
var present = false
proc onRow(rs: ptr sqlite3_stmt) {.gcsafe, raises: [].} =
present = true
?b.db.runRead(
"SELECT 1 FROM kv WHERE category = ? AND key = ? LIMIT 1;",
[catBytes(category), keyBytes(key)],
onRow,
)
return ok(present)
proc deleteOne*(
b: KvBackend, category: string, key: Key
): Result[bool, PersistencyError] =
## Returns true if a row was actually removed.
let existed = ?b.existsOne(category, key)
if not existed:
return ok(false)
let r = b.deleteStmt.exec((catBytes(category), keyBytes(key)))
if r.isErr:
return err(toErr("delete: " & r.error))
return ok(true)
proc scanRange*(
b: KvBackend, category: string, range: KeyRange, reverse = false
): Result[seq[KvRow], PersistencyError] =
let openEnded = bytes(range.stop).len == 0
let direction = if reverse: "DESC" else: "ASC"
let sql =
if openEnded:
"SELECT key, payload FROM kv WHERE category = ? AND key >= ? ORDER BY key " &
direction & ";"
else:
"SELECT key, payload FROM kv WHERE category = ? AND key >= ? AND key < ? ORDER BY key " &
direction & ";"
var rows: seq[KvRow] = @[]
proc onRow(rs: ptr sqlite3_stmt) {.gcsafe, raises: [].} =
let k = readBlob(rs, 0.cint)
let p = readBlob(rs, 1.cint)
rows.add((rawKey(k), p))
if openEnded:
?b.db.runRead(sql, [catBytes(category), keyBytes(range.start)], onRow)
else:
?b.db.runRead(
sql, [catBytes(category), keyBytes(range.start), keyBytes(range.stop)], onRow
)
return ok(rows)
proc countRange*(
b: KvBackend, category: string, range: KeyRange
): Result[int, PersistencyError] =
let openEnded = bytes(range.stop).len == 0
let sql =
if openEnded:
"SELECT COUNT(*) FROM kv WHERE category = ? AND key >= ?;"
else:
"SELECT COUNT(*) FROM kv WHERE category = ? AND key >= ? AND key < ?;"
var n: int64 = 0
proc onRow(rs: ptr sqlite3_stmt) {.gcsafe, raises: [].} =
n = sqlite3_column_int64(rs, 0.cint)
if openEnded:
?b.db.runRead(sql, [catBytes(category), keyBytes(range.start)], onRow)
else:
?b.db.runRead(
sql, [catBytes(category), keyBytes(range.start), keyBytes(range.stop)], onRow
)
return ok(int(n))

271
waku/persistency/backend_thread.nim Normal file
View File

@ -0,0 +1,271 @@
## Internal per-job storage thread.
##
## Exposes two operations to ``persistency.nim``:
## * ``startStorageThread(ctx, dbPath)`` — spawn one worker, block until
## it signals ready (or error). Returns a ``JobRuntime``.
## * ``stopStorageThread(rt)`` — signal shutdown, join, free.
##
## The worker:
## 1. installs the supplied BrokerContext on its threadvar
## 2. opens the SQLite backend (creating the file + schema if absent)
## 3. registers the PersistEvent listener and the 5 RequestBroker
## providers under that context
## 4. runs the chronos event loop until shutdown is signalled
## 5. clears providers + listeners, closes the backend
##
## The arg struct lives in shared memory (``allocShared0``). The dbPath is
## carried as a shared cstring buffer rather than a Nim string to avoid
## refc ref-count traffic across threads. The arg is freed by
## ``stopStorageThread`` after ``joinThread`` returns.
import std/[options, os]
import std/atomics # std/concurrency/atomics is the same module in Nim 2.2
import chronos, chronicles, results
import brokers/[event_broker, request_broker, broker_context]
import ./[types, backend_comm, backend_sqlite]
export broker_context, backend_comm
logScope:
topics = "persistency thread"
type
ReadyState {.pure.} = enum
Pending = 0
Ready = 1
Error = 2
StorageThreadArg = object
ctx: BrokerContext
dbPath: cstring ## allocShared0'd; freed in closeJob
dbPathLen: int ## bytes including the trailing NUL
shutdownFlag: Atomic[int]
readyFlag: Atomic[int] ## values from ReadyState
errBuf: array[256, char] ## last error message, NUL-terminated
StorageThread = Thread[ptr StorageThreadArg]
# ── arg helpers ─────────────────────────────────────────────────────────
proc allocArg(ctx: BrokerContext, dbPath: string): ptr StorageThreadArg =
let arg = cast[ptr StorageThreadArg](allocShared0(sizeof(StorageThreadArg)))
arg.ctx = ctx
arg.dbPathLen = dbPath.len + 1
arg.dbPath = cast[cstring](allocShared0(arg.dbPathLen))
if dbPath.len > 0:
copyMem(arg.dbPath, unsafeAddr dbPath[0], dbPath.len)
return arg
proc freeArg(a: ptr StorageThreadArg) =
if a.isNil():
return
if a.dbPath != nil:
deallocShared(a.dbPath)
deallocShared(a)
proc recordErr(a: ptr StorageThreadArg, msg: string) =
let n = min(msg.len, a.errBuf.len - 1)
for i in 0 ..< n:
a.errBuf[i] = msg[i]
a.errBuf[n] = '\0'
a.readyFlag.store(int(ReadyState.Error), moRelease)
proc errMsg(a: ptr StorageThreadArg): string =
$cast[cstring](a.errBuf[0].addr)
# ── provider closures ───────────────────────────────────────────────────
proc encode(e: PersistencyError): string =
encodeErr(e)
template unwrapErr(r: untyped): string =
## Disambiguates Result's `error` accessor from chronicles' `error` macro
## by binding through an explicitly-typed local before stringifying.
block:
let pe: PersistencyError = r.error()
encode(pe)
proc registerProviders(backend: KvBackend, ctx: BrokerContext): Result[void, string] =
## Wires the 5 RequestBroker providers + the PersistEvent listener.
## All closures capture `backend` by reference (it lives for the entire
## thread lifetime).
proc onGet(category: string, key: Key): Future[Result[KvGet, string]] {.async.} =
let r = backend.getOne(category, key)
if r.isErr:
return err(unwrapErr(r))
return ok(KvGet(value: r.get()))
proc onExists(
category: string, key: Key
): Future[Result[KvExists, string]] {.async.} =
let r = backend.existsOne(category, key)
if r.isErr:
return err(unwrapErr(r))
return ok(KvExists(value: r.get()))
proc onScan(
category: string, range: KeyRange, reverse: bool
): Future[Result[KvScan, string]] {.async.} =
let r = backend.scanRange(category, range, reverse)
if r.isErr:
return err(unwrapErr(r))
return ok(KvScan(rows: r.get()))
proc onCount(
category: string, range: KeyRange
): Future[Result[KvCount, string]] {.async.} =
let r = backend.countRange(category, range)
if r.isErr:
return err(unwrapErr(r))
return ok(KvCount(n: r.get()))
proc onDelete(
category: string, key: Key
): Future[Result[KvDelete, string]] {.async.} =
let r = backend.deleteOne(category, key)
if r.isErr:
return err(unwrapErr(r))
return ok(KvDelete(existed: r.get()))
# PersistEvent listener — fire-and-forget; we log on backend failure
# because the caller has no return channel.
proc onPersist(ev: PersistEvent): Future[void] {.async: (raises: []).} =
let r = backend.applyOps(ev.ops)
if r.isErr:
let pe: PersistencyError = r.error()
error "PersistEvent applyOps failed",
error = pe.msg, kind = $pe.kind, opCount = ev.ops.len
KvGet.setProvider(ctx, onGet).isOkOr:
return err("KvGet.setProvider: " & error)
let existsRes = KvExists.setProvider(ctx, onExists)
if existsRes.isErr:
return err("KvExists.setProvider: " & existsRes.error())
let scanRes = KvScan.setProvider(ctx, onScan)
if scanRes.isErr:
return err("KvScan.setProvider: " & scanRes.error())
let countRes = KvCount.setProvider(ctx, onCount)
if countRes.isErr:
return err("KvCount.setProvider: " & countRes.error())
let delRes = KvDelete.setProvider(ctx, onDelete)
if delRes.isErr:
return err("KvDelete.setProvider: " & delRes.error())
let listenRes = PersistEvent.listen(ctx, onPersist)
if listenRes.isErr:
return err("PersistEvent.listen: " & listenRes.error())
return ok()
proc clearProviders(ctx: BrokerContext) =
KvGet.clearProvider(ctx)
KvExists.clearProvider(ctx)
KvScan.clearProvider(ctx)
KvCount.clearProvider(ctx)
KvDelete.clearProvider(ctx)
PersistEvent.dropAllListeners(ctx)
# ── thread proc ─────────────────────────────────────────────────────────
proc storageThreadMain(arg: ptr StorageThreadArg) {.thread.} =
## Worker thread entrypoint. Errors during setup are surfaced via
## arg.errBuf + readyFlag=ReadyState.Error; the spawning thread checks both.
setThreadBrokerContext(arg.ctx)
let path = $arg.dbPath
let backendRes =
try:
openBackend(path)
except CatchableError as e:
arg.recordErr("openBackend raised: " & e.msg)
return
if backendRes.isErr:
arg.recordErr("openBackend: " & backendRes.error.msg)
return
let backend = backendRes.get()
let regRes =
try:
registerProviders(backend, arg.ctx)
except CatchableError as e:
backend.close()
arg.recordErr("registerProviders raised: " & e.msg)
return
if regRes.isErr:
backend.close()
arg.recordErr(regRes.error)
return
arg.readyFlag.store(int(ReadyState.Ready), moRelease)
proc awaitShutdown() {.async.} =
while arg.shutdownFlag.load(moAcquire) != 1:
try:
await sleepAsync(milliseconds(10))
except CatchableError:
discard
try:
waitFor awaitShutdown()
except CatchableError as e:
error "storage thread loop crashed", err = e.msg
clearProviders(arg.ctx)
backend.close()
# ── lifecycle ───────────────────────────────────────────────────────────
type JobRuntime* = ref object
## Opaque per-job runtime owned by `persistency.nim`. Holds the typed
## Thread handle + shared arg pointer so closeJob can shut the worker
## down. Created by `startStorageThread` and torn down by
## `stopStorageThread`.
arg*: ptr StorageThreadArg
thread*: StorageThread
proc startStorageThread*(
ctx: BrokerContext, dbPath: string
): Result[JobRuntime, PersistencyError] =
## Spawn a storage worker for one job. Blocks until the worker either
## signals ready (returns the runtime) or signals error (joins, frees,
## returns peBackend with the worker's error message).
let arg = allocArg(ctx, dbPath)
arg.shutdownFlag.store(0, moRelease)
arg.readyFlag.store(int(ReadyState.Pending), moRelease)
var rt = JobRuntime(arg: arg)
try:
createThread(rt.thread, storageThreadMain, arg)
except ResourceExhaustedError as e:
freeArg(arg)
return err(persistencyErr(peBackend, "createThread: " & e.msg))
# Spin-wait for ready or error. The thread does its setup synchronously
# before signaling, so this is bounded by SQLite open time.
while true:
let s = arg.readyFlag.load(moAcquire)
if s == int(ReadyState.Ready):
return ok(rt)
if s == int(ReadyState.Error):
let msg = errMsg(arg)
joinThread(rt.thread)
freeArg(arg)
return err(persistencyErr(peBackend, msg))
sleep(1)
proc stopStorageThread*(rt: JobRuntime) =
## Signal shutdown, join the worker, free the shared arg. Idempotent in
## the sense that it tolerates a nil arg (already stopped).
if rt == nil or rt.arg == nil:
return
rt.arg.shutdownFlag.store(1, moRelease)
joinThread(rt.thread)
freeArg(rt.arg)
rt.arg = nil

180
waku/persistency/keys.nim Normal file
View File

@ -0,0 +1,180 @@
## Composite-key encoding.
##
## Keys are byte-wise lexicographically comparable so SQLite's BLOB
## ordering reproduces tuple ordering of the original components. Each
## component contributes a self-delimiting, sort-stable byte sequence
## through an `encodePart` overload; the generic fallback recurses through
## `tuple | object` fields, so any user type whose fields are themselves
## encodable can be used as a key part without ceremony.
##
## ## Encoding by type
##
## | Nim type | Bytes emitted |
## |-------------------------|------------------------------------------------------------------|
## | `string`, `openArray[byte]` | 2-byte BE length prefix + payload bytes (max 65535 bytes) |
## | `int64`, `int`, .. | XOR with 0x8000_0000_0000_0000 then 8-byte BE (sign-flip) |
## | `uint64`, `uint32`, .. | 8-byte BE |
## | `bool` | 1 byte (0/1) |
## | `byte`, `char` | 1 byte |
## | `enum E` | sign-flipped 8-byte BE of `ord(v).int64` |
## | `Key` | raw bytes (lets you embed a pre-built key inside another) |
## | `tuple | object` | each field encoded in declaration order, concatenated |
##
## ## Sort-order caveats
##
## - Length-prefixed strings sort by **length first, then byte order**. For
## uniform-length components (channel ids, hashes) this is identical to
## natural lex order; for variable-length text it is not.
## - `int64.low < -1 < 0 < 1 < int64.high` after byte comparison thanks to
## the sign flip.
## - Tuple/object ordering is component-major: field 0 dominates field 1
## dominates field 2, like a multi-column ORDER BY.
##
## ## Building keys
##
## `key(...)` is a variadic macro that calls `encodePart` per argument. It
## accepts mixed types in one call:
##
## ```nim
## let k = key("channel-42", 1'i64)
## let k2 = key("channel-42", (epoch: 1'i64, seqNum: 7'u64))
## let k3 = key(myEnumValue, myObject)
## ```
##
## For a single value, `toKey(v)` is the simpler form (same semantics).
{.push raises: [].}
import std/macros
import ./types
const
StringLenMax* = 0xFFFF
SignFlip = 0x8000_0000_0000_0000'u64
# ── Low-level byte helpers ──────────────────────────────────────────────
proc appendBE16(buf: var seq[byte], v: uint16) =
buf.add(byte((v shr 8) and 0xFF'u16))
buf.add(byte(v and 0xFF'u16))
proc appendBE64(buf: var seq[byte], v: uint64) =
for shift in countdown(56, 0, 8):
buf.add(byte((v shr shift) and 0xFF'u64))
# ── encodePart: primitives ──────────────────────────────────────────────
proc encodePart*(dest: var seq[byte], s: string) =
doAssert s.len <= StringLenMax, "string component exceeds 65535 bytes"
appendBE16(dest, uint16(s.len))
for c in s:
dest.add(byte(c))
proc encodePart*(dest: var seq[byte], raw: openArray[byte]) =
doAssert raw.len <= StringLenMax, "byte component exceeds 65535 bytes"
appendBE16(dest, uint16(raw.len))
for b in raw:
dest.add(b)
proc encodePart*(dest: var seq[byte], i: int64) =
appendBE64(dest, cast[uint64](i) xor SignFlip)
proc encodePart*(dest: var seq[byte], u: uint64) =
appendBE64(dest, u)
proc encodePart*(dest: var seq[byte], i: int) {.inline.} =
encodePart(dest, i.int64)
proc encodePart*(dest: var seq[byte], i: int32) {.inline.} =
encodePart(dest, i.int64)
proc encodePart*(dest: var seq[byte], i: int16) {.inline.} =
encodePart(dest, i.int64)
proc encodePart*(dest: var seq[byte], i: int8) {.inline.} =
encodePart(dest, i.int64)
proc encodePart*(dest: var seq[byte], u: uint32) {.inline.} =
encodePart(dest, u.uint64)
proc encodePart*(dest: var seq[byte], u: uint16) {.inline.} =
encodePart(dest, u.uint64)
proc encodePart*(dest: var seq[byte], b: bool) =
dest.add(if b: 1'u8 else: 0'u8)
proc encodePart*(dest: var seq[byte], b: byte) =
dest.add(b)
proc encodePart*(dest: var seq[byte], c: char) =
dest.add(byte(c))
proc encodePart*(dest: var seq[byte], k: Key) =
## Embed an already-encoded Key (e.g. a pre-built prefix) verbatim.
for b in bytes(k):
dest.add(b)
# ── encodePart: generic structural fallback ─────────────────────────────
proc encodePart*[E: enum](dest: var seq[byte], v: E) {.inline.} =
encodePart(dest, int64(ord(v)))
proc encodePart*[T: tuple | object](dest: var seq[byte], v: T) =
## Walks the type's fields in declaration order. Each field must itself
## have an `encodePart` overload (primitive, Key, or another struct).
for f in fields(v):
encodePart(dest, f)
# ── Public Key constructors ─────────────────────────────────────────────
proc add*[T](k: var Key, v: T) =
## In-place key extension. Equivalent to writing `encodePart` against the
## underlying byte buffer.
var buf = seq[byte](k)
encodePart(buf, v)
k = Key(buf)
proc toKey*[T](v: T): Key =
## Single-value Key constructor. Equivalent to `key(v)`.
var buf: seq[byte] = @[]
encodePart(buf, v)
return Key(buf)
macro key*(parts: varargs[typed]): Key =
## Variadic Key builder. Accepts any mix of types for which `encodePart`
## resolves -- including tuples and objects via the structural fallback.
##
## ```nim
## key() # empty Key
## key("ch", 1'i64) # 2-component
## key("ch", (1'i64, 7'u64)) # nested tuple flattens
## ```
let bufSym = genSym(nskVar, "keyBuf")
var body = newStmtList()
body.add quote do:
var `bufSym`: seq[byte] = @[]
for p in parts:
body.add quote do:
encodePart(`bufSym`, `p`)
body.add quote do:
Key(`bufSym`)
return newBlockStmt(body)
# ── Range helpers ───────────────────────────────────────────────────────
proc prefixRange*(prefix: Key): KeyRange =
## Build [prefix, prefix++) — a half-open range that captures every key
## starting with `prefix`. If `prefix` is all 0xFF, the upper bound is
## empty (open-ended); the backend treats `stop.len == 0` as "no upper
## bound".
var stop = bytes(prefix)
var i = stop.len - 1
while i >= 0:
if stop[i] != 0xFF'u8:
stop[i] = stop[i] + 1'u8
stop.setLen(i + 1)
return KeyRange(start: prefix, stop: Key(stop))
dec i
return KeyRange(start: prefix, stop: Key(@[]))
{.pop.}

53
waku/persistency/payload.nim Normal file
View File

@ -0,0 +1,53 @@
## Generic payload encoding.
##
## Symmetric with `keys.nim`: reuses the same `encodePart` family so any
## Nim type composable from primitives + tuples/objects can be turned
## into a `seq[byte]` for storage. Unlike keys, payloads do **not** need
## byte-wise lex order — but using the same encoder keeps the system
## small. If a tenant needs a different on-disk format (CBOR, protobuf,
## SSZ, ...) they can write their own `toPayload` overload or pass an
## already-encoded `seq[byte]` to `persistPut`.
##
## ```nim
## # Primitives:
## let p1 = payload("hello") # length-prefixed string bytes
## let p2 = payload(42'i64) # 8 bytes, sign-flipped BE
##
## # Composites:
## type Msg = object
## sender: string
## epoch: int64
## body: seq[byte]
## let p3 = toPayload(Msg(sender: "alice", epoch: 7, body: @[1'u8, 2, 3]))
##
## # Variadic when you want multiple values back-to-back:
## let p4 = payload("v1", 1'i64, body)
## ```
{.push raises: [].}
import std/macros
import ./keys
export keys.encodePart
proc toPayload*[T](v: T): seq[byte] =
## Single-value payload constructor. Equivalent to `payload(v)`.
var buf: seq[byte] = @[]
encodePart(buf, v)
return buf
macro payload*(parts: varargs[typed]): seq[byte] =
## Variadic payload builder. Same encoder as `key(...)`; only the return
## type differs.
let bufSym = genSym(nskVar, "payloadBuf")
var body = newStmtList()
body.add quote do:
var `bufSym`: seq[byte] = @[]
for p in parts:
body.add quote do:
encodePart(`bufSym`, `p`)
body.add bufSym
return newBlockStmt(body)
{.pop.}

433
waku/persistency/persistency.nim Normal file
View File

@ -0,0 +1,433 @@
## Public facade and main driver types for the persistency library.
##
## ``Persistency`` is the per-root coordinator; one instance owns one
## directory and any number of named jobs. ``Job`` is the per-job handle:
## one tenant, one DB file, one worker thread, one BrokerContext.
##
## ## Two ways to drive a job
##
## **By Job ref** — capture the handle from `openJob` and call methods on
## it. Cheapest, no map lookup per call:
##
## ```nim
## let p = Persistency.instance("/var/lib/wakustore").get()
## let j = p.openJob("alpha").get()
## await j.persistPut("msg", k, payload)
## let v = await j.get("msg", k)
## ```
##
## **By job id string** — useful when the caller doesn't want to thread
## the ``Job`` ref around (config-driven services, RPC dispatchers). The
## Job must still have been opened previously; the string-form procs look
## it up in `Persistency.jobs`:
##
## ```nim
## discard p.openJob("alpha")
## await p.persistPut("alpha", "msg", k, payload) # logs and resolves if not open
## let v = await p.get("alpha", "msg", k) # Result, peJobNotFound if missing
## ```
##
## ## Drain semantics
##
## Writes return a ``Future[void]`` that resolves once the PersistEvent
## has been pushed onto the worker thread's channel — **not** once the
## SQL has run. The listener is still fire-and-forget on the SQL side, so
## a read issued immediately after an awaited write is still racy by
## design in v1. To bridge the race:
## * use ``deleteAcked`` (it round-trips through the read path), or
## * poll ``exists`` until it returns true, or
## * yield with ``await sleepAsync(...)``.
{.push raises: [].}
import std/[locks, options, os, sequtils, tables]
import chronos, chronicles, results
import brokers/[event_broker, request_broker, broker_context]
import ./[types, keys, payload, backend_comm, backend_thread]
export types, keys, payload
logScope:
topics = "persistency"
const DefaultStoragePath* = "./data"
# ── Driver types ────────────────────────────────────────────────────────
type
Job* = ref object
## Per-job handle. Owns its BrokerContext and the worker thread that
## services it. Created and torn down via `Persistency.openJob` /
## `Persistency.closeJob`.
id*: string
context*: BrokerContext
runtime: JobRuntime ## internal — managed by openJob/closeJob
running*: bool
Persistency* = ref object
## Per-root coordinator. One Persistency instance manages a directory
## of per-job SQLite files at ``rootDir/<jobId>.db``.
rootDir*: string
jobs*: Table[string, Job]
# ── Singleton state ─────────────────────────────────────────────────────
#
# Persistency is a process-wide singleton: one rootDir at a time. The
# `instance` factory is the only public constructor; `new` below is
# private and skips the singleton bookkeeping (used internally and never
# called twice with conflicting rootDirs).
var
gPersistency {.global.}: Persistency
gPersistencyLock {.global.}: Lock
once:
gPersistencyLock.initLock()
# ── Lifecycle ───────────────────────────────────────────────────────────
proc dbPathFor(p: Persistency, jobId: string): string =
p.rootDir / (jobId & ".db")
proc new(T: type Persistency, rootDir: string): Result[T, PersistencyError] =
## Private. Build a Persistency value without touching the singleton
## slot. Validates ``rootDir`` but does **not** create it — directory
## materialisation is deferred to the first ``openJob`` call. Semantics:
##
## * If ``rootDir`` is empty, returns ``peInvalidArgument``.
## * If ``rootDir`` exists and is a directory, accept it.
## * If ``rootDir`` exists but is not a directory, returns
## ``peInvalidArgument``.
## * If ``rootDir`` does not exist, walk up the parent chain: the first
## existing ancestor must be a directory; otherwise returns
## ``peInvalidArgument``. This catches "obviously broken" paths early
## without actually touching the filesystem.
if rootDir.len == 0:
return err(persistencyErr(peInvalidArgument, "rootDir is empty"))
if fileExists(rootDir) and not dirExists(rootDir):
return err(
persistencyErr(
peInvalidArgument, "rootDir exists and is not a directory: " & rootDir
)
)
if not dirExists(rootDir):
var parent = parentDir(rootDir)
while parent.len > 0 and not dirExists(parent):
if fileExists(parent):
return err(
persistencyErr(
peInvalidArgument,
"rootDir ancestor exists and is not a directory: " & parent,
)
)
parent = parentDir(parent)
return ok(T(rootDir: rootDir, jobs: initTable[string, Job]()))
proc ensureRootDir(p: Persistency): Result[void, PersistencyError] =
## Materialise ``rootDir`` on demand. Idempotent; called from
## ``openJob`` so an unused Persistency leaves no directory behind.
if dirExists(p.rootDir):
return ok()
try:
createDir(p.rootDir)
except OSError, IOError:
return
err(persistencyErr(peBackend, "createDir failed: " & getCurrentExceptionMsg()))
return ok()
proc reset*(T: type Persistency) {.gcsafe.} =
## Tear down the singleton: close every open job, clear the Teardown
## provider, and free the slot so a subsequent ``Persistency.instance``
## starts fresh. Idempotent. Tests use this in `defer`;.
{.cast(gcsafe).}:
acquire(gPersistencyLock)
defer:
release(gPersistencyLock)
if gPersistency != nil:
let p = gPersistency
gPersistency = nil
p.close()
proc instance*(
T: type Persistency, rootDir: string
): Result[T, PersistencyError] {.gcsafe.} =
## Get-or-init the process-wide Persistency singleton.
##
## * First call: validates ``rootDir`` (without creating it) and
## registers the Teardown handler. The directory itself is created
## lazily by the first ``openJob`` call, so a Persistency that never
## opens a job leaves no filesystem footprint.
## * Later calls with the same ``rootDir``: returns the live instance
## (idempotent).
## * Later calls with a different ``rootDir``: returns
## ``peInvalidArgument`` — the singleton can only be re-targeted via
## ``Persistency.reset`` (or by the Teardown shutdown flow).
{.cast(gcsafe).}:
acquire(gPersistencyLock)
defer:
release(gPersistencyLock)
if gPersistency != nil:
if gPersistency.rootDir == rootDir:
return ok(gPersistency)
return err(
persistencyErr(
peInvalidArgument,
"Persistency already initialised with rootDir " & gPersistency.rootDir &
"; cannot re-init with " & rootDir,
)
)
let p = ?Persistency.new(rootDir)
gPersistency = p
return ok(p)
proc instance*(T: type Persistency): Result[T, PersistencyError] {.gcsafe.} =
## No-args form: succeeds only if the singleton is already initialised.
## Use this from services that must not be the first to touch
## persistency.
{.cast(gcsafe).}:
acquire(gPersistencyLock)
defer:
release(gPersistencyLock)
if gPersistency.isNil:
return err(persistencyErr(peClosed, "Persistency not initialised"))
return ok(gPersistency)
proc openJob*(p: Persistency, jobId: string): Result[Job, PersistencyError] =
## Open-or-create a job under this Persistency.
##
## * If the job is already open in this process, the existing ``Job``
## ref is returned (idempotent).
## * Otherwise ``rootDir`` is materialised on demand (created with
## missing parents on first use; no-op on subsequent calls), a worker
## thread is spawned, and the SQLite file at
## ``<rootDir>/<jobId>.db`` is opened. If the file does not exist it
## is created and the schema initialised; if it already exists it is
## reopened in place and its data is preserved.
let existing = p.jobs.getOrDefault(jobId, nil)
if existing != nil:
return ok(existing)
?p.ensureRootDir()
let ctx = NewBrokerContext()
let rt = ?startStorageThread(ctx, dbPathFor(p, jobId))
let job = Job(id: jobId, context: ctx, runtime: rt, running: true)
p.jobs[jobId] = job
return ok(job)
proc closeJob*(p: Persistency, jobId: string) =
## Stop the worker, join its thread, and forget the job. No-op if the
## job isn't open.
let job = p.jobs.getOrDefault(jobId, nil)
if job == nil:
return
stopStorageThread(job.runtime)
job.runtime = nil
job.running = false
p.jobs.del(jobId)
proc close*(p: Persistency) =
## Close every open job. Idempotent.
var ids: seq[string]
for id in p.jobs.keys:
ids.add(id)
for id in ids:
p.closeJob(id)
proc dropJob*(p: Persistency, jobId: string) =
## Close the job if open, then delete its DB file (plus -wal / -shm
## sidecars). Best-effort: a missing file is not an error.
p.closeJob(jobId)
let path = dbPathFor(p, jobId)
for suffix in ["", "-wal", "-shm"]:
try:
removeFile(path & suffix)
except OSError, IOError:
discard
# ── String lookup ───────────────────────────────────────────────────────
proc job*(p: Persistency, jobId: string): Result[Job, PersistencyError] =
## Look up an already-open job. Returns ``peJobNotFound`` if no such
## job has been opened (``openJob`` first).
let j = p.jobs.getOrDefault(jobId, nil)
if j != nil:
return ok(j)
else:
return err(persistencyErr(peJobNotFound, "no open job with id: " & jobId))
proc `[]`*(p: Persistency, jobId: string): Job {.raises: [KeyError].} =
## Subscript sugar for `job` — raises ``KeyError`` if the job isn't
## open. Prefer `job(p, id)` when you want a typed error.
p.jobs[jobId]
proc hasJob*(p: Persistency, jobId: string): bool {.inline.} =
p.jobs.hasKey(jobId)
# ── Writes (fire-and-forget) — Job form ─────────────────────────────────
proc persist*(t: Job, ops: seq[TxOp]): Future[void] {.async.} =
## Emit a batched persist event. The handler treats >1 ops as a single
## BEGIN IMMEDIATE/COMMIT transaction (see backend_sqlite.applyOps).
await PersistEvent.emit(t.context, PersistEvent(ops: ops))
proc persist*(t: Job, op: TxOp): Future[void] {.async.} =
await persist(t, @[op])
proc persistPut*(
t: Job, category: string, key: Key, payload: seq[byte]
): Future[void] {.async.} =
await persist(t, TxOp(category: category, key: key, kind: txPut, payload: payload))
proc persistDelete*(t: Job, category: string, key: Key): Future[void] {.async.} =
await persist(t, TxOp(category: category, key: key, kind: txDelete))
proc persistEncoded*[T](
t: Job, category: string, key: Key, value: T
): Future[void] {.async.} =
## Convenience: encode `value` via `toPayload` and put it. Use the raw
## `persistPut(..., seq[byte])` form when you already have bytes
## (e.g. an externally-produced CBOR blob).
await persistPut(t, category, key, toPayload(value))
# ── Writes (fire-and-forget) — string-lookup form ───────────────────────
#
# These look up the Job by id and dispatch. If the job isn't open we log
# a warning and drop the write — consistent with the fire-and-forget
# contract; the caller has no return channel to inspect.
proc jobOrWarn(p: Persistency, jobId: string): Job =
## Lookup helper for the fire-and-forget write paths. Returns nil and
## logs a warning if the job isn't open. Isolated as a non-generic proc
## so chronicles' `warn` macro expands cleanly (it doesn't, when called
## from inside a generic proc's body).
let job = p.jobs.getOrDefault(jobId, nil)
if job.isNil():
warn "persistency: write dropped, job not open", jobId
return job
template withJobOrWarn(p: Persistency, jobId: string, j, body: untyped) =
let `j` = p.jobOrWarn(jobId)
if not `j`.isNil():
body
proc persist*(p: Persistency, jobId: string, ops: seq[TxOp]): Future[void] {.async.} =
let j = p.jobOrWarn(jobId)
if not j.isNil():
await j.persist(ops)
proc persist*(p: Persistency, jobId: string, op: TxOp): Future[void] {.async.} =
await p.persist(jobId, @[op])
proc persistPut*(
p: Persistency, jobId: string, category: string, key: Key, payload: seq[byte]
): Future[void] {.async.} =
let j = p.jobOrWarn(jobId)
if not j.isNil():
await j.persistPut(category, key, payload)
proc persistDelete*(
p: Persistency, jobId: string, category: string, key: Key
): Future[void] {.async.} =
let j = p.jobOrWarn(jobId)
if not j.isNil():
await j.persistDelete(category, key)
proc persistEncoded*[T](
p: Persistency, jobId: string, category: string, key: Key, value: T
): Future[void] {.async.} =
let j = p.jobOrWarn(jobId)
if not j.isNil():
await j.persistEncoded(category, key, value)
# ── Reads (async, typed errors) — Job form ──────────────────────────────
template liftErr(s: string): PersistencyError =
decodeErr(s)
proc get*(
t: Job, category: string, key: Key
): Future[Result[Option[seq[byte]], PersistencyError]] {.async.} =
let r = (await KvGet.request(t.context, category, key)).valueOr:
return err(liftErr(error))
return ok(r.value)
proc exists*(
t: Job, category: string, key: Key
): Future[Result[bool, PersistencyError]] {.async.} =
let r = (await KvExists.request(t.context, category, key)).valueOr:
return err(liftErr(error))
return ok(r.value)
proc scan*(
t: Job, category: string, range: KeyRange, reverse = false
): Future[Result[seq[KvRow], PersistencyError]] {.async.} =
let r = (await KvScan.request(t.context, category, range, reverse)).valueOr:
return err(liftErr(error))
return ok(r.rows)
proc scanPrefix*(
t: Job, category: string, prefix: Key, reverse = false
): Future[Result[seq[KvRow], PersistencyError]] {.async.} =
let rng = prefixRange(prefix)
let r = (await KvScan.request(t.context, category, rng, reverse)).valueOr:
return err(liftErr(error))
return ok(r.rows)
proc count*(
t: Job, category: string, range: KeyRange
): Future[Result[int, PersistencyError]] {.async.} =
let r = (await KvCount.request(t.context, category, range)).valueOr:
return err(liftErr(error))
return ok(r.n)
proc deleteAcked*(
t: Job, category: string, key: Key
): Future[Result[bool, PersistencyError]] {.async.} =
## Goes through the read path so the caller learns whether a row was
## actually removed.
let r = (await KvDelete.request(t.context, category, key)).valueOr:
return err(liftErr(error))
return ok(r.existed)
# ── Reads (async, typed errors) — string-lookup form ────────────────────
proc get*(
p: Persistency, jobId: string, category: string, key: Key
): Future[Result[Option[seq[byte]], PersistencyError]] {.async.} =
let j = ?p.job(jobId)
return await j.get(category, key)
proc exists*(
p: Persistency, jobId: string, category: string, key: Key
): Future[Result[bool, PersistencyError]] {.async.} =
let j = ?p.job(jobId)
return await j.exists(category, key)
proc scan*(
p: Persistency, jobId: string, category: string, range: KeyRange, reverse = false
): Future[Result[seq[KvRow], PersistencyError]] {.async.} =
let j = ?p.job(jobId)
return await j.scan(category, range, reverse)
proc scanPrefix*(
p: Persistency, jobId: string, category: string, prefix: Key, reverse = false
): Future[Result[seq[KvRow], PersistencyError]] {.async.} =
let j = ?p.job(jobId)
return await j.scanPrefix(category, prefix, reverse)
proc count*(
p: Persistency, jobId: string, category: string, range: KeyRange
): Future[Result[int, PersistencyError]] {.async.} =
let j = ?p.job(jobId)
return await j.count(category, range)
proc deleteAcked*(
p: Persistency, jobId: string, category: string, key: Key
): Future[Result[bool, PersistencyError]] {.async.} =
let j = ?p.job(jobId)
return await j.deleteAcked(category, key)
{.pop.}

58
waku/persistency/schema.nim Normal file
View File

@ -0,0 +1,58 @@
## SQL schema and pragma setup for the persistency library.
##
## Single uniform schema per job DB file:
## kv(category BLOB, key BLOB, payload BLOB) PRIMARY KEY (category, key)
## WITHOUT ROWID
##
## category is declared BLOB (not TEXT) so it round-trips via the existing
## sqlite3_abi binding helpers (which do not yet expose bind_text). SQLite
## compares BLOBs byte-wise, which is exactly the ordering we want.
{.push raises: [].}
import results
import ../common/databases/[common, db_sqlite]
const
PersistencyUserVersion* = 1'i64
CreateKvTableSql* = """
CREATE TABLE IF NOT EXISTS kv (
category BLOB NOT NULL,
key BLOB NOT NULL,
payload BLOB NOT NULL,
PRIMARY KEY (category, key)
) WITHOUT ROWID;
"""
ApplyPragmasSql* = """
PRAGMA synchronous = NORMAL;
PRAGMA temp_store = MEMORY;
PRAGMA busy_timeout = 5000;
PRAGMA foreign_keys = OFF;
"""
proc applyPragmas*(db: SqliteDatabase): DatabaseResult[void] =
## Apply the connection-level pragmas. journal_mode=WAL is already set by
## SqliteDatabase.new.
for stmt in [
"PRAGMA synchronous = NORMAL;", "PRAGMA temp_store = MEMORY;",
"PRAGMA busy_timeout = 5000;", "PRAGMA foreign_keys = OFF;",
]:
db.query(stmt, NoopRowHandler).isOkOr:
return err("pragma failed: " & stmt & ": " & error)
return ok()
proc ensureSchema*(db: SqliteDatabase): DatabaseResult[void] =
db.query(CreateKvTableSql, NoopRowHandler).isOkOr:
return err("create kv table failed: " & error)
let userVersion = ?db.getUserVersion()
if userVersion == 0:
?db.setUserVersion(PersistencyUserVersion)
elif userVersion != PersistencyUserVersion:
return err(
"incompatible persistency user_version: got " & $userVersion & ", expected " &
$PersistencyUserVersion
)
return ok()

81
waku/persistency/types.nim Normal file
View File

@ -0,0 +1,81 @@
## Core types for the logos-delivery persistency library.
##
## The library is backend-neutral CRUD: jobs own their domain ports and
## map them onto the primitives exposed in persistency.nim. See
## persistency.nim for the public facade and brokers.nim for the
## cross-thread plumbing.
{.push raises: [].}
type
Key* = distinct seq[byte]
KeyRange* = object
start*: Key
stop*: Key ## exclusive; an empty `stop` means "no upper bound"
KvRow* = tuple[key: Key, payload: seq[byte]]
TxOpKind* = enum
txPut
txDelete
TxOp* = object
category*: string
key*: Key
case kind*: TxOpKind
of txPut:
payload*: seq[byte]
of txDelete:
discard
PersistencyErrorKind* = enum
peBackend
peClosed
peInvalidArgument
peTimeout
peJobNotFound
PersistencyError* = object
kind*: PersistencyErrorKind
msg*: string
backendCode*: int
proc bytes*(k: Key): lent seq[byte] {.inline.} =
seq[byte](k)
proc len*(k: Key): int {.inline.} =
seq[byte](k).len
proc `==`*(a, b: Key): bool {.inline.} =
seq[byte](a) == seq[byte](b)
proc `<`*(a, b: Key): bool =
let ab = seq[byte](a)
let bb = seq[byte](b)
let n = min(ab.len, bb.len)
for i in 0 ..< n:
if ab[i] != bb[i]:
return ab[i] < bb[i]
return ab.len < bb.len
proc `<=`*(a, b: Key): bool {.inline.} =
a == b or a < b
proc rawKey*(b: openArray[byte]): Key =
var s = newSeq[byte](b.len)
for i, v in b:
s[i] = v
return Key(s)
proc rawKey*(b: sink seq[byte]): Key {.inline.} =
Key(b)
proc persistencyErr*(
kind: PersistencyErrorKind, msg: string, backendCode = 0
): PersistencyError {.inline.} =
PersistencyError(kind: kind, msg: msg, backendCode: backendCode)
proc `$`*(e: PersistencyError): string =
"PersistencyError(" & $e.kind & ": " & e.msg &
(if e.backendCode != 0: ", code=" & $e.backendCode else: "") & ")"

2
waku/requests/health_requests.nim
View File

@ -1,4 +1,4 @@
import waku/common/broker/request_broker
import brokers/request_broker
import waku/api/types
import waku/node/health_monitor/[protocol_health, topic_health, health_report]

2
waku/requests/node_requests.nim
View File

@ -1,5 +1,5 @@
import std/options
import waku/common/broker/[request_broker, multi_request_broker]
import brokers/[request_broker, multi_request_broker]
import waku/waku_core/[topics]
RequestBroker(sync):

3
waku/requests/rln_requests.nim
View File

@ -1,4 +1,5 @@
import waku/common/broker/request_broker, waku/waku_core/message/message
import brokers/request_broker
import waku/waku_core/message/message
RequestBroker:
type RequestGenerateRlnProof* = object

7
waku/waku_filter_v2/client.nim
View File

@ -8,10 +8,11 @@ import
chronos,
libp2p/protocols/protocol,
bearssl/rand,
stew/byteutils
stew/byteutils,
brokers/broker_context
import
waku/
[node/peer_manager, waku_core, events/delivery_events, common/broker/broker_context],
waku/[node/peer_manager, waku_core, events/delivery_events],
./common,
./protocol_metrics,
./rpc_codec,

6
waku/waku_relay/protocol.nim
View File

@ -16,7 +16,8 @@ import
libp2p/protocols/pubsub/gossipsub,
libp2p/protocols/pubsub/rpc/messages,
libp2p/stream/connection,
libp2p/switch
libp2p/switch,
brokers/broker_context
import
waku/waku_core,
@ -24,7 +25,6 @@ import
waku/requests/health_requests,
waku/events/health_events,
./message_id,
waku/common/broker/broker_context,
waku/events/peer_events
from waku/waku_core/codecs import WakuRelayCodec
@ -526,7 +526,7 @@ method stop*(w: WakuRelay) {.async: (raises: []).} =
info "stop"
await procCall GossipSub(w).stop()
WakuPeerEvent.dropListener(w.brokerCtx, w.peerEventListener)
await WakuPeerEvent.dropListener(w.brokerCtx, w.peerEventListener)
if not w.topicHealthLoopHandle.isNil():
await w.topicHealthLoopHandle.cancelAndWait()

5
waku/waku_rln_relay/rln_relay.nim
View File

@ -13,7 +13,9 @@ import
libp2p/protocols/pubsub/rpc/messages,
libp2p/protocols/pubsub/pubsub,
results,
stew/[byteutils, arrayops]
stew/[byteutils, arrayops],
brokers/broker_context
import
./group_manager,
./rln,
@ -30,7 +32,6 @@ import
waku_core,
requests/rln_requests,
waku_keystore,
common/broker/broker_context,
]
logScope: