mirror of
https://github.com/logos-messaging/logos-delivery.git
synced 2026-05-20 21:59:38 +00:00
* persistency: per-job SQLite-backed storage layer (singleton, brokered)
Adds a backend-neutral CRUD library at waku/persistency/, plus the
nim-brokers dependency swap that enables it.
Architecture (ports-and-adapters):
* Persistency: process-wide singleton, one root directory.
* Job: one tenant, one DB file, one worker thread, one BrokerContext.
* Backend: SQLite via waku/common/databases/db_sqlite. Uniform schema
kv(category BLOB, key BLOB, payload BLOB) PRIMARY KEY (category, key)
WITHOUT ROWID, WAL mode.
* Writes are fire-and-forget via EventBroker(mt) PersistEvent.
* Reads are async via five RequestBroker(mt) shapes (KvGet, KvExists,
KvScan, KvCount, KvDelete). Reads return Result[T, PersistencyError].
* One storage thread per job; tenants isolated by BrokerContext.
Public surface (waku/persistency/persistency.nim):
Persistency.instance(rootDir) / Persistency.instance() / Persistency.reset()
p.openJob(id) / p.closeJob(id) / p.dropJob(id) / p.close()
p.job(id) / p[id] / p.hasJob(id)
Writes (Job form & string-id form, fire-and-forget):
persist / persistPut / persistDelete / persistEncoded
Reads (Job form & string-id form, async Result):
get / exists / scan / scanPrefix / count / deleteAcked
Key & payload encoding (keys.nim, payload.nim):
* encodePart family + variadic key(...) / payload(...) macros +
single-value toKey / toPayload.
* Primitives: string and openArray[byte] are 2-byte BE length + bytes;
int{8..64} are sign-flipped 8-byte BE; uint{16..64} are 8-byte BE;
bool/byte/char are 1 byte; enums are int64(ord(v)).
* Generic encodePart[T: tuple | object] recurses through fields() so
any composite Nim type is encodable without ceremony.
* Stable across Nim/C compiler upgrades: no sizeof, no memcpy, no
cast on pointers, no host-endianness dependency.
* `rawKey(bytes)` + `persistPut(..., openArray[byte])` let callers
bypass the built-in encoder with their own format (CBOR, protobuf...).
Lifecycle:
* Persistency.new is private; Persistency.instance is the only public
constructor. Same rootDir is idempotent; conflicting rootDir is
peInvalidArgument. Persistency.reset for test/restart paths.
* openJob opens-or-creates the per-job SQLite file; an existing file
is reused with its data preserved.
* Teardown integration: Persistency.instance registers a Teardown
MultiRequestBroker provider that closes all jobs and clears the
singleton slot when Waku.stop() issues Teardown.request.
Internal layering:
types.nim pure value types (Key, KeyRange, KvRow, TxOp,
PersistencyError)
keys.nim encodePart primitives + key(...) macro
payload.nim toPayload + payload(...) macro
schema.nim CREATE TABLE + connection pragmas + user_version
backend_sqlite.nim KvBackend, applyOps (single source of write SQL),
getOne/existsOne/deleteOne, scanRange (asc/desc,
half-open ranges, open-ended stop), countRange
backend_comm.nim EventBroker(mt) PersistEvent + 5 RequestBroker(mt)
declarations; encodeErr/decodeErr boundary helpers
backend_thread.nim startStorageThread / stopStorageThread (shared
allocShared0 arg, cstring dbPath, atomic
ready/shutdown flags); per-thread provider
registration
persistency.nim Persistency + Job types, singleton state, public
facade
../requests/lifecycle_requests.nim
Teardown MultiRequestBroker
Tests (69 cases, all passing):
test_keys.nim sort-order invariants (length-prefix strings,
sign-flipped ints, composite tuples, prefix
range)
test_backend.nim round-trip / replace / delete-return-value /
batched atomicity / asc-desc-half-open-open-
ended scans / category isolation / batch
txDelete
test_lifecycle.nim open-or-create rootDir / non-dir collision /
reopen across sessions / idempotent openJob /
two-tenant parallel isolation / closeJob joins
worker / dropJob removes file / acked delete
test_facade.nim put-then-get / atomic batch / scanPrefix
asc/desc / deleteAcked hit-miss /
fire-and-forget delete / two-tenant facade
isolation
test_encoding.nim tuple/named-tuple/object keys, embedded Key,
enum encoding, field-major composite sort,
payload struct encoding, end-to-end struct
round-trip through SQLite
test_string_lookup.nim peJobNotFound semantics / hasJob / subscript /
persistPut+get via id / reads short-circuit /
writes drop+warn / persistEncoded via id /
scan parity Job-ref vs id
test_singleton.nim idempotent same-rootDir / different-rootDir
rejection / no-arg instance lifecycle / reset
retargets / reset idempotence / Teardown.request
end-to-end
Prerequisite delivered in the same series: replace the in-tree broker
implementation with the external nim-brokers package; update all
broker call-sites (waku_filter_v2, waku_relay, waku_rln_relay,
delivery_service, peer_manager, requests/*, factory/*, api tests, etc.)
to the new package API; chat2 made to compile again.
Note: SDS adapter (Phase 5 of the design) is deferred -- nim-sds is
still developed side-by-side and the persistency layer is intentionally
SDS-agnostic.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* persistency: pin nim-brokers by URL+commit (workaround for stale registry)
The bare `brokers >= 2.0.1` form cannot resolve on machines where the
local nimble SAT solver enumerates only the registry-recorded 0.1.0 for
brokers. The nim-lang/packages entry for `brokers` carries no per-tag
metadata (only the URL), so until that registry entry is refreshed the
SAT solver clamps the available-versions list to 0.1.0 and rejects the
>= 2.0.1 constraint -- even though pkgs2 and pkgcache both have v2.0.1
cloned locally.
Pinning by URL+commit bypasses the registry path entirely. Inline
comment in waku.nimble documents the situation and the path back to
the bare form once nim-lang/packages is updated.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* persistency: nph format pass
Run `nph` on all 57 Nim files touched by this PR. Pure formatting:
17 files re-styled, no semantic change. Suite still 69/69.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Fix build, add local-storage-path config, lazy init of Persistency from Waku start
* fix: fix nix deps
* fixes for nix build, regenerate deps
* reverting accidental dependency changes
* Fixing deps
* Apply suggestions from code review
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* persistency tests: migrate to suite / asyncTest / await
Match the in-tree test convention (procSuite -> suite, sync test +
waitFor -> asyncTest + await):
- procSuite "X": -> suite "X":
- For tests doing async work: test -> asyncTest, waitFor -> await.
- Poll helpers (proc waitFor(t: Job, ...) in test_lifecycle.nim,
proc waitUntilExists(...) in test_facade.nim and
test_string_lookup.nim) -> Future[bool] {.async.}, internal
`waitFor X` -> `await X`, internal `sleep(N)` ->
`await sleepAsync(chronos.milliseconds(N))`.
- Renamed test_lifecycle.nim's helper proc from `waitFor(t: Job, ...)`
-> `pollExists(t: Job, ...)`; the previous name shadowed
chronos.waitFor in the chronos macro expansion.
- `chronos.milliseconds(N)` explicitly qualified because `std/times`
also exports `milliseconds` (returning TimeInterval, not Duration).
- `check await x` -> `let okN = await x; check okN` to dodge chronos's
"yield in expr not lowered" with await-as-macro-argument.
- `(await x).foo()` -> `let awN = await x; ... awN.foo() ...` for the
same reason.
waku/persistency/persistency.nim: nph also pulled the proc signatures
across multiple lines; restored explicit `Future[void] {.async.}`
return types after the colon (an intermediate nph pass had elided them).
Suite: 71 / 71 OK against the new async write surface.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* use idiomatic valueOr instead of ifs
* Reworked persistency shutdown, remove not necessary teardown mechanism
* Use const for DefaultStoragePath
* format to follow coding guidelines - no use of result and explicit returns - no functional change
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
796 lines
26 KiB
Nim
796 lines
26 KiB
Nim
{.used.}
|
|
|
|
import
|
|
std/[sequtils, strformat, tempfiles, osproc],
|
|
stew/byteutils,
|
|
testutils/unittests,
|
|
presto,
|
|
presto/client as presto_client,
|
|
libp2p/crypto/crypto
|
|
import brokers/broker_context
|
|
import
|
|
waku/[
|
|
common/base64,
|
|
waku_core,
|
|
waku_node,
|
|
rest_api/message_cache,
|
|
rest_api/endpoint/server,
|
|
rest_api/endpoint/client,
|
|
rest_api/endpoint/responses,
|
|
rest_api/endpoint/relay/types,
|
|
rest_api/endpoint/relay/handlers as relay_rest_interface,
|
|
rest_api/endpoint/relay/client as relay_rest_client,
|
|
waku_relay,
|
|
waku_rln_relay,
|
|
],
|
|
../testlib/wakucore,
|
|
../testlib/wakunode,
|
|
../resources/payloads,
|
|
../waku_rln_relay/[rln/waku_rln_relay_utils, utils_onchain]
|
|
|
|
proc testWakuNode(): WakuNode =
|
|
let
|
|
privkey = generateSecp256k1Key()
|
|
bindIp = parseIpAddress("0.0.0.0")
|
|
extIp = parseIpAddress("127.0.0.1")
|
|
port = Port(0)
|
|
|
|
newTestWakuNode(privkey, bindIp, port, some(extIp), some(port))
|
|
|
|
suite "Waku v2 Rest API - Relay":
|
|
var anvilProc {.threadVar.}: Process
|
|
var manager {.threadVar.}: OnchainGroupManager
|
|
|
|
setup:
|
|
anvilProc = runAnvil(stateFile = some(DEFAULT_ANVIL_STATE_PATH))
|
|
manager = waitFor setupOnchainGroupManager(deployContracts = false)
|
|
|
|
teardown:
|
|
stopAnvil(anvilProc)
|
|
|
|
asyncTest "Subscribe a node to an array of pubsub topics - POST /relay/v1/subscriptions":
|
|
# Given
|
|
let node = testWakuNode()
|
|
await node.start()
|
|
(await node.mountRelay()).isOkOr:
|
|
assert false, "Failed to mount relay"
|
|
|
|
var restPort = Port(0)
|
|
|
|
let restAddress = parseIpAddress("0.0.0.0")
|
|
let restServer = WakuRestServerRef.init(restAddress, restPort).tryGet()
|
|
|
|
restPort = restServer.httpServer.address.port # update with bound port for client use
|
|
|
|
let cache = MessageCache.init()
|
|
|
|
installRelayApiHandlers(restServer.router, node, cache)
|
|
restServer.start()
|
|
|
|
let
|
|
shard0 = RelayShard(clusterId: DefaultClusterId, shardId: 0)
|
|
shard1 = RelayShard(clusterId: DefaultClusterId, shardId: 1)
|
|
shard2 = RelayShard(clusterId: DefaultClusterId, shardId: 2)
|
|
|
|
let shards = @[$shard0, $shard1, $shard2]
|
|
|
|
let invalidTopic = "/test/2/this/is/a/content/topic/1"
|
|
|
|
var containsIncorrect = shards
|
|
containsIncorrect.add(invalidTopic)
|
|
|
|
# When contains incorrect pubsub topics, subscribe shall fail
|
|
let client = newRestHttpClient(initTAddress(restAddress, restPort))
|
|
let errorResponse = await client.relayPostSubscriptionsV1(containsIncorrect)
|
|
|
|
# Then
|
|
check:
|
|
errorResponse.status == 400
|
|
$errorResponse.contentType == $MIMETYPE_TEXT
|
|
errorResponse.data ==
|
|
"Invalid pubsub topic(s): @[\"/test/2/this/is/a/content/topic/1\"]"
|
|
|
|
# when all pubsub topics are correct, subscribe shall succeed
|
|
let response = await client.relayPostSubscriptionsV1(shards)
|
|
|
|
# Then
|
|
check:
|
|
response.status == 200
|
|
$response.contentType == $MIMETYPE_TEXT
|
|
response.data == "OK"
|
|
|
|
check:
|
|
cache.isPubsubSubscribed($shard0)
|
|
cache.isPubsubSubscribed($shard1)
|
|
cache.isPubsubSubscribed($shard2)
|
|
|
|
check:
|
|
toSeq(node.wakuRelay.subscribedTopics).len == shards.len
|
|
|
|
await restServer.stop()
|
|
await restServer.closeWait()
|
|
await node.stop()
|
|
|
|
asyncTest "Unsubscribe a node from an array of pubsub topics - DELETE /relay/v1/subscriptions":
|
|
# Given
|
|
let node = testWakuNode()
|
|
await node.start()
|
|
|
|
let
|
|
shard0 = RelayShard(clusterId: DefaultClusterId, shardId: 0)
|
|
shard1 = RelayShard(clusterId: DefaultClusterId, shardId: 1)
|
|
shard2 = RelayShard(clusterId: DefaultClusterId, shardId: 2)
|
|
shard3 = RelayShard(clusterId: DefaultClusterId, shardId: 3)
|
|
shard4 = RelayShard(clusterId: DefaultClusterId, shardId: 4)
|
|
|
|
(await node.mountRelay()).isOkOr:
|
|
assert false, "Failed to mount relay"
|
|
|
|
proc simpleHandler(
|
|
topic: PubsubTopic, msg: WakuMessage
|
|
): Future[void] {.async, gcsafe.} =
|
|
await sleepAsync(0.milliseconds)
|
|
|
|
for shard in @[$shard0, $shard1, $shard2, $shard3, $shard4]:
|
|
node.subscribe((kind: PubsubSub, topic: shard), simpleHandler).isOkOr:
|
|
assert false, "Failed to subscribe to pubsub topic: " & $error
|
|
|
|
var restPort = Port(0)
|
|
let restAddress = parseIpAddress("0.0.0.0")
|
|
let restServer = WakuRestServerRef.init(restAddress, restPort).tryGet()
|
|
|
|
restPort = restServer.httpServer.address.port # update with bound port for client use
|
|
|
|
let cache = MessageCache.init()
|
|
cache.pubsubSubscribe($shard0)
|
|
cache.pubsubSubscribe($shard1)
|
|
cache.pubsubSubscribe($shard2)
|
|
cache.pubsubSubscribe($shard3)
|
|
|
|
installRelayApiHandlers(restServer.router, node, cache)
|
|
restServer.start()
|
|
|
|
let shards = @[$shard0, $shard1, $shard2, $shard4]
|
|
|
|
# When
|
|
let client = newRestHttpClient(initTAddress(restAddress, restPort))
|
|
let response = await client.relayDeleteSubscriptionsV1(shards)
|
|
|
|
# Then
|
|
check:
|
|
response.status == 200
|
|
$response.contentType == $MIMETYPE_TEXT
|
|
response.data == "OK"
|
|
|
|
check:
|
|
not cache.isPubsubSubscribed($shard0)
|
|
not node.wakuRelay.isSubscribed($shard0)
|
|
not cache.isPubsubSubscribed($shard1)
|
|
not node.wakuRelay.isSubscribed($shard1)
|
|
not cache.isPubsubSubscribed($shard2)
|
|
not node.wakuRelay.isSubscribed($shard2)
|
|
cache.isPubsubSubscribed($shard3)
|
|
node.wakuRelay.isSubscribed($shard3)
|
|
not cache.isPubsubSubscribed($shard4)
|
|
not node.wakuRelay.isSubscribed($shard4)
|
|
|
|
await restServer.stop()
|
|
await restServer.closeWait()
|
|
await node.stop()
|
|
|
|
asyncTest "Get the latest messages for a pubsub topic - GET /relay/v1/messages/{topic}":
|
|
# Given
|
|
let node = testWakuNode()
|
|
await node.start()
|
|
(await node.mountRelay()).isOkOr:
|
|
assert false, "Failed to mount relay"
|
|
|
|
var restPort = Port(0)
|
|
let restAddress = parseIpAddress("0.0.0.0")
|
|
let restServer = WakuRestServerRef.init(restAddress, restPort).tryGet()
|
|
|
|
restPort = restServer.httpServer.address.port # update with bound port for client use
|
|
|
|
let pubSubTopic = "/waku/2/rs/0/0"
|
|
|
|
var messages = @[
|
|
fakeWakuMessage(
|
|
contentTopic = "content-topic-x",
|
|
payload = toBytes("TEST-1"),
|
|
meta = toBytes("test-meta"),
|
|
ephemeral = true,
|
|
)
|
|
]
|
|
|
|
# Prevent duplicate messages
|
|
for i in 0 ..< 2:
|
|
var msg = fakeWakuMessage(
|
|
contentTopic = "content-topic-x",
|
|
payload = toBytes("TEST-1"),
|
|
meta = toBytes("test-meta"),
|
|
ephemeral = true,
|
|
)
|
|
|
|
while msg == messages[i]:
|
|
msg = fakeWakuMessage(
|
|
contentTopic = "content-topic-x",
|
|
payload = toBytes("TEST-1"),
|
|
meta = toBytes("test-meta"),
|
|
ephemeral = true,
|
|
)
|
|
|
|
messages.add(msg)
|
|
|
|
let cache = MessageCache.init()
|
|
|
|
cache.pubsubSubscribe(pubSubTopic)
|
|
for msg in messages:
|
|
cache.addMessage(pubSubTopic, msg)
|
|
|
|
installRelayApiHandlers(restServer.router, node, cache)
|
|
restServer.start()
|
|
|
|
# When
|
|
let client = newRestHttpClient(initTAddress(restAddress, restPort))
|
|
let response = await client.relayGetMessagesV1(pubSubTopic)
|
|
|
|
# Then
|
|
check:
|
|
response.status == 200
|
|
$response.contentType == $MIMETYPE_JSON
|
|
response.data.len == 3
|
|
response.data.all do(msg: RelayWakuMessage) -> bool:
|
|
msg.payload == base64.encode("TEST-1") and
|
|
msg.contentTopic.get() == "content-topic-x" and msg.version.get() == 2 and
|
|
msg.timestamp.get() != Timestamp(0) and
|
|
msg.meta.get() == base64.encode("test-meta") and msg.ephemeral.get() == true
|
|
|
|
check:
|
|
cache.isPubsubSubscribed(pubSubTopic)
|
|
cache.getMessages(pubSubTopic).tryGet().len == 0
|
|
|
|
await restServer.stop()
|
|
await restServer.closeWait()
|
|
await node.stop()
|
|
|
|
asyncTest "Post a message to a pubsub topic - POST /relay/v1/messages/{topic}":
|
|
## "Relay API: publish and subscribe/unsubscribe":
|
|
# Given
|
|
let node = testWakuNode()
|
|
(await node.mountRelay()).isOkOr:
|
|
assert false, "Failed to mount relay"
|
|
let wakuRlnConfig = getWakuRlnConfig(manager = manager, index = MembershipIndex(1))
|
|
|
|
await node.mountRlnRelay(wakuRlnConfig)
|
|
await node.start()
|
|
# Registration is mandatory before sending messages with rln-relay
|
|
let manager = cast[OnchainGroupManager](node.wakuRlnRelay.groupManager)
|
|
let idCredentials = generateCredentials()
|
|
|
|
(waitFor manager.register(idCredentials, UserMessageLimit(20))).isOkOr:
|
|
assert false, "Failed to register identity credentials" & getCurrentExceptionMsg()
|
|
|
|
let rootUpdated = waitFor manager.updateRoots()
|
|
info "Updated root for node", rootUpdated
|
|
|
|
let proofRes = waitFor manager.fetchMerkleProofElements()
|
|
if proofRes.isErr():
|
|
assert false, "failed to fetch merkle proof: " & proofRes.error
|
|
manager.merkleProofCache = proofRes.get()
|
|
|
|
# RPC server setup
|
|
var restPort = Port(0)
|
|
let restAddress = parseIpAddress("0.0.0.0")
|
|
let restServer = WakuRestServerRef.init(restAddress, restPort).tryGet()
|
|
|
|
restPort = restServer.httpServer.address.port # update with bound port for client use
|
|
|
|
let cache = MessageCache.init()
|
|
|
|
installRelayApiHandlers(restServer.router, node, cache)
|
|
restServer.start()
|
|
|
|
let client = newRestHttpClient(initTAddress(restAddress, restPort))
|
|
|
|
let simpleHandler = proc(
|
|
topic: PubsubTopic, msg: WakuMessage
|
|
): Future[void] {.async, gcsafe.} =
|
|
await sleepAsync(0.milliseconds)
|
|
|
|
node.subscribe((kind: PubsubSub, topic: DefaultPubsubTopic), simpleHandler).isOkOr:
|
|
assert false, "Failed to subscribe to pubsub topic"
|
|
|
|
require:
|
|
toSeq(node.wakuRelay.subscribedTopics).len == 1
|
|
|
|
# When
|
|
let response = await client.relayPostMessagesV1(
|
|
DefaultPubsubTopic,
|
|
RelayWakuMessage(
|
|
payload: base64.encode("TEST-PAYLOAD"),
|
|
contentTopic: some(DefaultContentTopic),
|
|
timestamp: some(now()),
|
|
),
|
|
)
|
|
|
|
# Then
|
|
check:
|
|
response.status == 200
|
|
$response.contentType == $MIMETYPE_TEXT
|
|
response.data == "OK"
|
|
|
|
await restServer.stop()
|
|
await restServer.closeWait()
|
|
await node.stop()
|
|
|
|
# Autosharding API
|
|
|
|
asyncTest "Subscribe a node to an array of content topics - POST /relay/v1/auto/subscriptions":
|
|
# Given
|
|
let node = testWakuNode()
|
|
await node.start()
|
|
(await node.mountRelay()).isOkOr:
|
|
assert false, "Failed to mount relay"
|
|
require node.mountAutoSharding(1, 8).isOk
|
|
|
|
var restPort = Port(0)
|
|
let restAddress = parseIpAddress("0.0.0.0")
|
|
let restServer = WakuRestServerRef.init(restAddress, restPort).tryGet()
|
|
|
|
restPort = restServer.httpServer.address.port # update with bound port for client use
|
|
|
|
let cache = MessageCache.init()
|
|
|
|
installRelayApiHandlers(restServer.router, node, cache)
|
|
restServer.start()
|
|
|
|
let contentTopics = @[
|
|
ContentTopic("/app-1/2/default-content/proto"),
|
|
ContentTopic("/app-2/2/default-content/proto"),
|
|
ContentTopic("/app-3/2/default-content/proto"),
|
|
]
|
|
|
|
# When
|
|
let client = newRestHttpClient(initTAddress(restAddress, restPort))
|
|
let response = await client.relayPostAutoSubscriptionsV1(contentTopics)
|
|
|
|
# Then
|
|
check:
|
|
response.status == 200
|
|
$response.contentType == $MIMETYPE_TEXT
|
|
response.data == "OK"
|
|
|
|
check:
|
|
cache.isContentSubscribed(contentTopics[0])
|
|
cache.isContentSubscribed(contentTopics[1])
|
|
cache.isContentSubscribed(contentTopics[2])
|
|
|
|
check:
|
|
# Node should be subscribed to all shards
|
|
node.wakuRelay.subscribedTopics ==
|
|
@["/waku/2/rs/1/5", "/waku/2/rs/1/7", "/waku/2/rs/1/2"]
|
|
|
|
await restServer.stop()
|
|
await restServer.closeWait()
|
|
await node.stop()
|
|
|
|
asyncTest "Unsubscribe a node from an array of content topics - DELETE /relay/v1/auto/subscriptions":
|
|
# Given
|
|
let node = testWakuNode()
|
|
await node.start()
|
|
(await node.mountRelay()).isOkOr:
|
|
assert false, "Failed to mount relay"
|
|
require node.mountAutoSharding(1, 8).isOk
|
|
|
|
var restPort = Port(0)
|
|
let restAddress = parseIpAddress("0.0.0.0")
|
|
let restServer = WakuRestServerRef.init(restAddress, restPort).tryGet()
|
|
restServer.start()
|
|
|
|
restPort = restServer.httpServer.address.port # update with bound port for client use
|
|
|
|
let contentTopics = @[
|
|
ContentTopic("/waku/2/default-content1/proto"),
|
|
ContentTopic("/waku/2/default-content2/proto"),
|
|
ContentTopic("/waku/2/default-content3/proto"),
|
|
ContentTopic("/waku/2/default-contentX/proto"),
|
|
]
|
|
|
|
let cache = MessageCache.init()
|
|
cache.contentSubscribe(contentTopics[0])
|
|
cache.contentSubscribe(contentTopics[1])
|
|
cache.contentSubscribe(contentTopics[2])
|
|
cache.contentSubscribe("/waku/2/default-contentY/proto")
|
|
|
|
installRelayApiHandlers(restServer.router, node, cache)
|
|
|
|
# When
|
|
let client = newRestHttpClient(initTAddress(restAddress, restPort))
|
|
|
|
var response = await client.relayPostAutoSubscriptionsV1(contentTopics)
|
|
|
|
check:
|
|
response.status == 200
|
|
$response.contentType == $MIMETYPE_TEXT
|
|
response.data == "OK"
|
|
|
|
response = await client.relayDeleteAutoSubscriptionsV1(contentTopics)
|
|
|
|
# Then
|
|
check:
|
|
response.status == 200
|
|
$response.contentType == $MIMETYPE_TEXT
|
|
response.data == "OK"
|
|
|
|
check:
|
|
not cache.isContentSubscribed(contentTopics[1])
|
|
not cache.isContentSubscribed(contentTopics[2])
|
|
not cache.isContentSubscribed(contentTopics[3])
|
|
cache.isContentSubscribed("/waku/2/default-contentY/proto")
|
|
|
|
await restServer.stop()
|
|
await restServer.closeWait()
|
|
await node.stop()
|
|
|
|
asyncTest "Get the latest messages for a content topic - GET /relay/v1/auto/messages/{topic}":
|
|
# Given
|
|
let node = testWakuNode()
|
|
await node.start()
|
|
(await node.mountRelay()).isOkOr:
|
|
assert false, "Failed to mount relay"
|
|
require node.mountAutoSharding(1, 8).isOk
|
|
|
|
var restPort = Port(0)
|
|
let restAddress = parseIpAddress("0.0.0.0")
|
|
let restServer = WakuRestServerRef.init(restAddress, restPort).tryGet()
|
|
|
|
restPort = restServer.httpServer.address.port # update with bound port for client use
|
|
|
|
let contentTopic = DefaultContentTopic
|
|
|
|
var messages = @[
|
|
fakeWakuMessage(contentTopic = DefaultContentTopic, payload = toBytes("TEST-1"))
|
|
]
|
|
|
|
# Prevent duplicate messages
|
|
for i in 0 ..< 2:
|
|
var msg =
|
|
fakeWakuMessage(contentTopic = DefaultContentTopic, payload = toBytes("TEST-1"))
|
|
|
|
while msg == messages[i]:
|
|
msg = fakeWakuMessage(
|
|
contentTopic = DefaultContentTopic, payload = toBytes("TEST-1")
|
|
)
|
|
|
|
messages.add(msg)
|
|
|
|
let cache = MessageCache.init()
|
|
|
|
cache.contentSubscribe(contentTopic)
|
|
for msg in messages:
|
|
cache.addMessage(DefaultPubsubTopic, msg)
|
|
|
|
installRelayApiHandlers(restServer.router, node, cache)
|
|
restServer.start()
|
|
|
|
# When
|
|
let client = newRestHttpClient(initTAddress(restAddress, restPort))
|
|
let response = await client.relayGetAutoMessagesV1(contentTopic)
|
|
|
|
# Then
|
|
check:
|
|
response.status == 200
|
|
$response.contentType == $MIMETYPE_JSON
|
|
response.data.len == 3
|
|
response.data.all do(msg: RelayWakuMessage) -> bool:
|
|
msg.payload == base64.encode("TEST-1") and
|
|
msg.contentTopic.get() == DefaultContentTopic and msg.version.get() == 2 and
|
|
msg.timestamp.get() != Timestamp(0)
|
|
|
|
check:
|
|
cache.isContentSubscribed(contentTopic)
|
|
cache.getAutoMessages(contentTopic).tryGet().len == 0
|
|
# The cache is cleared when getMessage is called
|
|
|
|
await restServer.stop()
|
|
await restServer.closeWait()
|
|
await node.stop()
|
|
|
|
asyncTest "Post a message to a content topic - POST /relay/v1/auto/messages/{topic}":
|
|
## "Relay API: publish and subscribe/unsubscribe":
|
|
# Given
|
|
var meshNode: WakuNode
|
|
lockNewGlobalBrokerContext:
|
|
meshNode = testWakuNode()
|
|
(await meshNode.mountRelay()).isOkOr:
|
|
assert false, "Failed to mount relay"
|
|
require meshNode.mountAutoSharding(1, 8).isOk
|
|
|
|
let wakuRlnConfig =
|
|
getWakuRlnConfig(manager = manager, index = MembershipIndex(1))
|
|
|
|
await meshNode.mountRlnRelay(wakuRlnConfig)
|
|
await meshNode.start()
|
|
const testPubsubTopic = PubsubTopic("/waku/2/rs/1/0")
|
|
proc dummyHandler(
|
|
topic: PubsubTopic, msg: WakuMessage
|
|
): Future[void] {.async, gcsafe.} =
|
|
discard
|
|
|
|
meshNode.subscribe((kind: ContentSub, topic: DefaultContentTopic), dummyHandler).isOkOr:
|
|
raiseAssert "Failed to subscribe meshNode: " & error
|
|
|
|
var node: WakuNode
|
|
lockNewGlobalBrokerContext:
|
|
node = testWakuNode()
|
|
(await node.mountRelay()).isOkOr:
|
|
assert false, "Failed to mount relay"
|
|
require node.mountAutoSharding(1, 8).isOk
|
|
|
|
let wakuRlnConfig =
|
|
getWakuRlnConfig(manager = manager, index = MembershipIndex(1))
|
|
|
|
await node.mountRlnRelay(wakuRlnConfig)
|
|
await node.start()
|
|
await node.connectToNodes(@[meshNode.peerInfo.toRemotePeerInfo()])
|
|
|
|
# Registration is mandatory before sending messages with rln-relay
|
|
let manager = cast[OnchainGroupManager](node.wakuRlnRelay.groupManager)
|
|
let idCredentials = generateCredentials()
|
|
|
|
(waitFor manager.register(idCredentials, UserMessageLimit(20))).isOkOr:
|
|
assert false, "Failed to register identity credentials" & getCurrentExceptionMsg()
|
|
|
|
let rootUpdated = waitFor manager.updateRoots()
|
|
info "Updated root for node", rootUpdated
|
|
|
|
let proofRes = waitFor manager.fetchMerkleProofElements()
|
|
if proofRes.isErr():
|
|
assert false, "failed to fetch merkle proof: " & proofRes.error
|
|
manager.merkleProofCache = proofRes.get()
|
|
|
|
# RPC server setup
|
|
var restPort = Port(0)
|
|
let restAddress = parseIpAddress("0.0.0.0")
|
|
let restServer = WakuRestServerRef.init(restAddress, restPort).tryGet()
|
|
|
|
restPort = restServer.httpServer.address.port # update with bound port for client use
|
|
|
|
let cache = MessageCache.init()
|
|
installRelayApiHandlers(restServer.router, node, cache)
|
|
restServer.start()
|
|
|
|
let client = newRestHttpClient(initTAddress(restAddress, restPort))
|
|
|
|
let simpleHandler = proc(
|
|
topic: PubsubTopic, msg: WakuMessage
|
|
): Future[void] {.async, gcsafe.} =
|
|
await sleepAsync(0.milliseconds)
|
|
|
|
node.subscribe((kind: ContentSub, topic: DefaultContentTopic), simpleHandler).isOkOr:
|
|
assert false, "Failed to subscribe to content topic: " & $error
|
|
require:
|
|
toSeq(node.wakuRelay.subscribedTopics).len == 1
|
|
|
|
# When
|
|
let response = await client.relayPostAutoMessagesV1(
|
|
RelayWakuMessage(
|
|
payload: base64.encode("TEST-PAYLOAD"),
|
|
contentTopic: some(DefaultContentTopic),
|
|
timestamp: some(now()),
|
|
)
|
|
)
|
|
|
|
# Then
|
|
check:
|
|
response.status == 200
|
|
$response.contentType == $MIMETYPE_TEXT
|
|
response.data == "OK"
|
|
|
|
await restServer.stop()
|
|
await restServer.closeWait()
|
|
await node.stop()
|
|
|
|
asyncTest "Post a message to an invalid content topic - POST /relay/v1/auto/messages/{topic}":
|
|
## "Relay API: publish and subscribe/unsubscribe":
|
|
# Given
|
|
let node = testWakuNode()
|
|
(await node.mountRelay()).isOkOr:
|
|
assert false, "Failed to mount relay"
|
|
require node.mountAutoSharding(1, 8).isOk
|
|
|
|
let wakuRlnConfig = getWakuRlnConfig(manager = manager, index = MembershipIndex(1))
|
|
await node.mountRlnRelay(wakuRlnConfig)
|
|
await node.start()
|
|
|
|
# Registration is mandatory before sending messages with rln-relay
|
|
let manager = cast[OnchainGroupManager](node.wakuRlnRelay.groupManager)
|
|
let idCredentials = generateCredentials()
|
|
|
|
(waitFor manager.register(idCredentials, UserMessageLimit(20))).isOkOr:
|
|
assert false, "Failed to register identity credentials" & getCurrentExceptionMsg()
|
|
|
|
let rootUpdated = waitFor manager.updateRoots()
|
|
info "Updated root for node", rootUpdated
|
|
|
|
let proofRes = waitFor manager.fetchMerkleProofElements()
|
|
if proofRes.isErr():
|
|
assert false, "failed to fetch merkle proof: " & proofRes.error
|
|
manager.merkleProofCache = proofRes.get()
|
|
|
|
# RPC server setup
|
|
var restPort = Port(0)
|
|
let restAddress = parseIpAddress("0.0.0.0")
|
|
let restServer = WakuRestServerRef.init(restAddress, restPort).tryGet()
|
|
|
|
restPort = restServer.httpServer.address.port # update with bound port for client use
|
|
|
|
let cache = MessageCache.init()
|
|
installRelayApiHandlers(restServer.router, node, cache)
|
|
restServer.start()
|
|
|
|
let client = newRestHttpClient(initTAddress(restAddress, restPort))
|
|
|
|
let invalidContentTopic = "invalidContentTopic"
|
|
# When
|
|
let response = await client.relayPostAutoMessagesV1(
|
|
RelayWakuMessage(
|
|
payload: base64.encode("TEST-PAYLOAD"),
|
|
contentTopic: some(invalidContentTopic),
|
|
timestamp: some(int64(2022)),
|
|
)
|
|
)
|
|
|
|
# Then
|
|
check:
|
|
response.status == 400
|
|
$response.contentType == $MIMETYPE_TEXT
|
|
response.data ==
|
|
"Failed to publish. Autosharding error: invalid format: content-topic '" &
|
|
invalidContentTopic & "' must start with slash"
|
|
|
|
await restServer.stop()
|
|
await restServer.closeWait()
|
|
await node.stop()
|
|
|
|
asyncTest "Post a message larger than maximum size - POST /relay/v1/messages/{topic}":
|
|
# Given
|
|
let node = testWakuNode()
|
|
(await node.mountRelay()).isOkOr:
|
|
assert false, "Failed to mount relay"
|
|
let wakuRlnConfig = getWakuRlnConfig(manager = manager, index = MembershipIndex(1))
|
|
await node.mountRlnRelay(wakuRlnConfig)
|
|
await node.start()
|
|
|
|
# Registration is mandatory before sending messages with rln-relay
|
|
let manager = cast[OnchainGroupManager](node.wakuRlnRelay.groupManager)
|
|
let idCredentials = generateCredentials()
|
|
|
|
(waitFor manager.register(idCredentials, UserMessageLimit(20))).isOkOr:
|
|
assert false, "Failed to register identity credentials" & getCurrentExceptionMsg()
|
|
|
|
let rootUpdated = waitFor manager.updateRoots()
|
|
info "Updated root for node", rootUpdated
|
|
|
|
let proofRes = waitFor manager.fetchMerkleProofElements()
|
|
if proofRes.isErr():
|
|
assert false, "failed to fetch merkle proof: " & proofRes.error
|
|
manager.merkleProofCache = proofRes.get()
|
|
|
|
# RPC server setup
|
|
var restPort = Port(0)
|
|
let restAddress = parseIpAddress("0.0.0.0")
|
|
let restServer = WakuRestServerRef.init(restAddress, restPort).tryGet()
|
|
|
|
restPort = restServer.httpServer.address.port # update with bound port for client use
|
|
|
|
let cache = MessageCache.init()
|
|
|
|
installRelayApiHandlers(restServer.router, node, cache)
|
|
restServer.start()
|
|
|
|
let client = newRestHttpClient(initTAddress(restAddress, restPort))
|
|
|
|
let simpleHandler = proc(
|
|
topic: PubsubTopic, msg: WakuMessage
|
|
): Future[void] {.async, gcsafe.} =
|
|
await sleepAsync(0.milliseconds)
|
|
|
|
node.subscribe((kind: PubsubSub, topic: DefaultPubsubTopic), simpleHandler).isOkOr:
|
|
assert false, "Failed to subscribe to pubsub topic: " & $error
|
|
require:
|
|
toSeq(node.wakuRelay.subscribedTopics).len == 1
|
|
|
|
# When
|
|
let response = await client.relayPostMessagesV1(
|
|
DefaultPubsubTopic,
|
|
RelayWakuMessage(
|
|
payload: base64.encode(getByteSequence(DefaultMaxWakuMessageSize)),
|
|
# Message will be bigger than the max size
|
|
contentTopic: some(DefaultContentTopic),
|
|
timestamp: some(int64(2022)),
|
|
),
|
|
)
|
|
|
|
# Then
|
|
check:
|
|
response.status == 400
|
|
$response.contentType == $MIMETYPE_TEXT
|
|
response.data ==
|
|
fmt"Failed to publish: Message size exceeded maximum of {DefaultMaxWakuMessageSize} bytes"
|
|
|
|
await restServer.stop()
|
|
await restServer.closeWait()
|
|
await node.stop()
|
|
|
|
asyncTest "Post a message larger than maximum size - POST /relay/v1/auto/messages/{topic}":
|
|
# Given
|
|
let node = testWakuNode()
|
|
(await node.mountRelay()).isOkOr:
|
|
assert false, "Failed to mount relay"
|
|
require node.mountAutoSharding(1, 8).isOk
|
|
|
|
let wakuRlnConfig = getWakuRlnConfig(manager = manager, index = MembershipIndex(1))
|
|
await node.mountRlnRelay(wakuRlnConfig)
|
|
await node.start()
|
|
|
|
# Registration is mandatory before sending messages with rln-relay
|
|
let manager = cast[OnchainGroupManager](node.wakuRlnRelay.groupManager)
|
|
let idCredentials = generateCredentials()
|
|
|
|
(waitFor manager.register(idCredentials, UserMessageLimit(20))).isOkOr:
|
|
assert false, "Failed to register identity credentials" & getCurrentExceptionMsg()
|
|
|
|
let rootUpdated = waitFor manager.updateRoots()
|
|
info "Updated root for node", rootUpdated
|
|
|
|
let proofRes = waitFor manager.fetchMerkleProofElements()
|
|
if proofRes.isErr():
|
|
assert false, "failed to fetch merkle proof: " & proofRes.error
|
|
manager.merkleProofCache = proofRes.get()
|
|
|
|
# RPC server setup
|
|
var restPort = Port(0)
|
|
let restAddress = parseIpAddress("0.0.0.0")
|
|
let restServer = WakuRestServerRef.init(restAddress, restPort).tryGet()
|
|
|
|
restPort = restServer.httpServer.address.port # update with bound port for client use
|
|
|
|
let cache = MessageCache.init()
|
|
|
|
installRelayApiHandlers(restServer.router, node, cache)
|
|
restServer.start()
|
|
|
|
let client = newRestHttpClient(initTAddress(restAddress, restPort))
|
|
|
|
let simpleHandler = proc(
|
|
topic: PubsubTopic, msg: WakuMessage
|
|
): Future[void] {.async, gcsafe.} =
|
|
await sleepAsync(0.milliseconds)
|
|
|
|
node.subscribe((kind: PubsubSub, topic: DefaultPubsubTopic), simpleHandler).isOkOr:
|
|
assert false, "Failed to subscribe to pubsub topic: " & $error
|
|
require:
|
|
toSeq(node.wakuRelay.subscribedTopics).len == 1
|
|
|
|
# When
|
|
let response = await client.relayPostAutoMessagesV1(
|
|
RelayWakuMessage(
|
|
payload: base64.encode(getByteSequence(DefaultMaxWakuMessageSize)),
|
|
# Message will be bigger than the max size
|
|
contentTopic: some(DefaultContentTopic),
|
|
timestamp: some(int64(2022)),
|
|
)
|
|
)
|
|
|
|
# Then
|
|
check:
|
|
response.status == 400
|
|
$response.contentType == $MIMETYPE_TEXT
|
|
response.data ==
|
|
fmt"Failed to publish: Message size exceeded maximum of {DefaultMaxWakuMessageSize} bytes"
|
|
|
|
await restServer.stop()
|
|
await restServer.closeWait()
|
|
await node.stop()
|