mirror of
https://github.com/logos-messaging/logos-messaging-nim.git
synced 2026-05-18 00:09:52 +00:00
* persistency: per-job SQLite-backed storage layer (singleton, brokered)
Adds a backend-neutral CRUD library at waku/persistency/, plus the
nim-brokers dependency swap that enables it.
Architecture (ports-and-adapters):
* Persistency: process-wide singleton, one root directory.
* Job: one tenant, one DB file, one worker thread, one BrokerContext.
* Backend: SQLite via waku/common/databases/db_sqlite. Uniform schema
kv(category BLOB, key BLOB, payload BLOB) PRIMARY KEY (category, key)
WITHOUT ROWID, WAL mode.
* Writes are fire-and-forget via EventBroker(mt) PersistEvent.
* Reads are async via five RequestBroker(mt) shapes (KvGet, KvExists,
KvScan, KvCount, KvDelete). Reads return Result[T, PersistencyError].
* One storage thread per job; tenants isolated by BrokerContext.
Public surface (waku/persistency/persistency.nim):
Persistency.instance(rootDir) / Persistency.instance() / Persistency.reset()
p.openJob(id) / p.closeJob(id) / p.dropJob(id) / p.close()
p.job(id) / p[id] / p.hasJob(id)
Writes (Job form & string-id form, fire-and-forget):
persist / persistPut / persistDelete / persistEncoded
Reads (Job form & string-id form, async Result):
get / exists / scan / scanPrefix / count / deleteAcked
Key & payload encoding (keys.nim, payload.nim):
* encodePart family + variadic key(...) / payload(...) macros +
single-value toKey / toPayload.
* Primitives: string and openArray[byte] are 2-byte BE length + bytes;
int{8..64} are sign-flipped 8-byte BE; uint{16..64} are 8-byte BE;
bool/byte/char are 1 byte; enums are int64(ord(v)).
* Generic encodePart[T: tuple | object] recurses through fields() so
any composite Nim type is encodable without ceremony.
* Stable across Nim/C compiler upgrades: no sizeof, no memcpy, no
cast on pointers, no host-endianness dependency.
* `rawKey(bytes)` + `persistPut(..., openArray[byte])` let callers
bypass the built-in encoder with their own format (CBOR, protobuf...).
Lifecycle:
* Persistency.new is private; Persistency.instance is the only public
constructor. Same rootDir is idempotent; conflicting rootDir is
peInvalidArgument. Persistency.reset for test/restart paths.
* openJob opens-or-creates the per-job SQLite file; an existing file
is reused with its data preserved.
* Teardown integration: Persistency.instance registers a Teardown
MultiRequestBroker provider that closes all jobs and clears the
singleton slot when Waku.stop() issues Teardown.request.
Internal layering:
types.nim pure value types (Key, KeyRange, KvRow, TxOp,
PersistencyError)
keys.nim encodePart primitives + key(...) macro
payload.nim toPayload + payload(...) macro
schema.nim CREATE TABLE + connection pragmas + user_version
backend_sqlite.nim KvBackend, applyOps (single source of write SQL),
getOne/existsOne/deleteOne, scanRange (asc/desc,
half-open ranges, open-ended stop), countRange
backend_comm.nim EventBroker(mt) PersistEvent + 5 RequestBroker(mt)
declarations; encodeErr/decodeErr boundary helpers
backend_thread.nim startStorageThread / stopStorageThread (shared
allocShared0 arg, cstring dbPath, atomic
ready/shutdown flags); per-thread provider
registration
persistency.nim Persistency + Job types, singleton state, public
facade
../requests/lifecycle_requests.nim
Teardown MultiRequestBroker
Tests (69 cases, all passing):
test_keys.nim sort-order invariants (length-prefix strings,
sign-flipped ints, composite tuples, prefix
range)
test_backend.nim round-trip / replace / delete-return-value /
batched atomicity / asc-desc-half-open-open-
ended scans / category isolation / batch
txDelete
test_lifecycle.nim open-or-create rootDir / non-dir collision /
reopen across sessions / idempotent openJob /
two-tenant parallel isolation / closeJob joins
worker / dropJob removes file / acked delete
test_facade.nim put-then-get / atomic batch / scanPrefix
asc/desc / deleteAcked hit-miss /
fire-and-forget delete / two-tenant facade
isolation
test_encoding.nim tuple/named-tuple/object keys, embedded Key,
enum encoding, field-major composite sort,
payload struct encoding, end-to-end struct
round-trip through SQLite
test_string_lookup.nim peJobNotFound semantics / hasJob / subscript /
persistPut+get via id / reads short-circuit /
writes drop+warn / persistEncoded via id /
scan parity Job-ref vs id
test_singleton.nim idempotent same-rootDir / different-rootDir
rejection / no-arg instance lifecycle / reset
retargets / reset idempotence / Teardown.request
end-to-end
Prerequisite delivered in the same series: replace the in-tree broker
implementation with the external nim-brokers package; update all
broker call-sites (waku_filter_v2, waku_relay, waku_rln_relay,
delivery_service, peer_manager, requests/*, factory/*, api tests, etc.)
to the new package API; chat2 made to compile again.
Note: SDS adapter (Phase 5 of the design) is deferred -- nim-sds is
still developed side-by-side and the persistency layer is intentionally
SDS-agnostic.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* persistency: pin nim-brokers by URL+commit (workaround for stale registry)
The bare `brokers >= 2.0.1` form cannot resolve on machines where the
local nimble SAT solver enumerates only the registry-recorded 0.1.0 for
brokers. The nim-lang/packages entry for `brokers` carries no per-tag
metadata (only the URL), so until that registry entry is refreshed the
SAT solver clamps the available-versions list to 0.1.0 and rejects the
>= 2.0.1 constraint -- even though pkgs2 and pkgcache both have v2.0.1
cloned locally.
Pinning by URL+commit bypasses the registry path entirely. Inline
comment in waku.nimble documents the situation and the path back to
the bare form once nim-lang/packages is updated.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* persistency: nph format pass
Run `nph` on all 57 Nim files touched by this PR. Pure formatting:
17 files re-styled, no semantic change. Suite still 69/69.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Fix build, add local-storage-path config, lazy init of Persistency from Waku start
* fix: fix nix deps
* fixes for nix build, regenerate deps
* reverting accidental dependency changes
* Fixing deps
* Apply suggestions from code review
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* persistency tests: migrate to suite / asyncTest / await
Match the in-tree test convention (procSuite -> suite, sync test +
waitFor -> asyncTest + await):
- procSuite "X": -> suite "X":
- For tests doing async work: test -> asyncTest, waitFor -> await.
- Poll helpers (proc waitFor(t: Job, ...) in test_lifecycle.nim,
proc waitUntilExists(...) in test_facade.nim and
test_string_lookup.nim) -> Future[bool] {.async.}, internal
`waitFor X` -> `await X`, internal `sleep(N)` ->
`await sleepAsync(chronos.milliseconds(N))`.
- Renamed test_lifecycle.nim's helper proc from `waitFor(t: Job, ...)`
-> `pollExists(t: Job, ...)`; the previous name shadowed
chronos.waitFor in the chronos macro expansion.
- `chronos.milliseconds(N)` explicitly qualified because `std/times`
also exports `milliseconds` (returning TimeInterval, not Duration).
- `check await x` -> `let okN = await x; check okN` to dodge chronos's
"yield in expr not lowered" with await-as-macro-argument.
- `(await x).foo()` -> `let awN = await x; ... awN.foo() ...` for the
same reason.
waku/persistency/persistency.nim: nph also pulled the proc signatures
across multiple lines; restored explicit `Future[void] {.async.}`
return types after the colon (an intermediate nph pass had elided them).
Suite: 71 / 71 OK against the new async write surface.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* use idiomatic valueOr instead of ifs
* Reworked persistency shutdown, remove not necessary teardown mechanism
* Use const for DefaultStoragePath
* format to follow coding guidelines - no use of result and explicit returns - no functional change
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
356 lines
10 KiB
Nim
356 lines
10 KiB
Nim
import
|
|
chronicles,
|
|
chronos,
|
|
confutils,
|
|
confutils/defs,
|
|
confutils/std/net,
|
|
eth/keys,
|
|
libp2p/crypto/crypto,
|
|
libp2p/crypto/secp,
|
|
nimcrypto/utils,
|
|
std/strutils,
|
|
regex
|
|
import waku/waku_core
|
|
|
|
type
|
|
Fleet* = enum
|
|
none
|
|
prod
|
|
test
|
|
|
|
EthRpcUrl* = distinct string
|
|
|
|
Chat2Conf* = object ## General node config
|
|
logLevel* {.
|
|
desc: "Sets the log level.", defaultValue: LogLevel.INFO, name: "log-level"
|
|
.}: LogLevel
|
|
|
|
nodekey* {.desc: "P2P node private key as 64 char hex string.", name: "nodekey".}:
|
|
Option[crypto.PrivateKey]
|
|
|
|
listenAddress* {.
|
|
defaultValue: defaultListenAddress(config),
|
|
desc: "Listening address for the LibP2P traffic.",
|
|
name: "listen-address"
|
|
.}: IpAddress
|
|
|
|
tcpPort* {.desc: "TCP listening port.", defaultValue: 60000, name: "tcp-port".}:
|
|
Port
|
|
|
|
udpPort* {.desc: "UDP listening port.", defaultValue: 60000, name: "udp-port".}:
|
|
Port
|
|
|
|
portsShift* {.
|
|
desc: "Add a shift to all port numbers.", defaultValue: 0, name: "ports-shift"
|
|
.}: uint16
|
|
|
|
nat* {.
|
|
desc:
|
|
"Specify method to use for determining public address. " &
|
|
"Must be one of: any, none, upnp, pmp, extip:<IP>.",
|
|
defaultValue: "any"
|
|
.}: string
|
|
|
|
## Persistence config
|
|
dbPath* {.
|
|
desc: "The database path for peristent storage", defaultValue: "", name: "db-path"
|
|
.}: string
|
|
|
|
persistPeers* {.
|
|
desc: "Enable peer persistence: true|false",
|
|
defaultValue: false,
|
|
name: "persist-peers"
|
|
.}: bool
|
|
|
|
persistMessages* {.
|
|
desc: "Enable message persistence: true|false",
|
|
defaultValue: false,
|
|
name: "persist-messages"
|
|
.}: bool
|
|
|
|
## Relay config
|
|
relay* {.
|
|
desc: "Enable relay protocol: true|false", defaultValue: true, name: "relay"
|
|
.}: bool
|
|
|
|
staticnodes* {.
|
|
desc: "Peer multiaddr to directly connect with. Argument may be repeated.",
|
|
name: "staticnode"
|
|
.}: seq[string]
|
|
|
|
keepAlive* {.
|
|
desc: "Enable keep-alive for idle connections: true|false",
|
|
defaultValue: false,
|
|
name: "keep-alive"
|
|
.}: bool
|
|
|
|
clusterId* {.
|
|
desc:
|
|
"Cluster id that the node is running in. Node in a different cluster id is disconnected.",
|
|
defaultValue: 0,
|
|
name: "cluster-id"
|
|
.}: uint16
|
|
|
|
shards* {.
|
|
desc:
|
|
"Shards index to subscribe to [0..NUM_SHARDS_IN_NETWORK-1]. Argument may be repeated.",
|
|
defaultValue: @[uint16(0)],
|
|
name: "shard"
|
|
.}: seq[uint16]
|
|
|
|
## Store config
|
|
store* {.
|
|
desc: "Enable store protocol: true|false", defaultValue: true, name: "store"
|
|
.}: bool
|
|
|
|
storenode* {.
|
|
desc: "Peer multiaddr to query for storage.", defaultValue: "", name: "storenode"
|
|
.}: string
|
|
|
|
## Filter config
|
|
filter* {.
|
|
desc: "Enable filter protocol: true|false", defaultValue: false, name: "filter"
|
|
.}: bool
|
|
|
|
filternode* {.
|
|
desc: "Peer multiaddr to request content filtering of messages.",
|
|
defaultValue: "",
|
|
name: "filternode"
|
|
.}: string
|
|
|
|
## Lightpush config
|
|
lightpush* {.
|
|
desc: "Enable lightpush protocol: true|false",
|
|
defaultValue: false,
|
|
name: "lightpush"
|
|
.}: bool
|
|
|
|
lightpushnode* {.
|
|
desc: "Peer multiaddr to request lightpush of published messages.",
|
|
defaultValue: "",
|
|
name: "lightpushnode"
|
|
.}: string
|
|
|
|
## Metrics config
|
|
metricsServer* {.
|
|
desc: "Enable the metrics server: true|false",
|
|
defaultValue: false,
|
|
name: "metrics-server"
|
|
.}: bool
|
|
|
|
metricsServerAddress* {.
|
|
desc: "Listening address of the metrics server.",
|
|
defaultValue:
|
|
IpAddress(family: IpAddressFamily.IPv4, address_v4: [127'u8, 0, 0, 1]),
|
|
name: "metrics-server-address"
|
|
.}: IpAddress
|
|
|
|
metricsServerPort* {.
|
|
desc: "Listening HTTP port of the metrics server.",
|
|
defaultValue: 8008,
|
|
name: "metrics-server-port"
|
|
.}: uint16
|
|
|
|
metricsLogging* {.
|
|
desc: "Enable metrics logging: true|false",
|
|
defaultValue: true,
|
|
name: "metrics-logging"
|
|
.}: bool
|
|
|
|
## DNS discovery config
|
|
dnsDiscovery* {.
|
|
desc:
|
|
"Deprecated, please set dns-discovery-url instead. Enable discovering nodes via DNS",
|
|
defaultValue: false,
|
|
name: "dns-discovery"
|
|
.}: bool
|
|
|
|
dnsDiscoveryUrl* {.
|
|
desc: "URL for DNS node list in format 'enrtree://<key>@<fqdn>'",
|
|
defaultValue: "",
|
|
name: "dns-discovery-url"
|
|
.}: string
|
|
|
|
dnsAddrsNameServers* {.
|
|
desc:
|
|
"DNS name server IPs to query for DNS multiaddrs resolution. Argument may be repeated.",
|
|
defaultValue: @[
|
|
IpAddress(family: IpAddressFamily.IPv4, address_v4: [1'u8, 1, 1, 1]),
|
|
IpAddress(family: IpAddressFamily.IPv4, address_v4: [1'u8, 0, 0, 1]),
|
|
],
|
|
name: "dns-addrs-name-server"
|
|
.}: seq[IpAddress]
|
|
|
|
## Chat2 configuration
|
|
fleet* {.
|
|
desc:
|
|
"Select the fleet to connect to. This sets the DNS discovery URL to the selected fleet.",
|
|
defaultValue: Fleet.prod,
|
|
name: "fleet"
|
|
.}: Fleet
|
|
|
|
contentTopic* {.
|
|
desc: "Content topic for chat messages.",
|
|
defaultValue: "/toy-chat/2/huilong/proto",
|
|
name: "content-topic"
|
|
.}: string
|
|
|
|
## Websocket Configuration
|
|
websocketSupport* {.
|
|
desc: "Enable websocket: true|false",
|
|
defaultValue: false,
|
|
name: "websocket-support"
|
|
.}: bool
|
|
|
|
websocketPort* {.
|
|
desc: "WebSocket listening port.", defaultValue: 8000, name: "websocket-port"
|
|
.}: Port
|
|
|
|
websocketSecureSupport* {.
|
|
desc: "WebSocket Secure Support.",
|
|
defaultValue: false,
|
|
name: "websocket-secure-support"
|
|
.}: bool
|
|
|
|
## rln-relay configuration
|
|
rlnRelay* {.
|
|
desc: "Enable spam protection through rln-relay: true|false",
|
|
defaultValue: false,
|
|
name: "rln-relay"
|
|
.}: bool
|
|
|
|
rlnRelayChainId* {.
|
|
desc:
|
|
"Chain ID of the provided contract (optional, will fetch from RPC provider if not used)",
|
|
defaultValue: 0,
|
|
name: "rln-relay-chain-id"
|
|
.}: uint
|
|
|
|
rlnRelayCredPath* {.
|
|
desc: "The path for peristing rln-relay credential",
|
|
defaultValue: "",
|
|
name: "rln-relay-cred-path"
|
|
.}: string
|
|
|
|
rlnRelayCredIndex* {.
|
|
desc: "the index of the onchain commitment to use", name: "rln-relay-cred-index"
|
|
.}: Option[uint]
|
|
|
|
rlnRelayDynamic* {.
|
|
desc: "Enable waku-rln-relay with on-chain dynamic group management: true|false",
|
|
defaultValue: false,
|
|
name: "rln-relay-dynamic"
|
|
.}: bool
|
|
|
|
rlnRelayIdKey* {.
|
|
desc: "Rln relay identity secret key as a Hex string",
|
|
defaultValue: "",
|
|
name: "rln-relay-id-key"
|
|
.}: string
|
|
|
|
rlnRelayIdCommitmentKey* {.
|
|
desc: "Rln relay identity commitment key as a Hex string",
|
|
defaultValue: "",
|
|
name: "rln-relay-id-commitment-key"
|
|
.}: string
|
|
|
|
ethClientUrls* {.
|
|
desc:
|
|
"HTTP address of an Ethereum testnet client e.g., http://localhost:8540/. Argument may be repeated.",
|
|
defaultValue: newSeq[EthRpcUrl](0),
|
|
name: "rln-relay-eth-client-address"
|
|
.}: seq[EthRpcUrl]
|
|
|
|
rlnRelayEthContractAddress* {.
|
|
desc: "Address of membership contract on an Ethereum testnet",
|
|
defaultValue: "",
|
|
name: "rln-relay-eth-contract-address"
|
|
.}: string
|
|
|
|
rlnRelayCredPassword* {.
|
|
desc: "Password for encrypting RLN credentials",
|
|
defaultValue: "",
|
|
name: "rln-relay-cred-password"
|
|
.}: string
|
|
|
|
rlnRelayUserMessageLimit* {.
|
|
desc:
|
|
"Set a user message limit for the rln membership registration. Must be a positive integer. Default is 1.",
|
|
defaultValue: 1,
|
|
name: "rln-relay-user-message-limit"
|
|
.}: uint64
|
|
|
|
rlnEpochSizeSec* {.
|
|
desc:
|
|
"Epoch size in seconds used to rate limit RLN memberships. Default is 1 second.",
|
|
defaultValue: 1,
|
|
name: "rln-relay-epoch-sec"
|
|
.}: uint64
|
|
|
|
# NOTE: Keys are different in nim-libp2p
|
|
proc parseCmdArg*(T: type crypto.PrivateKey, p: string): T =
|
|
try:
|
|
let key = SkPrivateKey.init(utils.fromHex(p)).tryGet()
|
|
# XXX: Here at the moment
|
|
result = crypto.PrivateKey(scheme: Secp256k1, skkey: key)
|
|
except CatchableError as e:
|
|
raise newException(ValueError, "Invalid private key")
|
|
|
|
proc completeCmdArg*(T: type crypto.PrivateKey, val: string): seq[string] =
|
|
return @[]
|
|
|
|
proc parseCmdArg*(T: type IpAddress, p: string): T =
|
|
try:
|
|
result = parseIpAddress(p)
|
|
except CatchableError as e:
|
|
raise newException(ValueError, "Invalid IP address")
|
|
|
|
proc completeCmdArg*(T: type IpAddress, val: string): seq[string] =
|
|
return @[]
|
|
|
|
proc parseCmdArg*(T: type Port, p: string): T =
|
|
try:
|
|
result = Port(parseInt(p))
|
|
except CatchableError as e:
|
|
raise newException(ValueError, "Invalid Port number")
|
|
|
|
proc completeCmdArg*(T: type Port, val: string): seq[string] =
|
|
return @[]
|
|
|
|
proc parseCmdArg*(T: type Option[uint], p: string): T =
|
|
try:
|
|
some(parseUint(p))
|
|
except CatchableError:
|
|
raise newException(ValueError, "Invalid unsigned integer")
|
|
|
|
proc completeCmdArg*(T: type EthRpcUrl, val: string): seq[string] =
|
|
return @[]
|
|
|
|
proc parseCmdArg*(T: type EthRpcUrl, s: string): T =
|
|
## allowed patterns:
|
|
## http://url:port
|
|
## https://url:port
|
|
## http://url:port/path
|
|
## https://url:port/path
|
|
## http://url/with/path
|
|
## http://url:port/path?query
|
|
## https://url:port/path?query
|
|
## disallowed patterns:
|
|
## any valid/invalid ws or wss url
|
|
var httpPattern =
|
|
re2"^(https?):\/\/((localhost)|([\w_-]+(?:(?:\.[\w_-]+)+)))(:[0-9]{1,5})?([\w.,@?^=%&:\/~+#-]*[\w@?^=%&\/~+#-])*"
|
|
var wsPattern =
|
|
re2"^(wss?):\/\/((localhost)|([\w_-]+(?:(?:\.[\w_-]+)+)))(:[0-9]{1,5})?([\w.,@?^=%&:\/~+#-]*[\w@?^=%&\/~+#-])*"
|
|
if regex.match(s, wsPattern):
|
|
raise newException(
|
|
ValueError, "Websocket RPC URL is not supported, Please use an HTTP URL"
|
|
)
|
|
if not regex.match(s, httpPattern):
|
|
raise newException(ValueError, "Invalid HTTP RPC URL")
|
|
return EthRpcUrl(s)
|
|
|
|
func defaultListenAddress*(conf: Chat2Conf): IpAddress =
|
|
# TODO: How should we select between IPv4 and IPv6
|
|
# Maybe there should be a config option for this.
|
|
(static IpAddress(family: IpAddressFamily.IPv4, address_v4: [0'u8, 0, 0, 0]))
|