logos-delivery/waku/persistency/backend_comm.nim
NagyZoltanPeter 42e0aa43d1
feat: persistency (#3880)
* persistency: per-job SQLite-backed storage layer (singleton, brokered)

Adds a backend-neutral CRUD library at waku/persistency/, plus the
nim-brokers dependency swap that enables it.

Architecture (ports-and-adapters):
  * Persistency: process-wide singleton, one root directory.
  * Job: one tenant, one DB file, one worker thread, one BrokerContext.
  * Backend: SQLite via waku/common/databases/db_sqlite. Uniform schema
    kv(category BLOB, key BLOB, payload BLOB) PRIMARY KEY (category, key)
    WITHOUT ROWID, WAL mode.
  * Writes are fire-and-forget via EventBroker(mt) PersistEvent.
  * Reads are async via five RequestBroker(mt) shapes (KvGet, KvExists,
    KvScan, KvCount, KvDelete). Reads return Result[T, PersistencyError].
  * One storage thread per job; tenants isolated by BrokerContext.

Public surface (waku/persistency/persistency.nim):
  Persistency.instance(rootDir) / Persistency.instance() / Persistency.reset()
  p.openJob(id) / p.closeJob(id) / p.dropJob(id) / p.close()
  p.job(id) / p[id] / p.hasJob(id)
  Writes (Job form & string-id form, fire-and-forget):
    persist / persistPut / persistDelete / persistEncoded
  Reads (Job form & string-id form, async Result):
    get / exists / scan / scanPrefix / count / deleteAcked

Key & payload encoding (keys.nim, payload.nim):
  * encodePart family + variadic key(...) / payload(...) macros +
    single-value toKey / toPayload.
  * Primitives: string and openArray[byte] are 2-byte BE length + bytes;
    int{8..64} are sign-flipped 8-byte BE; uint{16..64} are 8-byte BE;
    bool/byte/char are 1 byte; enums are int64(ord(v)).
  * Generic encodePart[T: tuple | object] recurses through fields() so
    any composite Nim type is encodable without ceremony.
  * Stable across Nim/C compiler upgrades: no sizeof, no memcpy, no
    cast on pointers, no host-endianness dependency.
  * `rawKey(bytes)` + `persistPut(..., openArray[byte])` let callers
    bypass the built-in encoder with their own format (CBOR, protobuf...).

Lifecycle:
  * Persistency.new is private; Persistency.instance is the only public
    constructor. Same rootDir is idempotent; conflicting rootDir is
    peInvalidArgument. Persistency.reset for test/restart paths.
  * openJob opens-or-creates the per-job SQLite file; an existing file
    is reused with its data preserved.
  * Teardown integration: Persistency.instance registers a Teardown
    MultiRequestBroker provider that closes all jobs and clears the
    singleton slot when Waku.stop() issues Teardown.request.

Internal layering:
  types.nim          pure value types (Key, KeyRange, KvRow, TxOp,
                     PersistencyError)
  keys.nim           encodePart primitives + key(...) macro
  payload.nim        toPayload + payload(...) macro
  schema.nim         CREATE TABLE + connection pragmas + user_version
  backend_sqlite.nim KvBackend, applyOps (single source of write SQL),
                     getOne/existsOne/deleteOne, scanRange (asc/desc,
                     half-open ranges, open-ended stop), countRange
  backend_comm.nim   EventBroker(mt) PersistEvent + 5 RequestBroker(mt)
                     declarations; encodeErr/decodeErr boundary helpers
  backend_thread.nim startStorageThread / stopStorageThread (shared
                     allocShared0 arg, cstring dbPath, atomic
                     ready/shutdown flags); per-thread provider
                     registration
  persistency.nim    Persistency + Job types, singleton state, public
                     facade
  ../requests/lifecycle_requests.nim
                     Teardown MultiRequestBroker

Tests (69 cases, all passing):
  test_keys.nim          sort-order invariants (length-prefix strings,
                         sign-flipped ints, composite tuples, prefix
                         range)
  test_backend.nim       round-trip / replace / delete-return-value /
                         batched atomicity / asc-desc-half-open-open-
                         ended scans / category isolation / batch
                         txDelete
  test_lifecycle.nim     open-or-create rootDir / non-dir collision /
                         reopen across sessions / idempotent openJob /
                         two-tenant parallel isolation / closeJob joins
                         worker / dropJob removes file / acked delete
  test_facade.nim        put-then-get / atomic batch / scanPrefix
                         asc/desc / deleteAcked hit-miss /
                         fire-and-forget delete / two-tenant facade
                         isolation
  test_encoding.nim      tuple/named-tuple/object keys, embedded Key,
                         enum encoding, field-major composite sort,
                         payload struct encoding, end-to-end struct
                         round-trip through SQLite
  test_string_lookup.nim peJobNotFound semantics / hasJob / subscript /
                         persistPut+get via id / reads short-circuit /
                         writes drop+warn / persistEncoded via id /
                         scan parity Job-ref vs id
  test_singleton.nim     idempotent same-rootDir / different-rootDir
                         rejection / no-arg instance lifecycle / reset
                         retargets / reset idempotence / Teardown.request
                         end-to-end

Prerequisite delivered in the same series: replace the in-tree broker
implementation with the external nim-brokers package; update all
broker call-sites (waku_filter_v2, waku_relay, waku_rln_relay,
delivery_service, peer_manager, requests/*, factory/*, api tests, etc.)
to the new package API; chat2 made to compile again.

Note: SDS adapter (Phase 5 of the design) is deferred -- nim-sds is
still developed side-by-side and the persistency layer is intentionally
SDS-agnostic.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* persistency: pin nim-brokers by URL+commit (workaround for stale registry)

The bare `brokers >= 2.0.1` form cannot resolve on machines where the
local nimble SAT solver enumerates only the registry-recorded 0.1.0 for
brokers. The nim-lang/packages entry for `brokers` carries no per-tag
metadata (only the URL), so until that registry entry is refreshed the
SAT solver clamps the available-versions list to 0.1.0 and rejects the
>= 2.0.1 constraint -- even though pkgs2 and pkgcache both have v2.0.1
cloned locally.

Pinning by URL+commit bypasses the registry path entirely. Inline
comment in waku.nimble documents the situation and the path back to
the bare form once nim-lang/packages is updated.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* persistency: nph format pass

Run `nph` on all 57 Nim files touched by this PR. Pure formatting:
17 files re-styled, no semantic change. Suite still 69/69.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* Fix build, add local-storage-path config, lazy init of Persistency from Waku start

* fix: fix nix deps

* fixes for nix build, regenerate deps

* reverting accidental dependency changes

* Fixing deps

* Apply suggestions from code review

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>

* persistency tests: migrate to suite / asyncTest / await

Match the in-tree test convention (procSuite -> suite, sync test +
waitFor -> asyncTest + await):

- procSuite "X": -> suite "X":
- For tests doing async work: test -> asyncTest, waitFor -> await.
- Poll helpers (proc waitFor(t: Job, ...) in test_lifecycle.nim,
  proc waitUntilExists(...) in test_facade.nim and
  test_string_lookup.nim) -> Future[bool] {.async.}, internal
  `waitFor X` -> `await X`, internal `sleep(N)` ->
  `await sleepAsync(chronos.milliseconds(N))`.
- Renamed test_lifecycle.nim's helper proc from `waitFor(t: Job, ...)`
  -> `pollExists(t: Job, ...)`; the previous name shadowed
  chronos.waitFor in the chronos macro expansion.
- `chronos.milliseconds(N)` explicitly qualified because `std/times`
  also exports `milliseconds` (returning TimeInterval, not Duration).
- `check await x` -> `let okN = await x; check okN` to dodge chronos's
  "yield in expr not lowered" with await-as-macro-argument.
- `(await x).foo()` -> `let awN = await x; ... awN.foo() ...` for the
  same reason.

waku/persistency/persistency.nim: nph also pulled the proc signatures
across multiple lines; restored explicit `Future[void] {.async.}`
return types after the colon (an intermediate nph pass had elided them).

Suite: 71 / 71 OK against the new async write surface.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* use idiomatic valueOr instead of ifs

* Reworked persistency shutdown, remove not necessary teardown mechanism

* Use const for DefaultStoragePath

* format to follow coding guidelines - no use of result and explicit returns - no functional change

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2026-05-16 00:09:07 +02:00

162 lines
4.9 KiB
Nim

## Cross-thread broker declarations for the persistency library.
##
## One EventBroker (writes, fire-and-forget) and five RequestBrokers (reads
## + acked delete). All in multi-thread (mt) mode: the listener / provider runs on the
## job's storage thread; callers on any thread reach it via the shared
## BrokerContext owned by the Job.
##
## ## Error type, important
##
## nim-brokers' RequestBroker macro hard-codes the response shape as
## `Future[Result[ResponseType, string]]` — the error channel is `string`,
## not our `PersistencyError`. We honour the broker contract here and lift
## back to `PersistencyError` at the public facade (persistency.nim). The
## convention for the broker-level string is `"<kind>: <msg>"` so the
## facade can reconstruct the `PersistencyErrorKind`.
##
## ## Response shapes
##
## The five Kv* types are *response* objects (the value the provider
## returns). Per-request inputs sit on the `signature` proc parameters.
{.push raises: [].}
import std/[options, strutils]
import chronos, results
import brokers/[event_broker, request_broker, broker_context]
import brokers/internal/mt_codec
import ./types
export broker_context
# ── mt codec overloads for non-POD library types ────────────────────────
#
# brokers 2.0.0's mtMarshalValue / mtUnmarshalValue handle scalars, enums,
# strings, seqs, arrays, and plain object/tuple recursion -- but they do
# not see through `distinct seq[byte]`, nor do they know how to dispatch
# a variant (case) object. We provide explicit overloads for the types
# that appear in our broker payloads.
proc mtMarshalValue*(
buf: ptr UncheckedArray[byte], cap: int, value: Key, pos: var int
): bool {.gcsafe.} =
## Encode a Key as the raw seq[byte] it wraps.
mtMarshalValue(buf, cap, bytes(value), pos)
proc mtUnmarshalValue*(
buf: ptr UncheckedArray[byte], len: int, value: var Key, pos: var int
): bool {.gcsafe.} =
var s: seq[byte]
if not mtUnmarshalValue(buf, len, s, pos):
return false
value = Key(s)
return true
proc mtMarshalValue*(
buf: ptr UncheckedArray[byte], cap: int, value: TxOp, pos: var int
): bool {.gcsafe.} =
## TxOp is a case object: write the discriminator, then only the
## fields that belong to the active branch.
if not mtMarshalValue(buf, cap, value.category, pos):
return false
if not mtMarshalValue(buf, cap, value.key, pos):
return false
let kind = uint8(ord(value.kind))
if not mtMarshalValue(buf, cap, kind, pos):
return false
case value.kind
of txPut:
if not mtMarshalValue(buf, cap, value.payload, pos):
return false
of txDelete:
discard
return true
proc mtUnmarshalValue*(
buf: ptr UncheckedArray[byte], len: int, value: var TxOp, pos: var int
): bool {.gcsafe.} =
var
category: string
key: Key
kindByte: uint8
if not mtUnmarshalValue(buf, len, category, pos):
return false
if not mtUnmarshalValue(buf, len, key, pos):
return false
if not mtUnmarshalValue(buf, len, kindByte, pos):
return false
case TxOpKind(kindByte)
of txPut:
var payload: seq[byte]
if not mtUnmarshalValue(buf, len, payload, pos):
return false
value = TxOp(category: category, key: key, kind: txPut, payload: payload)
of txDelete:
value = TxOp(category: category, key: key, kind: txDelete)
return true
EventBroker(mt):
type PersistEvent* = object
ops*: seq[TxOp]
RequestBroker(mt):
type KvGet* = object
value*: Option[seq[byte]]
proc signature*(category: string, key: Key): Future[Result[KvGet, string]] {.async.}
RequestBroker(mt):
type KvExists* = object
value*: bool
proc signature*(
category: string, key: Key
): Future[Result[KvExists, string]] {.async.}
RequestBroker(mt):
type KvScan* = object
rows*: seq[KvRow]
proc signature*(
category: string, range: KeyRange, reverse: bool
): Future[Result[KvScan, string]] {.async.}
RequestBroker(mt):
type KvCount* = object
n*: int
proc signature*(
category: string, range: KeyRange
): Future[Result[KvCount, string]] {.async.}
RequestBroker(mt):
type KvDelete* = object
existed*: bool
proc signature*(
category: string, key: Key
): Future[Result[KvDelete, string]] {.async.}
# ── string<->PersistencyError boundary helpers ──────────────────────────
const ErrSep = ": "
proc encodeErr*(e: PersistencyError): string =
## Encode a PersistencyError into the broker's string channel. The facade
## decodes via `decodeErr`.
$e.kind & ErrSep & e.msg
proc decodeErr*(s: string): PersistencyError =
## Inverse of encodeErr. Falls back to peBackend if the prefix is missing.
let idx = s.find(ErrSep)
if idx < 0:
return persistencyErr(peBackend, s)
let head = s[0 ..< idx]
let tail = s[idx + ErrSep.len .. ^1]
for k in PersistencyErrorKind:
if $k == head:
return persistencyErr(k, tail)
persistencyErr(peBackend, s)
{.pop.}