NagyZoltanPeter 42e0aa43d1
feat: persistency (#3880)
* persistency: per-job SQLite-backed storage layer (singleton, brokered)

Adds a backend-neutral CRUD library at waku/persistency/, plus the
nim-brokers dependency swap that enables it.

Architecture (ports-and-adapters):
  * Persistency: process-wide singleton, one root directory.
  * Job: one tenant, one DB file, one worker thread, one BrokerContext.
  * Backend: SQLite via waku/common/databases/db_sqlite. Uniform schema
    kv(category BLOB, key BLOB, payload BLOB) PRIMARY KEY (category, key)
    WITHOUT ROWID, WAL mode.
  * Writes are fire-and-forget via EventBroker(mt) PersistEvent.
  * Reads are async via five RequestBroker(mt) shapes (KvGet, KvExists,
    KvScan, KvCount, KvDelete). Reads return Result[T, PersistencyError].
  * One storage thread per job; tenants isolated by BrokerContext.

Public surface (waku/persistency/persistency.nim):
  Persistency.instance(rootDir) / Persistency.instance() / Persistency.reset()
  p.openJob(id) / p.closeJob(id) / p.dropJob(id) / p.close()
  p.job(id) / p[id] / p.hasJob(id)
  Writes (Job form & string-id form, fire-and-forget):
    persist / persistPut / persistDelete / persistEncoded
  Reads (Job form & string-id form, async Result):
    get / exists / scan / scanPrefix / count / deleteAcked

Key & payload encoding (keys.nim, payload.nim):
  * encodePart family + variadic key(...) / payload(...) macros +
    single-value toKey / toPayload.
  * Primitives: string and openArray[byte] are 2-byte BE length + bytes;
    int{8..64} are sign-flipped 8-byte BE; uint{16..64} are 8-byte BE;
    bool/byte/char are 1 byte; enums are int64(ord(v)).
  * Generic encodePart[T: tuple | object] recurses through fields() so
    any composite Nim type is encodable without ceremony.
  * Stable across Nim/C compiler upgrades: no sizeof, no memcpy, no
    cast on pointers, no host-endianness dependency.
  * `rawKey(bytes)` + `persistPut(..., openArray[byte])` let callers
    bypass the built-in encoder with their own format (CBOR, protobuf...).

Lifecycle:
  * Persistency.new is private; Persistency.instance is the only public
    constructor. Same rootDir is idempotent; conflicting rootDir is
    peInvalidArgument. Persistency.reset for test/restart paths.
  * openJob opens-or-creates the per-job SQLite file; an existing file
    is reused with its data preserved.
  * Teardown integration: Persistency.instance registers a Teardown
    MultiRequestBroker provider that closes all jobs and clears the
    singleton slot when Waku.stop() issues Teardown.request.

Internal layering:
  types.nim          pure value types (Key, KeyRange, KvRow, TxOp,
                     PersistencyError)
  keys.nim           encodePart primitives + key(...) macro
  payload.nim        toPayload + payload(...) macro
  schema.nim         CREATE TABLE + connection pragmas + user_version
  backend_sqlite.nim KvBackend, applyOps (single source of write SQL),
                     getOne/existsOne/deleteOne, scanRange (asc/desc,
                     half-open ranges, open-ended stop), countRange
  backend_comm.nim   EventBroker(mt) PersistEvent + 5 RequestBroker(mt)
                     declarations; encodeErr/decodeErr boundary helpers
  backend_thread.nim startStorageThread / stopStorageThread (shared
                     allocShared0 arg, cstring dbPath, atomic
                     ready/shutdown flags); per-thread provider
                     registration
  persistency.nim    Persistency + Job types, singleton state, public
                     facade
  ../requests/lifecycle_requests.nim
                     Teardown MultiRequestBroker

Tests (69 cases, all passing):
  test_keys.nim          sort-order invariants (length-prefix strings,
                         sign-flipped ints, composite tuples, prefix
                         range)
  test_backend.nim       round-trip / replace / delete-return-value /
                         batched atomicity / asc-desc-half-open-open-
                         ended scans / category isolation / batch
                         txDelete
  test_lifecycle.nim     open-or-create rootDir / non-dir collision /
                         reopen across sessions / idempotent openJob /
                         two-tenant parallel isolation / closeJob joins
                         worker / dropJob removes file / acked delete
  test_facade.nim        put-then-get / atomic batch / scanPrefix
                         asc/desc / deleteAcked hit-miss /
                         fire-and-forget delete / two-tenant facade
                         isolation
  test_encoding.nim      tuple/named-tuple/object keys, embedded Key,
                         enum encoding, field-major composite sort,
                         payload struct encoding, end-to-end struct
                         round-trip through SQLite
  test_string_lookup.nim peJobNotFound semantics / hasJob / subscript /
                         persistPut+get via id / reads short-circuit /
                         writes drop+warn / persistEncoded via id /
                         scan parity Job-ref vs id
  test_singleton.nim     idempotent same-rootDir / different-rootDir
                         rejection / no-arg instance lifecycle / reset
                         retargets / reset idempotence / Teardown.request
                         end-to-end

Prerequisite delivered in the same series: replace the in-tree broker
implementation with the external nim-brokers package; update all
broker call-sites (waku_filter_v2, waku_relay, waku_rln_relay,
delivery_service, peer_manager, requests/*, factory/*, api tests, etc.)
to the new package API; chat2 made to compile again.

Note: SDS adapter (Phase 5 of the design) is deferred -- nim-sds is
still developed side-by-side and the persistency layer is intentionally
SDS-agnostic.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* persistency: pin nim-brokers by URL+commit (workaround for stale registry)

The bare `brokers >= 2.0.1` form cannot resolve on machines where the
local nimble SAT solver enumerates only the registry-recorded 0.1.0 for
brokers. The nim-lang/packages entry for `brokers` carries no per-tag
metadata (only the URL), so until that registry entry is refreshed the
SAT solver clamps the available-versions list to 0.1.0 and rejects the
>= 2.0.1 constraint -- even though pkgs2 and pkgcache both have v2.0.1
cloned locally.

Pinning by URL+commit bypasses the registry path entirely. Inline
comment in waku.nimble documents the situation and the path back to
the bare form once nim-lang/packages is updated.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* persistency: nph format pass

Run `nph` on all 57 Nim files touched by this PR. Pure formatting:
17 files re-styled, no semantic change. Suite still 69/69.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* Fix build, add local-storage-path config, lazy init of Persistency from Waku start

* fix: fix nix deps

* fixes for nix build, regenerate deps

* reverting accidental dependency changes

* Fixing deps

* Apply suggestions from code review

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>

* persistency tests: migrate to suite / asyncTest / await

Match the in-tree test convention (procSuite -> suite, sync test +
waitFor -> asyncTest + await):

- procSuite "X": -> suite "X":
- For tests doing async work: test -> asyncTest, waitFor -> await.
- Poll helpers (proc waitFor(t: Job, ...) in test_lifecycle.nim,
  proc waitUntilExists(...) in test_facade.nim and
  test_string_lookup.nim) -> Future[bool] {.async.}, internal
  `waitFor X` -> `await X`, internal `sleep(N)` ->
  `await sleepAsync(chronos.milliseconds(N))`.
- Renamed test_lifecycle.nim's helper proc from `waitFor(t: Job, ...)`
  -> `pollExists(t: Job, ...)`; the previous name shadowed
  chronos.waitFor in the chronos macro expansion.
- `chronos.milliseconds(N)` explicitly qualified because `std/times`
  also exports `milliseconds` (returning TimeInterval, not Duration).
- `check await x` -> `let okN = await x; check okN` to dodge chronos's
  "yield in expr not lowered" with await-as-macro-argument.
- `(await x).foo()` -> `let awN = await x; ... awN.foo() ...` for the
  same reason.

waku/persistency/persistency.nim: nph also pulled the proc signatures
across multiple lines; restored explicit `Future[void] {.async.}`
return types after the colon (an intermediate nph pass had elided them).

Suite: 71 / 71 OK against the new async write surface.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* use idiomatic valueOr instead of ifs

* Reworked persistency shutdown, remove not necessary teardown mechanism

* Use const for DefaultStoragePath

* format to follow coding guidelines - no use of result and explicit returns - no functional change

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2026-05-16 00:09:07 +02:00
2026-05-16 00:09:07 +02:00
2026-03-12 19:13:09 +01:00
2026-05-16 00:09:07 +02:00
2026-03-17 14:15:35 +01:00
2026-05-16 00:09:07 +02:00
2026-05-16 00:09:07 +02:00
2026-05-16 00:09:07 +02:00
2026-05-16 00:09:07 +02:00
2022-11-21 09:31:03 +01:00
2026-05-16 00:09:07 +02:00
2026-05-16 00:09:07 +02:00
2026-05-07 18:19:34 +02:00
2026-05-16 00:09:07 +02:00
2026-01-30 11:52:25 +00:00
2026-05-16 00:09:07 +02:00
2026-01-30 01:06:00 +01:00
2026-05-16 00:09:07 +02:00

Logos Messaging Nim

Introduction

This repository implements a set of libp2p protocols aimed to bring private communications.

  • Nim implementation of these specs.
  • C library that exposes the implemented protocols.
  • CLI application that allows you to run an lmn node.
  • Examples.
  • Various tests of above.

For more details see the source code

How to Build & Run ( Linux, MacOS & WSL )

These instructions are generic. For more detailed instructions, see the source code above.

Prerequisites

The standard developer tools, including a C compiler, GNU Make, Bash, and Git. More information on these installations can be found here.

In some distributions (Fedora linux for example), you may need to install which utility separately. Nimbus build system is relying on it.

You'll also need an installation of Rust and its toolchain (specifically rustc and cargo). The easiest way to install these, is using rustup:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Wakunode

# The first `make` invocation will update all Git submodules.
# You'll run `make update` after each `git pull` in the future to keep those submodules updated.
make wakunode2

# Build with custom compilation flags. Do not use NIM_PARAMS unless you know what you are doing.
# Replace with your own flags
make wakunode2 NIMFLAGS="-d:chronicles_colors:none -d:disableMarchNative"

# Run with DNS bootstrapping
./build/wakunode2 --dns-discovery --dns-discovery-url=DNS_BOOTSTRAP_NODE_URL

# See available command line options
./build/wakunode2 --help

To join the network, you need to know the address of at least one bootstrap node. Please refer to the Waku README for more information.

For more on how to run wakunode2, refer to:

Issues

WSL

If you encounter difficulties building the project on WSL, consider placing the project within WSL's filesystem, avoiding the /mnt/ directory.

How to Build & Run ( Windows )

Windows Build Instructions

1. Install Required Tools

  • Git Bash Terminal: Download and install from https://git-scm.com/download/win
  • MSYS2:
    a. Download installer from https://www.msys2.org
    b. Install at "C:" (default location). Remove/rename the msys folder in case of previous installation. c. Use the mingw64 terminal from msys64 directory for package installation.

2. Install Dependencies

Open MSYS2 mingw64 terminal and run the following one-by-one :

pacman -Syu --noconfirm  
pacman -S --noconfirm --needed mingw-w64-x86_64-toolchain  
pacman -S --noconfirm --needed base-devel make cmake upx  
pacman -S --noconfirm --needed mingw-w64-x86_64-rust  
pacman -S --noconfirm --needed mingw-w64-x86_64-postgresql  
pacman -S --noconfirm --needed mingw-w64-x86_64-gcc  
pacman -S --noconfirm --needed mingw-w64-x86_64-gcc-libs  
pacman -S --noconfirm --needed mingw-w64-x86_64-libwinpthread-git  
pacman -S --noconfirm --needed mingw-w64-x86_64-zlib  
pacman -S --noconfirm --needed mingw-w64-x86_64-openssl  
pacman -S --noconfirm --needed mingw-w64-x86_64-python

3. Build Wakunode

  • Open Git Bash as administrator
  • clone nwaku and cd nwaku
  • Execute: ./scripts/build_windows.sh

4. Troubleshooting

If wakunode2.exe isn't generated:

  • Missing Dependencies: Verify with:
    which make cmake gcc g++ rustc cargo python3 upx
    If missing, revisit Step 2 or ensure MSYS2 is at C:\
  • Installation Conflicts: Remove existing MinGW/MSYS2/Git Bash installations and perform fresh install

Developing

Nim Runtime

This repository is bundled with a Nim runtime that includes the necessary dependencies for the project.

Before you can utilize the runtime you'll need to build the project, as detailed in a previous section. This will generate a vendor directory containing various dependencies, including the nimbus-build-system which has the bundled nim runtime.

After successfully building the project, you may bring the bundled runtime into scope by running:

source env.sh

If everything went well, you should see your prompt suffixed with [Nimbus env]$. Now you can run nim commands as usual.

Test Suite

# Run all the Waku tests
make test

# Run a specific test file
make test <test_file_path>
# e.g. : make test tests/wakunode2/test_all.nim

# Run a specific test name from a specific test file
make test <test_file_path> <test_name>
# e.g. : make test tests/wakunode2/test_all.nim "node setup is successful with default configuration"

Building single test files

During development it is helpful to build and run a single test file. To support this make has a specific target:

targets:

  • build/<relative path to your test file.nim>
  • test/<relative path to your test file.nim>

Binary will be created as <path to your test file.nim>.bin under the build directory .

# Build and run your test file separately
make test/tests/common/test_enr_builder.nim

Testing against js-waku

Refer to js-waku repo for instructions.

Formatting

Nim files are expected to be formatted using the nph version present in vendor/nph.

You can easily format file with the make nph/<relative path to nim> file command. For example:

make nph/waku/waku_core.nim

A convenient git hook is provided to automatically format file at commit time. Run the following command to install it:

make install-nph

Examples

Examples can be found in the examples folder. This includes a fully featured chat example.

Tools

Different tools and their corresponding how-to guides can be found in the tools folder.

Bugs, Questions & Features

For an inquiry, or if you would like to propose new features, feel free to open a general issue.

For bug reports, please tag your issue with the bug label.

If you believe the reported issue requires critical attention, please use the critical label to assist with triaging.

To get help, or participate in the conversation, join the Waku Discord server.

Docs

Description
Logos Messaging protocols implemented in Nim
Readme
Languages
Nim 95.7%
Shell 1.4%
C 0.8%
Makefile 0.7%
Nix 0.7%
Other 0.6%