logos-co/logos-cpp-sdk
1
0
Fork 0
mirror of https://github.com/logos-co/logos-cpp-sdk.git synced 2026-08-30 17:21:15 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
master
logos-cpp-sdk/doctests/cpp-sdk-module-runtime.test.yaml
T
Dario LipicarandClaude Opus 5 198f0317ca feat(optional): ?T is two-state, and the generators finally read it (#125) 
* feat(optional): ?T is two-state, and the generators finally read it

No generator in any language read the optional flag — it had never been
implemented. `?T` was a HARD REJECT on the cdylib backend ("module not
cdylib-eligible"), `std::optional<T>` in an impl header fell through to the
opaque `any` with no diagnostic, and a `? name: T` field was emitted as a
required `T`. Three real contracts in the workspace already declare optionals
and were silently getting one of those three answers.

`?T` is TWO-state: a value of T, or empty. Never three — "one LIDL type <-> one
type per language" leaves nowhere for a third state, because every target has
exactly one empty inhabitant.

ONE MEANING, TWO SPELLINGS. `? name: T` (the field flag) and `name: ?T` (the
type kind) are the same declaration. Backends no longer answer that themselves:
logos-lidl's fieldIsOptional/fieldValueType are re-exported from lidl_compat.h
and every site THIS COMMIT TOUCHES reads them, so the two spellings emit
byte-identical code on the cdylib and client backends. That
caught a live drift on the way in — lidlRecordCollidesWithBytesTag read `f.type`
and so refused `? _bytes: tstr` while letting `_bytes: ?tstr` straight through,
one declaration with two answers.

THE WIRE RULE DEPENDS ON THE SLOT. Absent and explicit null are the SAME state
on decode and DIFFERENT on encode:
  - decode is liberal, by exactly one inhabitant: in an optional slot absent and
    null both mean empty; in a required slot both stay errors. A present value
    goes through the decoder a required T would get, so a wrong type still fails
    at the same path — optional widens the domain, it does not switch checking
    off. `?bstr` therefore keeps the LENIENT bytes decode a bare `bstr` gets,
    rather than silently becoming stricter in the optional slot.
  - encode has one canonical form: empty OMITS the key where the slot is NAMED
    (a record field) and is spelled null where it is POSITIONAL (an argument, a
    return, an event parameter — no key to omit, and arity must not change). Key
    omission lives in the record emitter because a Codec only ever sees a value,
    never the slot it sits in. A round trip therefore canonicalises.
  - `?any` collapses onto `any`: nlohmann::json already carries null, so
    std::optional<LogosMap> would give the slot two spellings of empty.

The dispatch gate now admits a missing trailing optional argument and
materialises it as null, exactly the way a missing record field already was. A
method with no optional parameter emits the byte-identical gate it always did.

Header-first: `std::optional<T>` <-> `?T`, composing with records and
containers. `std::optional<std::optional<T>>` has NO LIDL type (three C++ states
over a two-state wire), so it maps down to `?T` — which makes the author's own
declaration stop compiling against the generated codec, deliberately — and says
so at derivation time instead of leaving a conversion error in generated code.

The Qt/Lp consumer surface is NOT fixed and does not pretend to be. The wrappers
real modules get come from legacy/main.cpp, where the AST is flattened to a
single Qt type-name string per slot before optionality could be seen; Qt has no
optional metatype, so `?T` lands on QVariant — the right shape (an invalid
QVariant is Qt's empty inhabitant) with no type. The generator now prints a Note
naming every flattened slot so an affected build is never silent, and
docs/project.md records exactly what a Qt consumer will still do with an
optional field.

Verified by output equivalence, not by a green build: the generator was built
before and after and run over every .lidl in the workspace plus the impl-header
fixtures, in cdylib, consumer-qt, consumer-lp, client and header-first modes.
428 of 465 artefacts are byte-identical; all 37 that differ belong to one of the
four contracts that declare an optional (the 38th path is the manifest). The
harness's sensitivity is pinned by a negative control: qt vs lp output differs
in 45 files. The emitted codec was additionally compiled under -Wall -Wextra and
run against the rules above — omission, absent==null, required-still-rejects,
present-but-wrong-still-fails, and canonicalising round trip.

Tests: 199 pass, 0 fail (180 before, 19 new).

Requires logos-lidl's optionality accessors and logos-protocol's
Codec<std::optional<T>>.

NOT FIXED, AND IT IS THE PATH THAT MATTERS MOST. The legacy interface-wrapper
path is untouched, and it is the one every real module builds through
(buildPlugin.nix:145 -> logos-cpp-generator --general-only). There the two
spellings still diverge:

  ? maybe: tstr   ->  QString maybe{};   __m.value("maybe").toString()
  maybe: ?tstr    ->  QVariant maybe{};  __m.value("maybe")

and --api-style lp diverges too, neither side being std::optional. So R3 holds
on the backends below and NOT on the Qt consumer a shipping module actually
gets. logos-chat-module -- the contract that prompted this work -- uses the
field-flag spelling, so it lands on the branch that silently defaults.

The cause is upstream of codegen: legacy/main.cpp's moduleRecordsToJson and
moduleMethodsToJson flatten every TypeExpr to a single Qt TYPE-NAME STRING, so
optionality (along with nesting, map key types and descriptions) is gone before
generator_lib.cpp sees it. Widening that interface is a larger change and is
deliberately not attempted here. The only R3 test on a Qt surface covers
lidl_gen_client.cpp, which is on no live build path.

* chore: re-pin logos-lidl to master for the optionality accessors

lidl_compat.h re-exports typeIsOptional / optionalValueType / fieldIsOptional /
fieldValueType / paramIsOptional / paramValueType, which landed in
logos-lidl#7. The pinned lidl predated it, so CI failed to compile.

logos-lidl 8c95d4f -> 35f33d8. Tests: 199 pass, 0 fail.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* chore: re-pin logos-protocol to master for Codec<std::optional<T>>

The generated record codecs emit Codec<std::optional<T>> for an optional
field; that specialisation landed in logos-protocol#37 and the pinned
protocol predated it.

Note this repo's own tests would NOT have caught the omission -- the
generator tests string-assert emitted text rather than compiling it, so a
missing codec specialisation only surfaces when a real module compiles
generated optional code (logos-test-modules' ext provider).

logos-protocol 4359557 -> 72754ab. Tests: 199 pass, 0 fail.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(doctests): override logos-lidl alongside every logos-cpp-sdk override

The doc-tests build downstream repos (logoscore-cli, capability_module,
accounts_module) with --override-input logos-cpp-sdk. Nix does not carry the
overridden input's OWN lock, so those builds got this branch's cpp-sdk source
while still resolving logos-lidl from their own, older locks. The shipped
share/lidl-frontend/lidl_compat.h then calls accessors that lidl does not
have:

  lidl_compat.h:46: error: 'paramValueType' has not been declared in 'lidl'
  lidl_compat.h:92: error: 'fieldValueType' was not declared in this scope

Every --override-input logos-cpp-sdk now has a matching
--override-input <same-path>/logos-cpp-sdk/logos-lidl.

This is specific to the override path. A normal consumer running
'nix flake update logos-cpp-sdk' inherits cpp-sdk's own lock, which pins the
lidl carrying these accessors, and is unaffected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(doctests): move logos-lidl at the qt-sdk nodes, not under logos-cpp-sdk

The doc-tests failed to build logos-qt-generator:

  share/lidl-frontend/lidl_compat.h:46: error: 'paramValueType' has not been
  declared in 'lidl'

MECHANISM. This SDK installs cpp-generator/experimental/lidl_compat.h into
$out/share/lidl-frontend/, and logos-qt-sdk's logos-qt-generator *compiles*
that installed header against qt-sdk's OWN logos-lidl input. Under
logos-qt-sdk, logos-lidl is a SIBLING of logos-cpp-sdk, not a descendant:

  logos-qt-sdk
  |-- logos-cpp-sdk   <- --override-input moves this to the commit under test
  `-- logos-lidl      <- stays on qt-sdk's lock (8c95d4f), lacks the accessors

logos-logoscore-cli and logos-module-builder both declare
`logos-qt-sdk.inputs.logos-cpp-sdk.follows = "logos-cpp-sdk"` but no lidl
follows, so overriding the SDK hands qt-sdk a new lidl_compat.h next to its
old lidl. The failing derivation is logos-qt-generator — not anything in
logos-cpp-sdk, which is why the previous attempt aimed at the wrong node.

THE FIX is one `<path-to-logos-qt-sdk>/logos-lidl` override per qt-sdk node
that ends up on the SDK under test. A tree-walk over the resolved lock found
four in logoscore-cli's closure and two per module build; with the overrides
applied the walk reports zero remaining.

WHAT WAS REMOVED, and why it was doing nothing:

  * The `.../logos-cpp-sdk/logos-lidl` overrides added in bef3ef5 were no-ops.
    With only `--override-input logos-cpp-sdk <sha>`, that node's logos-lidl
    already resolves to 35f33d87 out of cpp-sdk's own lock — nix >= 2.26
    carries an overridden input's lock, and CI runs Determinate Nix. Verified
    by resolving the lock with and without them: byte-identical.
  * The `logos-module-client/...` overrides never matched anything. Nix says so
    out loud ("does not match any input"): logoscore-cli has no such root
    input; module-client only appears under logos-test-modules/, outside the
    runtime closure. The prose claiming it pins the SDK is corrected too.

cpp-sdk-concurrent-dispatch is fixed here as well — it failed the same way and
carried no lidl overrides at all.

VERIFIED locally against bef3ef5, the exact commit CI failed on:

  * accounts .lgx  -> exit 0, logos-accounts_module-module-lib.lgx (5,939,898 B)
  * logoscore CLI  -> exit 0, ./logos/bin/logoscore reports
                      "logos-cpp-sdk bef3ef57d3f489073672e70a786c550df7edd003"
  * negative control (same command minus the single qt-sdk lidl flag) fails
    with CI's exact derivation,
    /nix/store/pf96n2ldvhy6sq39ygkh5zdqx7dcn4df-logos-qt-generator-0.1.0.drv
  * no "does not match any input" warnings remain on any command

The durable fix is a one-line bump of logos-qt-sdk's own flake.lock logos-lidl
to master (logos-lidl#7 is purely additive: six new inline helpers, nothing
removed or renamed). Once qt-sdk carries it, every override added here can go.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 07:19:47 -03:00

300 lines
15 KiB
YAML
Raw Permalink Blame History

name: "Running a Real Module Against This C++ SDK"
output: cpp-sdk-module-runtime.md
release: ""
intro: |
`logos-cpp-sdk` is the foundation every other Logos component is built on: it
pins nixpkgs/Qt and ships the SDK that `logos-liblogos` (`logos_host`,
`liblogos_core`), the module client, and every module compile and link against
— `LogosAPI`, `LogosResult`, the IPC layer, and the code generator. A change
here ripples through the entire stack, so the way to know it is safe is to run
a real module on top of it. This doc-test does exactly that, end-to-end through
the headless `logoscore` runtime:
1. Build the `logoscore` CLI, **overriding `logos-cpp-sdk` with the commit
under test** — and overriding it in the same way for every consumer in
`logoscore`'s closure (`logos-liblogos` and the `capability_module`'s
`logos-module-builder`). Because the published flakes pin the SDK
independently (there is no single `follows` unifying them), all of these
must point at the same commit so the whole runtime is built and linked
against one consistent SDK ABI. Each `logos-qt-sdk` dragged onto the new
SDK also gets its sibling `logos-lidl` moved with it — see the note in
the first section.
2. Build the `lgpm` local package manager.
3. Build the real [`accounts_module`](https://github.com/logos-co/logos-accounts-module)
as an `.lgx` package straight from its own flake — **also built against the
SDK commit under test** — and install it into a `./modules` directory with
`lgpm`.
4. Start `logoscore` in daemon mode (`-D`), load `accounts_module`, introspect
it, and call its methods — verifying a module compiled against this SDK
actually loads and round-trips real values over the SDK's IPC.
Because every layer (host, module loader, IPC, and the module itself) is built
against the SDK commit under test, a green run is real evidence that this
change keeps the module runtime working from the bottom of the stack up.
what_you_build: "The real `accounts_module`, installed with `lgpm` and called through a `logoscore` daemon — every layer compiled against this C++ SDK commit."
what_you_learn:
- How to build `logoscore` against a specific `logos-cpp-sdk` commit by overriding it across every consumer in the closure
- Why a foundational input pinned independently by several flakes needs the override applied at each consumer, not just once
- How to build a real module's `.lgx` against the same SDK commit
- How to install an `.lgx` into a modules directory with `lgpm`
- How to start the `logoscore` daemon, load a module, and call its methods
prerequisites:
- |
**Nix** with flakes enabled. Install from [nixos.org](https://nixos.org/download.html), then enable flakes:
```bash
mkdir -p ~/.config/nix
echo 'experimental-features = nix-command flakes' >> ~/.config/nix/nix.conf
```
Verify: `nix flake --help >/dev/null 2>&1 && echo "Flakes enabled"`
- "**git** — to clone the module repository."
- "A Linux or macOS machine."
sections:
- title: "Build logoscore against this C++ SDK"
step: true
text: |
Build the `logoscore` CLI from its published flake, but **override
`logos-cpp-sdk` to the commit under test** — and apply the same override to
every consumer that pins the SDK in `logoscore`'s closure. The result is
symlinked to `./logos/`.
> Unlike a leaf input, the SDK is pinned independently by `logos-liblogos`
> and the `capability_module`'s `logos-module-builder` — there is no single
> `follows` tying them together in the published flakes. So we override it
> at each of those paths (e.g.
> `--override-input logos-liblogos/logos-cpp-sdk …`) to keep the whole
> runtime on one consistent SDK ABI. Each override URL carries a `{release}`
> placeholder the doc-test runner expands to a concrete ref: locally that is
> this checkout's `HEAD` (see `run.sh`); in CI it is the commit being
> tested. With no pin it falls back to latest `master`.
> **Why the `logos-qt-sdk/logos-lidl` overrides.** This SDK installs
> `share/lidl-frontend/lidl_compat.h`, and `logos-qt-sdk`'s
> `logos-qt-generator` *compiles* that header against **its own**
> `logos-lidl` input. `logos-lidl` is a **sibling** of `logos-cpp-sdk`
> under `logos-qt-sdk`, not a descendant, so overriding the SDK moves the
> header forward while qt-sdk keeps its older lidl — and the generator
> fails to compile the shim. Each `logos-qt-sdk` node that ends up on the
> SDK under test therefore needs its `logos-lidl` moved with it.
steps:
- title: "Build the CLI with the SDK override"
run: |
nix build 'github:logos-co/logos-logoscore-cli{release}' \
--override-input logos-cpp-sdk 'github:logos-co/logos-cpp-sdk{release}' \
--override-input logos-qt-sdk/logos-lidl 'github:logos-co/logos-lidl' \
--override-input logos-liblogos/logos-cpp-sdk 'github:logos-co/logos-cpp-sdk{release}' \
--override-input logos-liblogos/logos-qt-sdk/logos-lidl 'github:logos-co/logos-lidl' \
--override-input logos-capability-module/logos-module-builder 'github:logos-co/logos-module-builder{release}' \
--override-input logos-capability-module/logos-module-builder/logos-cpp-sdk 'github:logos-co/logos-cpp-sdk{release}' \
--override-input logos-capability-module/logos-module-builder/logos-qt-sdk/logos-lidl 'github:logos-co/logos-lidl' \
--override-input logos-capability-module/logos-module-builder/logos-test-framework/logos-qt-sdk/logos-lidl 'github:logos-co/logos-lidl' \
--out-link ./logos
code_block: |
nix build 'github:logos-co/logos-logoscore-cli' \
--override-input logos-cpp-sdk 'github:logos-co/logos-cpp-sdk' \
--override-input logos-qt-sdk/logos-lidl 'github:logos-co/logos-lidl' \
--override-input logos-liblogos/logos-cpp-sdk 'github:logos-co/logos-cpp-sdk' \
--override-input logos-liblogos/logos-qt-sdk/logos-lidl 'github:logos-co/logos-lidl' \
--override-input logos-capability-module/logos-module-builder/logos-cpp-sdk 'github:logos-co/logos-cpp-sdk' \
--override-input logos-capability-module/logos-module-builder/logos-qt-sdk/logos-lidl 'github:logos-co/logos-lidl' \
--override-input logos-capability-module/logos-module-builder/logos-test-framework/logos-qt-sdk/logos-lidl 'github:logos-co/logos-lidl' \
--out-link ./logos
check_file: "logos/bin/logoscore"
post_text: |
The build produces `logos/bin/logoscore` plus bundled runtime libraries
and a `logos/modules/` directory containing the built-in
`capability_module` (required for the auth handshake when loading
modules). Every overridden input resolves to the same SDK commit, so
`logos_host`, `liblogos_core`, the module client, and the capability
module are all rebuilt and linked against this SDK.
- title: "Build the lgpm package manager"
step: true
text: |
`lgpm` installs `.lgx` packages into a modules directory and scans what is
installed. Build it from the `logos-package-manager` flake and link it as
`./lgpm`. (`lgpm` is plain C++ with no SDK dependency, so it needs no
override.)
steps:
- title: "Build lgpm"
run: "nix build 'github:logos-co/logos-package-manager#cli' -o lgpm"
check_file: "lgpm/bin/lgpm"
post_text: "The executable is at `./lgpm/bin/lgpm`."
- title: "Build and install the accounts module against this SDK"
step: true
text: |
Clone [`logos-accounts-module`](https://github.com/logos-co/logos-accounts-module),
build its `.lgx` straight from its flake's `#lgx` output **against the SDK
commit under test**, and install it into a local `./modules` directory with
`lgpm`. Building the module against the same SDK as the runtime keeps the
plugin ABI-compatible with the host that will load it. Every module built
with [`logos-module-builder`](https://github.com/logos-co/logos-module-builder)
exposes a ready-to-install `#lgx`, and the builder owns the module's SDK
pin — so the override path here is
`logos-module-builder/logos-cpp-sdk`.
steps:
- title: "Clone the module"
text: |
We clone over HTTPS so the step works in CI; over SSH the URL is
`git@github.com:logos-co/logos-accounts-module.git`.
run: "git clone --depth 1 https://github.com/logos-co/logos-accounts-module.git"
check_file: "logos-accounts-module/flake.nix"
- title: "Build the module's .lgx against this SDK"
text: |
Build the `#lgx` output, overriding the module builder's `logos-cpp-sdk`
to the commit under test, and link it as `./accounts-lgx`. (This
compiles the module and its SDK dependencies through Nix, so the first
build is slow.)
run: |
nix build 'path:./logos-accounts-module#lgx' \
--override-input logos-module-builder 'github:logos-co/logos-module-builder{release}' \
--override-input logos-module-builder/logos-cpp-sdk 'github:logos-co/logos-cpp-sdk{release}' \
--override-input logos-module-builder/logos-qt-sdk/logos-lidl 'github:logos-co/logos-lidl' \
-o accounts-lgx
code_block: |
# From inside the clone this is simply:
# nix build '.#lgx' --override-input logos-module-builder/logos-cpp-sdk 'github:logos-co/logos-cpp-sdk'
nix build 'path:./logos-accounts-module#lgx' \
--override-input logos-module-builder/logos-cpp-sdk 'github:logos-co/logos-cpp-sdk' \
--override-input logos-module-builder/logos-qt-sdk/logos-lidl 'github:logos-co/logos-lidl' \
-o accounts-lgx
post_text: "The `.lgx` package is now under `./accounts-lgx/`:"
extra_run:
run: "ls accounts-lgx/*.lgx"
- title: "Seed the modules directory with the bundled capability module"
text: |
`accounts_module` is loaded through the host's capability layer, so the
modules directory also needs the `capability_module` that ships with
`logoscore` (and that we just rebuilt against this SDK). Copy it across
first.
run: |
mkdir -p modules
cp -RL ./logos/modules/. ./modules/
check_file: "modules/capability_module/manifest.json"
- title: "Install the .lgx with lgpm"
text: |
Install the freshly-built package into `./modules`. `accounts_module` is
a `core` module, so it goes to `--modules-dir`. The package is unsigned
(a local dev build), so we pass `--allow-unsigned`.
run: "./lgpm/bin/lgpm --modules-dir ./modules --allow-unsigned install --file accounts-lgx/*.lgx"
expect_contains:
- "Installed to:"
- title: "Confirm the install"
text: "Scan the directory and confirm the module landed:"
run: "./lgpm/bin/lgpm --modules-dir ./modules list"
expect_contains:
- "accounts_module"
check_file: "modules/accounts_module/manifest.json"
- title: "Run the daemon and call the module"
step: true
text: |
Start `logoscore` in daemon mode pointed at `./modules`, then use the client
subcommands to load `accounts_module`, introspect it, and call its methods.
Daemon output is captured in `logs.txt`.
steps:
- title: "Start the daemon"
text: |
Start logoscore in daemon mode in the background, capturing output to
`logs.txt`:
run: "sh -c './logos/bin/logoscore -D -m ./modules > logs.txt 2>&1 &'"
code_block: "logoscore -D -m ./modules > logs.txt &"
post_text: |
The `-D` flag starts the daemon. The client subcommands below connect to
this running process via the config written under `~/.logoscore/`.
- run: "sleep 3"
- title: "Inspect the startup log"
text: "Review the daemon's startup output:"
run: "cat logs.txt"
- title: "Check daemon status"
text: "Verify the daemon is running:"
run: "./logos/bin/logoscore status"
code_block: "logoscore status"
- title: "List discovered modules"
text: "`accounts_module` should be visible in the scan directory:"
run: "./logos/bin/logoscore list-modules"
code_block: "logoscore list-modules"
expect_contains:
- "accounts_module"
- title: "Load the module"
text: "Load `accounts_module` into the running daemon:"
run: "./logos/bin/logoscore load-module accounts_module"
code_block: "logoscore load-module accounts_module"
expect_contains:
- "accounts_module"
- title: "Confirm the module is loaded"
text: |
Re-run `status`; the module that was `not_loaded` before now reports
`loaded`:
run: "./logos/bin/logoscore status"
code_block: "logoscore status"
expect_contains:
- "accounts_module"
- '"status":"loaded"'
- title: "Introspect the module with module-info"
text: |
`module-info` lists the `Q_INVOKABLE` methods the module exposes — the
same methods you can `call`. Each one is dispatched over the SDK's IPC
layer:
run: "./logos/bin/logoscore module-info accounts_module"
code_block: "logoscore module-info accounts_module"
expect_contains:
- "accounts_module"
- "createRandomMnemonic"
- title: "Call a method"
text: |
Generate a fresh 12-word BIP-39 mnemonic. `createRandomMnemonic` takes
the word count and returns the phrase — a real round-trip dispatched
over the SDK's IPC layer into the module compiled against this SDK:
run: "./logos/bin/logoscore call accounts_module createRandomMnemonic 12"
code_block: "logoscore call accounts_module createRandomMnemonic 12"
expect_contains:
- '"result"'
- title: "Call a second method"
text: |
`lengthToEntropyStrength` maps a mnemonic word count to its entropy
strength in bits — 12 words is 128 bits. This exercises an `int`
round-trip through `LogosResult` over the SDK's IPC:
run: "./logos/bin/logoscore call accounts_module lengthToEntropyStrength 12"
code_block: "logoscore call accounts_module lengthToEntropyStrength 12"
expect_contains:
- '"result":128'
- title: "Stop the daemon"
text: "Shut the daemon down cleanly:"
run: "./logos/bin/logoscore stop"
code_block: "logoscore stop"
post_text: |
The daemon removes its state file and exits.
- run: "sleep 2"
- title: "Confirm the daemon has stopped"
text: |
With no daemon running, the client reports `not_running` and exits
non-zero, so we add `|| true` to let the doc-test assert on the output:
run: "./logos/bin/logoscore status || true"
code_block: "logoscore status"
expect_contains:
- '"status":"not_running"'
Reference in New Issue View Git Blame Copy Permalink