From c7444bc29a1b480505025d0ed1679bedb71bfa29 Mon Sep 17 00:00:00 2001 From: Dario Lipicar Date: Thu, 16 Jul 2026 18:50:40 -0300 Subject: [PATCH] Follow-ups to #100: lp-consumer bstr decode, Qt-free cdylib event types, binary-event coverage (#102) * cdylib events: Qt-free types, and drop the unused bytes encoder Three follow-ups to the bstr event fix, all in the cdylib events sidecar -- a Qt-FREE translation unit: - An `any`/map event parameter was emitted as a bare QVariant/QVariantMap, which does not compile there. Spell those as their nlohmann aliases (LogosMap / LogosList) and pull in when they appear. - std::vector> fell through the impl-header parser's unknown-type fallback to `any`, so the cdylib gate admitted it and the generator then emitted QVariant. Parse it as `[bstr]` so the gate rejects it with a message naming the offending parameter. - The bytes encoder was emitted into every module's sidecar, leaving an unused static function (-Wunused-function) wherever no event carries binary data. Emit it only when a bstr event parameter exists. Co-Authored-By: Claude Opus 4.8 * lp consumer: decode bstr into std::vector The Qt-free (`lp`) consumer wrappers -- what every universal C++ module gets for its dependencies -- had no QByteArray in their type tables, so a `bstr` event parameter, method argument, or return degraded to QVariant and then to LogosMap. A consumer subscribing to a binary event was handed the raw tagged JSON object {"_bytes": ""} instead of the bytes, with no generated decode. Teach the tables about QByteArray (-> std::vector) and marshal it through the canonical tagged form in both directions: logos::bytesToJson on the way out, logos::jsonToBytes on the way in. Those live in logos_json.h -- Qt-free and protocol-free, so the generated wrappers and module code can share them. The Qt apiStyle already did this via QByteArray::toBase64/fromBase64. Without this, a subscriber written the obvious way -- onBinaryReady([](const std::string&, const std::vector& payload) {...}) -- compiles (nlohmann::json has an implicit conversion operator) and then throws at runtime on every event, so the callback body silently never runs. Co-Authored-By: Claude Opus 4.8 * tests: cover binary event payloads by value, not just by source text The regression test for #99 asserts on generated source text, so it stays green against an encoder that emits the wrong bytes. Add the value-level half: - tests/sdk/test_logos_json_bytes.cpp exercises the canonical tagged-bytes codec against the RFC 4648 vectors, the URL-safe alphabet, every len%3 tail group, embedded NULs and high bytes, a 109,447-byte payload (the size from #99), and the lenient/padded decode paths. - tests/experimental/test_lidl_gen_cdylib.cpp additionally pins the Qt-free spelling of JSON event payloads, the rejection of [bstr], and the omission of the bytes encoder from modules whose events carry no binary data. Co-Authored-By: Claude Opus 4.8 * doctests: prove a binary event payload survives the round trip Neither doc-test covered bytes-in-an-event -- the gap #99 fell through. The generator round-trip carried `bstr` only as a method argument and return, and the composition doc-test, which is the one that actually runs two modules under logoscore and subscribes to an event, carried only a string. So a generator that dropped every bstr event argument kept both of them green. - cpp-sdk-module-composition: greeter_module gains a `blobReady(label, payload)` event and an `emitBlob(size)` method; orchestrator_module subscribes and reports the length AND a checksum of what it received. Length alone would not catch a corrupted payload -- a wrong alphabet round-trips to the same size. - cpp-sdk-generator-roundtrip: sensor_module gains a `capture(id, frame: bstr)` event, and a new step shows the generated event body encoding it through lidlBytesToJson rather than pushing it raw. Co-Authored-By: Claude Opus 4.8 * logos_json.h: include for size_t The tagged-bytes codec uses size_t but relied on it arriving transitively through the other includes. Include directly so the header is self-contained. (Copilot review, PR #102.) Co-Authored-By: Claude Opus 4.8 --------- Co-authored-by: Claude Opus 4.8 --- .../experimental/impl_header_parser.cpp | 9 ++ .../experimental/lidl_gen_cdylib.cpp | 70 +++++++- cpp-generator/legacy/generator_lib.cpp | 17 +- cpp/logos_json.h | 83 ++++++++++ cpp/logos_lp_client.h | 5 + .../cpp-sdk-generator-roundtrip.test.yaml | 27 +++- doctests/cpp-sdk-module-composition.test.yaml | 132 +++++++++++++++- tests/experimental/test_lidl_gen_cdylib.cpp | 149 ++++++++++++++++-- tests/sdk/CMakeLists.txt | 1 + tests/sdk/test_logos_json_bytes.cpp | 115 ++++++++++++++ 10 files changed, 582 insertions(+), 26 deletions(-) create mode 100644 tests/sdk/test_logos_json_bytes.cpp diff --git a/cpp-generator/experimental/impl_header_parser.cpp b/cpp-generator/experimental/impl_header_parser.cpp index 7599f87..d77bb2c 100644 --- a/cpp-generator/experimental/impl_header_parser.cpp +++ b/cpp-generator/experimental/impl_header_parser.cpp @@ -65,6 +65,15 @@ static TypeExpr cppTypeToLidl(const QString& raw) if (inner == "uint8_t") { return { TypeExpr::Primitive, "bstr", {} }; } + // std::vector> — an array of byte strings. Spelled + // out so it lands on `[bstr]` rather than the opaque `any` fallback + // below: `any` would make the cdylib gate admit it and then emit a bare + // QVariant into the Qt-free TU. As `[bstr]` the gate rejects it with a + // message naming the offending parameter. + if (inner == "std::vector") { + TypeExpr elem = { TypeExpr::Primitive, "bstr", {} }; + return { TypeExpr::Array, "", { elem } }; + } if (inner == "int64_t") { TypeExpr elem = { TypeExpr::Primitive, "int", {} }; return { TypeExpr::Array, "", { elem } }; diff --git a/cpp-generator/experimental/lidl_gen_cdylib.cpp b/cpp-generator/experimental/lidl_gen_cdylib.cpp index f59d069..578cc1e 100644 --- a/cpp-generator/experimental/lidl_gen_cdylib.cpp +++ b/cpp-generator/experimental/lidl_gen_cdylib.cpp @@ -76,6 +76,51 @@ QString stdReturnToJson(const MethodDecl& md, const QString& var) return "nlohmann::json(" + var + ")"; } +// Qt-free spelling of a LIDL type. lidlTypeToStd() falls back to Qt containers +// (QVariant / QVariantMap / QVariantList) for the composite types, but a cdylib +// TU is Qt-free by definition and typeSupported() admits `any` and maps — so +// spell those as their nlohmann aliases (LogosMap / LogosList) instead. Without +// this the events sidecar emits a bare `QVariant` parameter and does not +// compile. +QString lidlTypeToStdCdylib(const TypeExpr& te) +{ + if (te.kind == TypeExpr::Primitive && te.name == "any") + return "LogosMap"; + if (te.kind == TypeExpr::Map) + return "LogosMap"; + if (te.kind == TypeExpr::Array && te.elements.size() == 1 + && te.elements[0].kind == TypeExpr::Primitive + && te.elements[0].name == "any") + return "LogosList"; + return lidlTypeToStd(te); +} + +// True when the module declares at least one `bstr` event parameter — the only +// reason the events sidecar needs the bytes encoder. Emitting it unconditionally +// leaves an unused static function (a -Wunused-function warning) in every module +// whose events carry no binary data. +bool hasBytesEventParam(const ModuleDecl& module) +{ + for (const EventDecl& ed : module.events) + for (const ParamDecl& pd : ed.params) + if (pd.type.kind == TypeExpr::Primitive && pd.type.name == "bstr") + return true; + return false; +} + +// True when any event parameter is spelled LogosMap / LogosList, so the sidecar +// needs for those aliases. +bool hasJsonEventParam(const ModuleDecl& module) +{ + for (const EventDecl& ed : module.events) + for (const ParamDecl& pd : ed.params) { + const QString t = lidlTypeToStdCdylib(pd.type); + if (t == "LogosMap" || t == "LogosList") + return true; + } + return false; +} + void emitBytesEncodeHelpers(QTextStream& s) { s << "// Canonical tagged bytes form {\"_bytes\": base64url} (see logos_protocol.h)\n"; @@ -483,16 +528,29 @@ QString lidlMakeEventsSourceCdylib(const ModuleDecl& module, s << "#include \n\n"; s << "#include \n"; s << "#include \n"; - s << "#include \n\n"; - s << "namespace {\n\n"; - emitBytesEncodeHelpers(s); - s << "} // namespace\n\n"; + s << "#include \n"; + // LogosMap / LogosList (nlohmann aliases) appear in the emitted signatures + // whenever an event carries a map or an `any` payload. + if (hasJsonEventParam(module)) + s << "#include \n"; + s << "\n"; + + // Only the modules that actually emit binary event payloads need the bytes + // encoder; emitting it everywhere would leave it unused (and warned about). + if (hasBytesEventParam(module)) { + s << "namespace {\n\n"; + emitBytesEncodeHelpers(s); + s << "} // namespace\n\n"; + } for (const EventDecl& ed : module.events) { s << "void " << implClass << "::" << ed.name << "("; for (int i = 0; i < ed.params.size(); ++i) { - const QString stdType = lidlTypeToStd(ed.params[i].type); - if (stdType == "std::string" || stdType.startsWith("std::vector")) + const QString stdType = lidlTypeToStdCdylib(ed.params[i].type); + // Must match the author's declaration in the `logos_events:` block: + // the non-scalar types are conventionally taken by const-ref there. + if (stdType == "std::string" || stdType.startsWith("std::vector") + || stdType == "LogosMap" || stdType == "LogosList") s << "const " << stdType << "& " << ed.params[i].name; else s << stdType << " " << ed.params[i].name; diff --git a/cpp-generator/legacy/generator_lib.cpp b/cpp-generator/legacy/generator_lib.cpp index 28e2a9a..f0a6653 100644 --- a/cpp-generator/legacy/generator_lib.cpp +++ b/cpp-generator/legacy/generator_lib.cpp @@ -34,7 +34,7 @@ QString mapParamType(const QString& qtType) { const QString base = normalizeType(qtType); static const QSet known = { - "void","bool","int","double","float","QString","QStringList","QJsonArray","QVariantList","QVariantMap","QVariant" + "void","bool","int","double","float","QString","QStringList","QByteArray","QJsonArray","QVariantList","QVariantMap","QVariant" }; if (known.contains(base)) return base; // Fallback to QVariant for unknown types @@ -46,7 +46,7 @@ QString mapReturnType(const QString& qtType) const QString base = normalizeType(qtType); if (base.isEmpty() || base == "void") return QString("void"); static const QSet known = { - "bool","int","double","float","QString","QStringList","QJsonArray","QVariantList","QVariantMap","QVariant","LogosResult" + "bool","int","double","float","QString","QStringList","QByteArray","QJsonArray","QVariantList","QVariantMap","QVariant","LogosResult" }; if (known.contains(base)) return base; return QString("QVariant"); @@ -84,6 +84,7 @@ static QString mapParamTypeStd(const QString& qtType) const QString base = mapParamType(qtType); if (base == "QString") return "std::string"; if (base == "QStringList") return "std::vector"; + if (base == "QByteArray") return "std::vector"; if (base == "QJsonArray") return "LogosList"; if (base == "QVariantList") return "LogosList"; if (base == "QVariantMap") return "LogosMap"; @@ -98,6 +99,7 @@ static QString mapReturnTypeStd(const QString& qtType) if (base == "void") return "void"; if (base == "QString") return "std::string"; if (base == "QStringList") return "std::vector"; + if (base == "QByteArray") return "std::vector"; if (base == "QJsonArray") return "LogosList"; if (base == "QVariantList") return "LogosList"; if (base == "QVariantMap") return "LogosMap"; @@ -119,6 +121,9 @@ static QString stdParamToQVariant(const QString& qtType, const QString& argName) return "[&]{ QStringList _q; _q.reserve(static_cast(" + argName + ".size())); for (const auto& _s : " + argName + ") _q.append(QString::fromStdString(_s)); return _q; }()"; + if (base == "QByteArray") + return "QByteArray(reinterpret_cast(" + argName + + ".data()), static_cast(" + argName + ".size()))"; if (base == "QJsonArray") return "QJsonDocument::fromJson(QByteArray::fromStdString(" + argName + ".dump())).array()"; @@ -157,6 +162,9 @@ static QString qVariantToStdReturn(const QString& qtType, const QString& varExpr return "[&]{ std::vector _v; const QStringList _q = " + varExpr + ".toStringList(); _v.reserve(static_cast(_q.size())); " "for (const QString& _s : _q) _v.push_back(_s.toStdString()); return _v; }()"; + if (base == "QByteArray") + return "[&]{ const QByteArray _b = " + varExpr + + ".toByteArray(); return std::vector(_b.begin(), _b.end()); }()"; if (base == "QJsonArray" || base == "QVariantList") return "LogosList::parse(QJsonDocument(QJsonArray::fromVariantList(" + varExpr + ".toList())).toJson(QJsonDocument::Compact).toStdString())"; @@ -834,6 +842,10 @@ static QString lpPushExpr(const QString& qtType, const QString& argName) if (std == "StdLogosResult") return "nlohmann::json{{\"success\", " + argName + ".success}, {\"value\", " + argName + ".value}, {\"error\", " + argName + ".error}}"; + // Bytes must go out in the canonical tagged form; pushed raw, nlohmann would + // serialize the vector as a plain JSON array of numbers. + if (std == "std::vector") + return "logos::bytesToJson(" + argName + ")"; return argName; } @@ -848,6 +860,7 @@ static QString lpFromJsonExpr(const QString& qtType, const QString& jv) if (t == "double") return "(" + jv + ".is_number() ? " + jv + ".get() : 0.0)"; if (t == "bool") return "(" + jv + ".is_boolean() ? " + jv + ".get() : false)"; if (t == "std::vector") return "logos::jsonToStringVec(" + jv + ")"; + if (t == "std::vector") return "logos::jsonToBytes(" + jv + ")"; if (t == "LogosMap") return "(" + jv + ".is_object() ? " + jv + " : LogosMap::object())"; if (t == "LogosList") return "(" + jv + ".is_array() ? " + jv + " : LogosList::array())"; if (t == "StdLogosResult") return "logos::jsonToStdResult(" + jv + ")"; diff --git a/cpp/logos_json.h b/cpp/logos_json.h index e1efaa9..90bdb3b 100644 --- a/cpp/logos_json.h +++ b/cpp/logos_json.h @@ -1,8 +1,91 @@ #pragma once #include +#include +#include +#include +#include + // Semantic aliases for nlohmann::json used in universal module impl classes. // The code generator recognizes these names and emits QVariantMap / QVariantList // conversions in the Qt glue layer, so impl classes remain Qt-free. using LogosMap = nlohmann::json; using LogosList = nlohmann::json; + +namespace logos { + +// The canonical tagged form for binary payloads on the wire: +// +// {"_bytes": ""} +// +// It is what logos-protocol emits and expects (logos_json_convert.cpp, +// implementations/plain/json_mapping.cpp), and it is lossless for arbitrary +// bytes — including embedded NULs, which a plain JSON string would not survive. +// The Qt side reaches this form through QByteArray::toBase64/fromBase64 with +// Base64UrlEncoding | OmitTrailingEquals; these are the Qt-free equivalents, +// used by the generated `lp` wrappers and by universal (Qt-free) module code. + +inline std::string b64UrlEncode(const std::vector& bytes) +{ + static const char* alpha = + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"; + std::string out; + size_t i = 0; + while (i + 3 <= bytes.size()) { + uint32_t n = (uint32_t(bytes[i]) << 16) | (uint32_t(bytes[i + 1]) << 8) + | uint32_t(bytes[i + 2]); + out += alpha[(n >> 18) & 0x3f]; out += alpha[(n >> 12) & 0x3f]; + out += alpha[(n >> 6) & 0x3f]; out += alpha[n & 0x3f]; + i += 3; + } + if (i < bytes.size()) { + uint32_t n = uint32_t(bytes[i]) << 16; + if (i + 1 < bytes.size()) n |= uint32_t(bytes[i + 1]) << 8; + out += alpha[(n >> 18) & 0x3f]; out += alpha[(n >> 12) & 0x3f]; + if (i + 1 < bytes.size()) out += alpha[(n >> 6) & 0x3f]; + } + return out; +} + +inline std::vector b64UrlDecode(const std::string& in) +{ + auto idx = [](char ch) -> int { + if (ch >= 'A' && ch <= 'Z') return ch - 'A'; + if (ch >= 'a' && ch <= 'z') return ch - 'a' + 26; + if (ch >= '0' && ch <= '9') return ch - '0' + 52; + if (ch == '-') return 62; + if (ch == '_') return 63; + return -1; // skips '=' padding and any stray character + }; + std::vector out; + uint32_t buf = 0; + int bits = 0; + for (char ch : in) { + const int v = idx(ch); + if (v < 0) continue; + buf = (buf << 6) | static_cast(v); + bits += 6; + if (bits >= 8) { + bits -= 8; + out.push_back(static_cast((buf >> bits) & 0xff)); + } + } + return out; +} + +// Bytes -> the tagged JSON object. +inline nlohmann::json bytesToJson(const std::vector& bytes) +{ + return nlohmann::json{{"_bytes", b64UrlEncode(bytes)}}; +} + +// The tagged JSON object -> bytes. Lenient, like the rest of the `lp` decode +// path: anything that is not a well-formed tagged-bytes object yields empty. +inline std::vector jsonToBytes(const nlohmann::json& j) +{ + if (!j.is_object() || !j.contains("_bytes") || !j["_bytes"].is_string()) + return {}; + return b64UrlDecode(j["_bytes"].get()); +} + +} // namespace logos diff --git a/cpp/logos_lp_client.h b/cpp/logos_lp_client.h index ba67f8a..6e15d20 100644 --- a/cpp/logos_lp_client.h +++ b/cpp/logos_lp_client.h @@ -26,6 +26,7 @@ #include "logos_protocol.h" // lp_* C ABI #include "logos_call_error.h" // logos::CallError +#include "logos_json.h" // logos::bytesToJson / logos::jsonToBytes #include "logos_result.h" // StdLogosResult namespace logos { @@ -41,6 +42,10 @@ inline std::vector jsonToStringVec(const nlohmann::json& j) { return out; } +// Binary payloads travel in the canonical tagged form +// {"_bytes": ""}; logos::bytesToJson / logos::jsonToBytes +// live in logos_json.h and are used by the generated wrappers on both sides. + inline StdLogosResult jsonToStdResult(const nlohmann::json& j) { StdLogosResult r; if (j.is_object()) { diff --git a/doctests/cpp-sdk-generator-roundtrip.test.yaml b/doctests/cpp-sdk-generator-roundtrip.test.yaml index 1c5731e..b954ed0 100644 --- a/doctests/cpp-sdk-generator-roundtrip.test.yaml +++ b/doctests/cpp-sdk-generator-roundtrip.test.yaml @@ -162,6 +162,11 @@ sections: /// Fires when a channel faults. /// Carries an error code, a message, and whether the fault is fatal. void fault(int64_t code, const std::string& message, bool fatal); + + /// Fires with a raw capture buffer — an event carrying a byte + /// string, which the generator must encode rather than pass + /// through as text. + void capture(uint64_t id, const std::vector& frame); }; - title: "Flow 1 — provider header → LIDL" @@ -183,7 +188,7 @@ sections: --output-dir extracted expect_contains: - "Generated LIDL:" - - "9 methods, 3 events" + - "9 methods, 4 events" check_file: "extracted/sensor_module.lidl" - title: "Inspect the extracted contract" text: | @@ -200,6 +205,7 @@ sections: - "method labels(ids: [uint]) -> [tstr]" - "method reset(id: tstr) -> result" - "event fault(code: int, message: tstr, fatal: bool)" + - "event capture(id: uint, frame: bstr)" - "Returns the new enabled state." - "Carries an error code, a message, and whether the fault is fatal." @@ -239,6 +245,25 @@ sections: - "logos_module_get_methods" - "SensorModuleImpl" + - title: "How each event marshals its payload" + text: | + The events file defines the body of every `logos_events:` declaration: + each argument is pushed into a JSON array and handed to the host's emit + callback. Scalars and strings go in as they are — but a byte string + cannot, because JSON has no binary type and a NUL would truncate it. + `frame` is therefore encoded into the canonical tagged form + `{"_bytes": ""}`, the same representation the transport and + the consumer both understand. A generator that pushed the raw value — + or dropped it — would leave every subscriber with an empty payload + ([#99](https://github.com/logos-co/logos-cpp-sdk/issues/99)). + run: "cat provider/sensor_module_events_cdylib.cpp" + code_block: "cat provider/sensor_module_events_cdylib.cpp" + expect_contains: + - "void SensorModuleImpl::reading(uint64_t id, double value)" + - "args.push_back(value);" + - "void SensorModuleImpl::capture(uint64_t id, const std::vector& frame)" + - "args.push_back(lidlBytesToJson(frame));" + - title: "Flow 3 — LIDL → consumer header" step: true text: | diff --git a/doctests/cpp-sdk-module-composition.test.yaml b/doctests/cpp-sdk-module-composition.test.yaml index 049db00..ba9c3ed 100644 --- a/doctests/cpp-sdk-module-composition.test.yaml +++ b/doctests/cpp-sdk-module-composition.test.yaml @@ -13,11 +13,12 @@ intro: | It is fully self-contained — no pre-existing module, no `requires:` chain: 1. Create `greeter_module`, a small **callee** with a couple of methods - (`greet`, `addInts`, `greetCount`) and a `greeted` event. + (`greet`, `addInts`, `greetCount`) and two events — `greeted`, carrying a + string, and `blobReady`, carrying raw bytes. 2. Create `orchestrator_module`, a **caller** that declares `greeter_module` as a dependency and composes it through the generated `modules().greeter_module` wrapper — synchronously, asynchronously, and by - subscribing to its event. + subscribing to both of its events. 3. Build **both** modules' `.lgx` packages **against the C++ SDK commit under test**, so the generated wrappers, the plugin glue, and the IPC layer all come from this SDK. @@ -38,6 +39,7 @@ what_you_learn: - How to build a module — and its module dependency — against a specific `logos-cpp-sdk` commit - How to load two modules in `logoscore` and chain calls so the caller drives the callee - How async replies and event subscriptions survive between `call` commands under the daemon + - How a **binary** (`bstr`) event payload crosses the boundary intact, encoded and decoded by generated code on both sides prerequisites: - | @@ -179,10 +181,21 @@ sections: /// Greets the name and also emits a `greeted` event carrying it. void greetNotify(const std::string& name); + /// Builds a blob of `size` bytes and emits it on `blobReady`. + /// Returns the number of bytes emitted. + int64_t emitBlob(int64_t size); + logos_events: /// Emitted by greetNotify() with the produced greeting string. void greeted(const std::string& greeting); + /// Emitted by emitBlob() carrying raw bytes. `bstr` payloads take + /// the canonical tagged form on the wire; the generated code on + /// both sides encodes and decodes them, so the author on either + /// end only ever sees a `std::vector`. + void blobReady(const std::string& label, + const std::vector& payload); + private: int64_t m_greetCount = 0; }; @@ -219,6 +232,20 @@ sections: greeted("Hello, " + name + "!"); } + int64_t GreeterModuleImpl::emitBlob(int64_t size) + { + // A deterministic blob the subscriber can check byte-for-byte. + // It deliberately contains 0x00 and bytes >= 0x80 — the values a + // text encoding would mangle. + std::vector payload; + payload.reserve(static_cast(size)); + for (int64_t i = 0; i < size; ++i) + payload.push_back(static_cast((i * 7 + 11) & 0xff)); + + blobReady("blob", payload); + return static_cast(payload.size()); + } + - title: "Create the caller: orchestrator_module" step: true text: | @@ -325,6 +352,7 @@ sections: #include #include + #include #include // LogosMap #include // LogosModuleContext base + modules() @@ -361,10 +389,25 @@ sections: /// empty until the event fires. std::string lastGreetedEvent() const; + /// Subscribes to greeter_module's `blobReady` event — the binary + /// one. Returns "ok" once registered. + std::string subscribeBlob(); + + /// How many bytes the `blobReady` subscription actually received, + /// or -1 until the event fires. + int64_t lastBlobSize() const; + + /// A checksum over those bytes — proves the payload arrived + /// intact, not merely with the right length. + int64_t lastBlobChecksum() const; + private: std::string m_asyncGreeting; std::string m_lastGreetedEvent; bool m_subscribed = false; + bool m_blobSubscribed = false; + int64_t m_lastBlobSize = -1; + int64_t m_lastBlobChecksum = -1; }; - title: "src/orchestrator_module_impl.cpp — the implementation" @@ -437,6 +480,37 @@ sections: return m_lastGreetedEvent; } + std::string OrchestratorModuleImpl::subscribeBlob() + { + if (m_blobSubscribed) return "ok"; + // The binary event. The callback takes real bytes: the tagged + // wire form is encoded by the greeter's generated event body and + // decoded by this generated subscriber, so neither author writes + // any base64. + m_blobSubscribed = modules().greeter_module.onBlobReady( + [this](const std::string& label, + const std::vector& payload) { + (void)label; + m_lastBlobSize = static_cast(payload.size()); + int64_t sum = 0; + for (size_t i = 0; i < payload.size(); ++i) + sum += static_cast(payload[i]) + * static_cast(i % 31 + 1); + m_lastBlobChecksum = sum; + }); + return m_blobSubscribed ? "ok" : "failed"; + } + + int64_t OrchestratorModuleImpl::lastBlobSize() const + { + return m_lastBlobSize; + } + + int64_t OrchestratorModuleImpl::lastBlobChecksum() const + { + return m_lastBlobChecksum; + } + - title: "Build both modules against this SDK" step: true text: | @@ -657,6 +731,50 @@ sections: expect_contains: - '"result":"Hello, Events!"' + - title: "Subscribe to the greeter's binary event" + text: | + The same flow, but the event carries a **byte string** (`bstr`) rather + than text. Binary payloads cannot ride in a JSON string — a NUL would + truncate them and any byte above 0x7f would be mangled — so they travel + in the canonical tagged form `{"_bytes": ""}`. Both halves of + that are generated: the greeter's event body encodes, this subscriber + decodes, and neither author writes a line of base64. + run: "./logos/bin/logoscore call orchestrator_module subscribeBlob" + code_block: "logoscore call orchestrator_module subscribeBlob" + expect_contains: + - '"result":"ok"' + + - title: "Emit 4096 bytes from the greeter" + text: "The greeter reports how many bytes it put on the wire." + run: "./logos/bin/logoscore call greeter_module emitBlob 4096" + code_block: "logoscore call greeter_module emitBlob 4096" + expect_contains: + - '"result":4096' + + - run: "sleep 1" + + - title: "The subscriber received every byte" + text: | + `lastBlobSize` is the length the subscription actually saw. This is the + assertion that pins [#99](https://github.com/logos-co/logos-cpp-sdk/issues/99), + where the generator dropped `bstr` event arguments: the greeter emitted + a full payload and the subscriber received `0` bytes. + run: "./logos/bin/logoscore call orchestrator_module lastBlobSize" + code_block: "logoscore call orchestrator_module lastBlobSize" + expect_contains: + - '"result":4096' + + - title: "...and the bytes are the right bytes" + text: | + Length alone would not catch a corrupted payload — a wrong base64 + alphabet or a botched tail group round-trips to the same size. The + checksum is computed over the received bytes and must match the blob the + greeter built. + run: "./logos/bin/logoscore call orchestrator_module lastBlobChecksum" + code_block: "logoscore call orchestrator_module lastBlobChecksum" + expect_contains: + - '"result":8354754' + - title: "Stop the daemon" run: "./logos/bin/logoscore stop" code_block: "logoscore stop" @@ -677,9 +795,19 @@ sections: | Composed sync calls | `greetReport()` → `greet` + `addInts` + `greetCount` | one map of three results | | Typed **async** call | `startAsyncGreet()` → `greetAsync(..., cb)` | `queued`, then `"Hello, Async!"` | | Typed **event** subscription | `subscribeGreeted()` → `onGreeted(cb)` | captured `"Hello, Events!"` | + | **Binary** event payload | `subscribeBlob()` → `onBlobReady(cb)` | all 4096 bytes, checksum intact | Every path went through `modules().greeter_module`, the wrapper the SDK's code generator emitted from the `greeter_module` dependency — and both modules, the wrapper, and the runtime were built against the SDK commit under test. A green run means inter-module composition still works on this SDK, end to end. + + The binary row is the one with teeth. Bytes are the only payload that + cannot ride in a JSON string, so they are the only one that needs an + encoder on the emitting side and a decoder on the receiving side — two + pieces of generated code that must agree exactly. When they did not + ([#99](https://github.com/logos-co/logos-cpp-sdk/issues/99)), everything + above still passed: the module emitted a full payload, the transport + carried it, and the subscriber received zero bytes. Asserting on the + *length and the contents* of what actually arrived is what catches that. diff --git a/tests/experimental/test_lidl_gen_cdylib.cpp b/tests/experimental/test_lidl_gen_cdylib.cpp index 766e3f8..20ab89c 100644 --- a/tests/experimental/test_lidl_gen_cdylib.cpp +++ b/tests/experimental/test_lidl_gen_cdylib.cpp @@ -1,28 +1,147 @@ +// Code-generation tests for the cdylib backend's events sidecar. +// +// The sidecar is a Qt-FREE translation unit, so two classes of defect live +// here: dropping a payload (logos-cpp-sdk#99 — every `bstr` event argument was +// serialized as an empty tagged value), and emitting a Qt type into a TU that +// cannot compile one. +// +// These assert on generated source text. The bytes the emitted encoder actually +// produces are covered by value in tests/sdk/test_logos_json_bytes.cpp. + #include #include "lidl_gen_cdylib.h" +namespace { + +TypeExpr prim(const char* name) +{ + return {TypeExpr::Primitive, name, {}}; +} + +ParamDecl param(const char* name, const TypeExpr& type) +{ + ParamDecl p; + p.name = name; + p.type = type; + return p; +} + +ModuleDecl moduleWithEvent(const char* eventName, const std::vector& params) +{ + ModuleDecl m; + m.name = "delivery_module"; + + EventDecl e; + e.name = eventName; + e.params = params; + m.events.push_back(e); + return m; +} + +QString eventsSourceFor(const ModuleDecl& m) +{ + return lidlMakeEventsSourceCdylib(m, "DeliveryModuleImpl", "delivery_module_plugin.h"); +} + +} // namespace + +// logos-cpp-sdk#99: `payload` was replaced by an empty tagged value, so a module +// could emit real bytes and every consumer still received zero of them. TEST(LidlGenCdylib, BinaryEventPayloadUsesCanonicalBytesEncoding) { - ModuleDecl module; - module.name = "delivery_module"; + const ModuleDecl m = moduleWithEvent("messageReceived", { + param("messageHash", prim("tstr")), + param("contentTopic", prim("tstr")), + param("payload", prim("bstr")), + param("timestamp", prim("int")), + }); - EventDecl event; - event.name = "messageReceived"; - - ParamDecl payload; - payload.name = "payload"; - payload.type = {TypeExpr::Primitive, "bstr", {}}; - event.params.push_back(payload); - module.events.push_back(event); - - const QString source = lidlMakeEventsSourceCdylib( - module, - "DeliveryModuleImpl", - "delivery_module_plugin.h"); + const QString source = eventsSourceFor(m); + // The real argument is serialized, through the canonical encoder... EXPECT_TRUE(source.contains("args.push_back(lidlBytesToJson(payload));")); EXPECT_TRUE(source.contains("std::string lidlB64UrlEncode")); EXPECT_TRUE(source.contains("nlohmann::json lidlBytesToJson")); + + // ...and the empty tagged value is gone. EXPECT_FALSE(source.contains("nlohmann::json{{\"_bytes\", \"\"}}")); + + // The other parameters are still passed straight through. + EXPECT_TRUE(source.contains("args.push_back(messageHash);")); + EXPECT_TRUE(source.contains("args.push_back(timestamp);")); + + // Bytes are taken by const-ref, matching the author's logos_events: block. + EXPECT_TRUE(source.contains("const std::vector& payload")); +} + +// The encoder is only needed by modules that actually emit binary payloads. +// Emitted unconditionally it is an unused static function in every other +// module's sidecar (-Wunused-function). +TEST(LidlGenCdylib, BytesEncoderOmittedWhenNoEventCarriesBytes) +{ + const ModuleDecl m = moduleWithEvent("fault", { + param("code", prim("int")), + param("message", prim("tstr")), + param("fatal", prim("bool")), + }); + + const QString source = eventsSourceFor(m); + + EXPECT_FALSE(source.contains("lidlB64UrlEncode")); + EXPECT_FALSE(source.contains("lidlBytesToJson")); + EXPECT_TRUE(source.contains("args.push_back(code);")); +} + +// The sidecar is compiled into the module's Qt-free cdylib, so a JSON payload +// has to be spelled as its nlohmann alias. Emitted as QVariantMap it does not +// compile at all. +TEST(LidlGenCdylib, JsonEventPayloadIsQtFree) +{ + ModuleDecl m; + m.name = "state_module"; + + EventDecl e; + e.name = "stateChanged"; + e.params.push_back(param("key", prim("tstr"))); + e.params.push_back(param("state", + TypeExpr{TypeExpr::Map, "", {prim("tstr"), prim("any")}})); + m.events.push_back(e); + + const QString source = + lidlMakeEventsSourceCdylib(m, "StateModuleImpl", "state_module_plugin.h"); + + EXPECT_TRUE(source.contains("const LogosMap& state")); + EXPECT_TRUE(source.contains("#include ")); + + // No Qt type may appear anywhere in a Qt-free TU. + EXPECT_FALSE(source.contains("QVariant")); +} + +// An array of byte strings is outside the cdylib-supported subset. It must be +// rejected by name at generation time rather than admitted and then emitted as +// a QVariant that fails to compile. +TEST(LidlGenCdylib, ArrayOfBytesEventParamIsRejected) +{ + const ModuleDecl m = moduleWithEvent("batchReceived", { + param("payloads", TypeExpr{TypeExpr::Array, "", {prim("bstr")}}), + }); + + QString error; + EXPECT_FALSE(lidlCdylibSupported(m, &error)); + EXPECT_TRUE(error.contains("batchReceived")); + EXPECT_TRUE(error.contains("payloads")); +} + +// The supported scalar / bytes payloads stay eligible. +TEST(LidlGenCdylib, SupportedEventParamsRemainEligible) +{ + const ModuleDecl m = moduleWithEvent("messageReceived", { + param("messageHash", prim("tstr")), + param("payload", prim("bstr")), + param("timestamp", prim("int")), + }); + + QString error; + EXPECT_TRUE(lidlCdylibSupported(m, &error)) << error.toStdString(); } diff --git a/tests/sdk/CMakeLists.txt b/tests/sdk/CMakeLists.txt index 642fadf..76944e8 100644 --- a/tests/sdk/CMakeLists.txt +++ b/tests/sdk/CMakeLists.txt @@ -6,6 +6,7 @@ add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/../../cpp ${CMAKE_CURRENT_BINARY_DI add_executable(sdk_tests test_logos_module_context.cpp + test_logos_json_bytes.cpp ) target_link_libraries(sdk_tests PRIVATE diff --git a/tests/sdk/test_logos_json_bytes.cpp b/tests/sdk/test_logos_json_bytes.cpp new file mode 100644 index 0000000..5935d12 --- /dev/null +++ b/tests/sdk/test_logos_json_bytes.cpp @@ -0,0 +1,115 @@ +// Value-level tests for the canonical tagged-bytes codec (logos_json.h). +// +// Binary payloads cross every module boundary as {"_bytes": ""}. +// The generated cdylib event sidecar encodes into that form and the generated +// `lp` consumer wrappers decode out of it, so a wrong alphabet, a stray '=', +// or a botched tail group silently corrupts every binary event in the system. +// The code-generation tests assert on generated *source text* and cannot catch +// that; these assert on actual bytes. + +#include + +#include + +#include +#include +#include + +namespace { + +std::vector bytesOf(const std::string& s) +{ + return std::vector(s.begin(), s.end()); +} + +} // namespace + +// RFC 4648 §10 test vectors, in the URL-safe alphabet without padding — the +// form logos-protocol emits (Base64UrlEncoding | OmitTrailingEquals). +TEST(LogosJsonBytes, EncodesRfc4648VectorsUnpadded) +{ + EXPECT_EQ(logos::b64UrlEncode(bytesOf("")), ""); + EXPECT_EQ(logos::b64UrlEncode(bytesOf("f")), "Zg"); + EXPECT_EQ(logos::b64UrlEncode(bytesOf("fo")), "Zm8"); + EXPECT_EQ(logos::b64UrlEncode(bytesOf("foo")), "Zm9v"); + EXPECT_EQ(logos::b64UrlEncode(bytesOf("foob")), "Zm9vYg"); + EXPECT_EQ(logos::b64UrlEncode(bytesOf("fooba")), "Zm9vYmE"); + EXPECT_EQ(logos::b64UrlEncode(bytesOf("foobar")), "Zm9vYmFy"); +} + +// The URL-safe alphabet uses '-' and '_' where standard base64 uses '+' and '/'. +// Getting this wrong still round-trips through our own decoder but corrupts +// every payload crossing to the Qt side, which decodes with Base64UrlEncoding. +TEST(LogosJsonBytes, UsesTheUrlSafeAlphabet) +{ + const std::vector bytes = {0xfb, 0xef, 0xbe}; // four sextets of 62 + const std::string enc = logos::b64UrlEncode(bytes); // "++++" in standard b64 + EXPECT_EQ(enc, "----"); + EXPECT_EQ(enc.find('+'), std::string::npos); + EXPECT_EQ(enc.find('/'), std::string::npos); + EXPECT_EQ(enc.find('='), std::string::npos); +} + +TEST(LogosJsonBytes, RoundTripsEveryTailLength) +{ + // Every length 0..64 covers all three len%3 tail groups repeatedly. + for (size_t n = 0; n <= 64; ++n) { + std::vector in(n); + for (size_t i = 0; i < n; ++i) + in[i] = static_cast((i * 7 + 11) & 0xff); + + const nlohmann::json tagged = logos::bytesToJson(in); + ASSERT_TRUE(tagged.is_object()); + ASSERT_TRUE(tagged.contains("_bytes")); + EXPECT_EQ(logos::jsonToBytes(tagged), in) << "length " << n; + } +} + +// The reason the tagged form exists at all: a plain JSON string would truncate +// at the first NUL and mangle anything >= 0x80. +TEST(LogosJsonBytes, SurvivesEmbeddedNulsAndHighBytes) +{ + const std::vector in = {0x00, 0xff, 0x00, 0x80, 0x7f, 0x00, 0xfe, 0xc3, 0x28}; + EXPECT_EQ(logos::jsonToBytes(logos::bytesToJson(in)), in); +} + +// A large payload — the shape of the proof blobs that surfaced this bug +// (logos-cpp-sdk#99 reported a 109,447-byte payload arriving empty). +TEST(LogosJsonBytes, RoundTripsALargePayload) +{ + std::vector in(109447); + for (size_t i = 0; i < in.size(); ++i) + in[i] = static_cast((i * 31 + 7) & 0xff); + + const std::vector out = logos::jsonToBytes(logos::bytesToJson(in)); + ASSERT_EQ(out.size(), in.size()); + EXPECT_EQ(out, in); +} + +// The empty payload must round-trip as empty — and, critically, must be +// distinguishable from the bug it masked: an event that dropped its bytes used +// to arrive as exactly this value. +TEST(LogosJsonBytes, EmptyPayloadRoundTrips) +{ + const nlohmann::json tagged = logos::bytesToJson({}); + EXPECT_EQ(tagged, nlohmann::json({{"_bytes", ""}})); + EXPECT_TRUE(logos::jsonToBytes(tagged).empty()); +} + +// Decoding is lenient in the same way the rest of the `lp` decode path is: +// a malformed value yields empty rather than throwing across the C ABI. +TEST(LogosJsonBytes, DecodeIsLenientOnMalformedInput) +{ + EXPECT_TRUE(logos::jsonToBytes(nlohmann::json()).empty()); + EXPECT_TRUE(logos::jsonToBytes(nlohmann::json("plain string")).empty()); + EXPECT_TRUE(logos::jsonToBytes(nlohmann::json::array({1, 2, 3})).empty()); + EXPECT_TRUE(logos::jsonToBytes(nlohmann::json{{"other", "key"}}).empty()); + EXPECT_TRUE(logos::jsonToBytes(nlohmann::json{{"_bytes", 42}}).empty()); +} + +// Padded input is not what we emit, but a peer that pads must still decode: +// '=' is skipped rather than treated as data. +TEST(LogosJsonBytes, DecodeAcceptsPaddedInput) +{ + EXPECT_EQ(logos::jsonToBytes(nlohmann::json{{"_bytes", "Zm9vYg=="}}), bytesOf("foob")); +}