Files
logos-protocol/cpp/logos_codec.h
T
Dario LipicarandClaude Opus 5 362b03fb1e feat(codec): one canonical LIDL ↔ JSON codec, generic over composition (#29)
* feat(codec): one canonical LIDL <-> JSON codec, generic over composition

The tagged-bytes encoding {"_bytes": "<base64url, unpadded>"} was implemented
SIX times — the Qt conversion here, the plain wire's json_mapping, the lp helper
in logos-cpp-sdk, a copy emitted into every generated cdylib module, the Rust
SDK and the Python client — and they disagreed on which inputs they accept:

  - {"_bytes":"AA","x":1} decoded as BYTES on the lp path (no size()==1 check)
    but as a MAP on the plain wire and in the glue.
  - Padded "AH-A_w==" gave correct bytes in one copy, empty in another, None in
    Rust.
  - A plain string / number / number-array argument was accepted by C++
    providers (Qt and CLI parity) and rejected by Rust ones.

logos_codec.h is the single implementation. Leaves: tstr, bstr, every signed and
unsigned integral spelling, every floating spelling, bool, any (recursion stops).
Composition is GENERIC — std::vector<T> and std::map/unordered_map<std::string,T>
for any supported T, at any depth — so [bstr], [[bstr]], {tstr: [bstr]} and bytes
nested in a map all encode canonically without anything enumerating combinations.

Codec<T> is a trait, so an unsupported T is an incomplete type: a compile error
naming the type, never a silent fallback. Decode throws CodecError carrying the
path ("[0][1]", ".k") instead of substituting a default — a mangled value must
not reach business logic. bstr keeps a documented lenient form for provider-side
arguments, because the Qt consumer path and the logoscore CLI both produce plain
strings and number arrays for byte parameters.

JsonArg exists for generated dispatch: it converts itself into whatever the
callee's parameter type is. Naming the type instead is a trap — spelling [uint]
as std::vector<uint64_t> (the LIDL mapping) does not bind to an author's
std::vector<uint32_t>, since distinct vector instantiations do not convert.

logos_codec.h joins the installed header set; nix/include.nix already globs
cpp/*.h.

Tests: 198/198. 15 new ones pin the contract rather than the happy path —
[[bstr]] tagged at depth, map-of-bytes, empty elements surviving as elements,
uint64 past 2^63, an integral JSON number decoding as float64, padded base64,
the multi-key {"_bytes":...} case being a map, and path-carrying failures.

Not yet converged onto this header (follow-ups): the Qt conversion in
logos_json_convert.cpp, and the plain wire's copy in json_mapping.cpp — the
latter needs a strict variant first, because it THROWS on malformed base64
(via its own logos::plain::CodecError) where every other copy is tolerant.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* refactor(codec): fold the Qt and plain-wire copies into the shared codec

The two remaining in-repo implementations now delegate:

  - logos_json_convert.cpp (the Qt CONSUMER path — argument encoding and return
    decoding) dropped Qt's toBase64/fromBase64 and its own tagged-bytes
    predicate. Only the QByteArray <-> std::vector<uint8_t> hop stays local, so
    the Qt path cannot drift from the wire or from providers: same alphabet, same
    padding rule, same single-key shape.
  - implementations/plain/json_mapping.cpp dropped its anonymous-namespace
    b64url_encode/decode.

The wire needed something the tolerant decode does not give it: it REJECTS a
corrupt frame rather than silently decoding fewer bytes. Hence
b64UrlDecodeChecked — strict about the alphabet and the length, tolerant of '='
padding — which json_mapping uses to keep throwing its own
logos::plain::CodecError. Consumer-facing decodes stay tolerant. Both behaviours
now come from one implementation instead of four that disagreed.

Also removed the local isTaggedBytes wrapper, which shadowed the shared one and
made unqualified calls ambiguous.

Tests: 199/199, with the strict decode's accept/reject set pinned (padding
tolerated, stray character rejected, impossible length rejected).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 15:01:49 -03:00

401 lines
15 KiB
C++

#ifndef LOGOS_CODEC_H
#define LOGOS_CODEC_H
// ---------------------------------------------------------------------------
// logos_codec.h — THE canonical JSON representation of LIDL values.
//
// One implementation, here. Before this header the same encoding lived in six
// places (the Qt conversion, the plain wire's json_mapping, the lp helper in
// logos-cpp-sdk, a copy emitted into every generated cdylib module, the Rust
// SDK and the Python client) and they disagreed on which inputs they accepted.
// Anything that needs to move a LIDL value in or out of JSON uses these.
//
// LEAF TYPES
// tstr std::string -> json string
// bstr std::vector<uint8_t> -> {"_bytes": "<base64url, unpadded>"}
// int any signed integral -> json integer
// uint any unsigned integral -> json integer
// float64 float / double -> json number
// bool bool -> json bool
// any nlohmann::json (LogosMap/List) -> verbatim; recursion STOPS here
//
// COMPOSITION — generic, no whitelist
// [T] std::vector<T> for any supported T
// {tstr: T} std::map / std::unordered_map<std::string,T> for any supported T
// ...at any depth. Bytes stay tagged wherever they occur, so [bstr], [[bstr]],
// {tstr: [bstr]} and bytes nested inside a map all encode canonically without
// anything having to enumerate the combination.
//
// WHY BYTES ARE TAGGED: JSON has no bytes primitive and a JSON string must be
// valid UTF-8, so raw binary (embedded NULs, anything >= 0x80) cannot ride a
// plain string. The single-key object is the wire form every language agrees on.
// Note the inherent ambiguity: a genuine one-key map named "_bytes" whose value
// is a string is indistinguishable from bytes. Don't name a map key "_bytes".
//
// DECODE STRICTNESS: shape mismatches throw CodecError, which callers surface as
// a structured error (the generated dispatch turns it into
// {"code":"dispatch_failed",...}) rather than silently substituting a default —
// silent defaults are how a mangled value reaches business logic. `bstr` alone
// keeps a documented lenient form (see bytesFromJsonLenient) because the Qt
// consumer path and the logoscore CLI's argument auto-typing both produce plain
// strings and number arrays for byte parameters.
// ---------------------------------------------------------------------------
#include <nlohmann/json.hpp>
#include <cstdint>
#include <map>
#include <stdexcept>
#include <string>
#include <type_traits>
#include <unordered_map>
#include <utility>
#include <vector>
namespace logos {
// Thrown by fromJson<T>() when the value does not match the expected LIDL type.
// `what()` carries the path to the offending element ("[2].payload") so a
// failure inside a nested container is actionable.
class CodecError : public std::runtime_error {
public:
explicit CodecError(const std::string& message)
: std::runtime_error(message) {}
};
namespace detail {
inline std::string joinPath(const std::string& path, const std::string& step)
{
return path.empty() ? step : path + step;
}
[[noreturn]] inline void typeError(const std::string& path, const char* expected,
const nlohmann::json& got)
{
throw CodecError("expected " + std::string(expected)
+ (path.empty() ? std::string(" at value") : " at " + path)
+ ", got " + std::string(got.type_name()));
}
} // namespace detail
// ── base64url, unpadded ────────────────────────────────────────────────────
inline std::string b64UrlEncode(const std::vector<uint8_t>& bytes)
{
static const char* alpha =
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_";
std::string out;
out.reserve((bytes.size() + 2) / 3 * 4);
size_t i = 0;
while (i + 3 <= bytes.size()) {
const uint32_t n = (uint32_t(bytes[i]) << 16) | (uint32_t(bytes[i + 1]) << 8)
| uint32_t(bytes[i + 2]);
out += alpha[(n >> 18) & 0x3f];
out += alpha[(n >> 12) & 0x3f];
out += alpha[(n >> 6) & 0x3f];
out += alpha[n & 0x3f];
i += 3;
}
if (i < bytes.size()) {
uint32_t n = uint32_t(bytes[i]) << 16;
if (i + 1 < bytes.size()) n |= uint32_t(bytes[i + 1]) << 8;
out += alpha[(n >> 18) & 0x3f];
out += alpha[(n >> 12) & 0x3f];
if (i + 1 < bytes.size()) out += alpha[(n >> 6) & 0x3f];
}
return out;
}
// Decodes unpadded base64url. Deliberately tolerant: '=' padding and stray
// characters (whitespace from a hand-written spec, a padded encoder on the
// other side) are skipped rather than aborting the whole value — one of the
// four behaviours the old copies disagreed on.
inline std::vector<uint8_t> b64UrlDecode(const std::string& in)
{
auto idx = [](char ch) -> int {
if (ch >= 'A' && ch <= 'Z') return ch - 'A';
if (ch >= 'a' && ch <= 'z') return ch - 'a' + 26;
if (ch >= '0' && ch <= '9') return ch - '0' + 52;
if (ch == '-') return 62;
if (ch == '_') return 63;
return -1;
};
std::vector<uint8_t> out;
out.reserve(in.size() * 3 / 4);
uint32_t buf = 0;
int bits = 0;
for (char ch : in) {
const int v = idx(ch);
if (v < 0) continue;
buf = (buf << 6) | static_cast<uint32_t>(v);
bits += 6;
if (bits >= 8) {
bits -= 8;
out.push_back(static_cast<uint8_t>((buf >> bits) & 0xff));
}
}
return out;
}
// Strict variant: returns false on any character outside the alphabet (padding
// aside) or an impossible length, leaving `out` empty. The plain wire validates
// its frames with this — a corrupt frame must be rejected, not silently decoded
// to fewer bytes. Consumer-facing decodes use the tolerant one above.
inline bool b64UrlDecodeChecked(const std::string& in, std::vector<uint8_t>& out)
{
auto idx = [](char ch) -> int {
if (ch >= 'A' && ch <= 'Z') return ch - 'A';
if (ch >= 'a' && ch <= 'z') return ch - 'a' + 26;
if (ch >= '0' && ch <= '9') return ch - '0' + 52;
if (ch == '-') return 62;
if (ch == '_') return 63;
return -1;
};
out.clear();
std::string body = in;
while (!body.empty() && body.back() == '=') body.pop_back();
if (body.size() % 4 == 1) return false;
uint32_t buf = 0;
int bits = 0;
for (char ch : body) {
const int v = idx(ch);
if (v < 0) {
out.clear();
return false;
}
buf = (buf << 6) | static_cast<uint32_t>(v);
bits += 6;
if (bits >= 8) {
bits -= 8;
out.push_back(static_cast<uint8_t>((buf >> bits) & 0xff));
}
}
return true;
}
// ── tagged bytes ───────────────────────────────────────────────────────────
inline nlohmann::json bytesToJson(const std::vector<uint8_t>& bytes)
{
return nlohmann::json{{"_bytes", b64UrlEncode(bytes)}};
}
// The canonical form: EXACTLY one key, "_bytes", holding a string. The size
// check matters — without it a map that merely contains a "_bytes" entry is
// read as bytes, which is how the lp helper and the plain wire disagreed.
inline bool isTaggedBytes(const nlohmann::json& j)
{
return j.is_object() && j.size() == 1 && j.contains("_bytes")
&& j["_bytes"].is_string();
}
inline std::vector<uint8_t> bytesFromJson(const nlohmann::json& j)
{
if (!isTaggedBytes(j))
detail::typeError("", "tagged bytes {\"_bytes\": \"...\"}", j);
return b64UrlDecode(j["_bytes"].get<std::string>());
}
// Canonical form, plus the shapes other layers legitimately produce for a byte
// parameter:
// string -> its raw bytes (a Qt consumer passing a QString, a CLI arg)
// number -> its decimal text as bytes (QVariant(int)->QByteArray parity)
// array of ints -> those byte values
// Providers decode arguments with this; everything else uses the canonical form.
inline std::vector<uint8_t> bytesFromJsonLenient(const nlohmann::json& j)
{
if (isTaggedBytes(j))
return b64UrlDecode(j["_bytes"].get<std::string>());
if (j.is_string()) {
const std::string s = j.get<std::string>();
return std::vector<uint8_t>(s.begin(), s.end());
}
if (j.is_number()) {
const std::string s = j.dump();
return std::vector<uint8_t>(s.begin(), s.end());
}
if (j.is_array()) {
std::vector<uint8_t> out;
out.reserve(j.size());
for (const auto& e : j)
if (e.is_number_integer() || e.is_number_unsigned())
out.push_back(static_cast<uint8_t>(e.get<int64_t>() & 0xff));
return out;
}
detail::typeError("", "bytes", j);
}
// ── the generic value codec ────────────────────────────────────────────────
//
// Codec<T> is a trait rather than a function template because C++ has no
// partial specialisation of function templates, and the composition rule
// (vector<T>, map<string,T>) is inherently partial. An unsupported T leaves
// Codec<T> incomplete, so the failure is a compile error at the call site
// naming the type — never a silent fallback.
namespace detail {
template <class T, class Enable = void> struct Codec;
// bstr — a FULL specialisation, so it wins over the generic vector<T> below.
template <> struct Codec<std::vector<uint8_t>, void> {
static nlohmann::json to(const std::vector<uint8_t>& v) { return bytesToJson(v); }
static std::vector<uint8_t> from(const nlohmann::json& j, const std::string&)
{
return bytesFromJsonLenient(j);
}
};
template <> struct Codec<bool, void> {
static nlohmann::json to(bool v) { return v; }
static bool from(const nlohmann::json& j, const std::string& path)
{
if (!j.is_boolean()) typeError(path, "bool", j);
return j.get<bool>();
}
};
template <> struct Codec<std::string, void> {
static nlohmann::json to(const std::string& v) { return v; }
static std::string from(const nlohmann::json& j, const std::string& path)
{
if (!j.is_string()) typeError(path, "string", j);
return j.get<std::string>();
}
};
// `any` stops the recursion: the value passes through untouched, so a LogosMap
// or LogosList keeps whatever the peer sent (including tagged bytes nested
// inside it, which the author decodes with bytesFromJson).
template <> struct Codec<nlohmann::json, void> {
static nlohmann::json to(const nlohmann::json& v) { return v; }
static nlohmann::json from(const nlohmann::json& j, const std::string&) { return j; }
};
// int / uint — every C++ integral spelling, bool excluded (specialised above).
template <class T>
struct Codec<T, std::enable_if_t<std::is_integral_v<T> && !std::is_same_v<T, bool>>> {
static nlohmann::json to(T v) { return v; }
static T from(const nlohmann::json& j, const std::string& path)
{
if (!j.is_number_integer() && !j.is_number_unsigned())
typeError(path, "integer", j);
return j.get<T>();
}
};
// float64 — an integral JSON number is accepted (2 and 2.0 are the same value
// to JSON, and every encoder that sees a whole double may emit either).
template <class T>
struct Codec<T, std::enable_if_t<std::is_floating_point_v<T>>> {
static nlohmann::json to(T v) { return v; }
static T from(const nlohmann::json& j, const std::string& path)
{
if (!j.is_number()) typeError(path, "number", j);
return j.get<T>();
}
};
// [T] — generic over the element type, so nesting composes.
template <class T> struct Codec<std::vector<T>, void> {
static nlohmann::json to(const std::vector<T>& v)
{
nlohmann::json out = nlohmann::json::array();
for (const T& e : v) out.push_back(Codec<T>::to(e));
return out;
}
static std::vector<T> from(const nlohmann::json& j, const std::string& path)
{
if (!j.is_array()) typeError(path, "array", j);
std::vector<T> out;
out.reserve(j.size());
for (size_t i = 0; i < j.size(); ++i)
out.push_back(Codec<T>::from(j[i], joinPath(path, "[" + std::to_string(i) + "]")));
return out;
}
};
// {tstr: T} — both standard associative containers, same rule.
template <class M> struct MapCodec {
using T = typename M::mapped_type;
static nlohmann::json to(const M& v)
{
nlohmann::json out = nlohmann::json::object();
for (const auto& kv : v) out[kv.first] = Codec<T>::to(kv.second);
return out;
}
static M from(const nlohmann::json& j, const std::string& path)
{
if (!j.is_object()) typeError(path, "object", j);
M out;
for (auto it = j.begin(); it != j.end(); ++it)
out.emplace(it.key(), Codec<T>::from(it.value(), joinPath(path, "." + it.key())));
return out;
}
};
template <class T> struct Codec<std::map<std::string, T>, void>
: MapCodec<std::map<std::string, T>> {};
template <class T> struct Codec<std::unordered_map<std::string, T>, void>
: MapCodec<std::unordered_map<std::string, T>> {};
} // namespace detail
// Encode a LIDL value of static type T into its canonical JSON form.
template <class T>
nlohmann::json toJson(const T& value)
{
return detail::Codec<T>::to(value);
}
// Decode canonical JSON into a LIDL value of static type T.
// Throws CodecError on a shape mismatch, naming the path.
template <class T>
T fromJson(const nlohmann::json& j)
{
return detail::Codec<T>::from(j, std::string());
}
// Same, with a path prefix so a failure inside a nested value reports where it
// came from ("arg2[0].payload" rather than "[0].payload").
template <class T>
T fromJson(const nlohmann::json& j, const std::string& path)
{
return detail::Codec<T>::from(j, path);
}
// A JSON value that decodes itself into whatever the callee's parameter type is.
//
// Generated dispatch code has the author's signature available only as an
// overload-resolution target, not as text it can name — and naming it is a trap:
// spelling `[uint]` as std::vector<uint64_t> (the LIDL mapping) does not bind to
// an author's `const std::vector<uint32_t>&`, because distinct vector
// instantiations do not convert. Passing this proxy instead makes the compiler
// instantiate the conversion with the EXACT parameter type, so the author's own
// spelling is what gets decoded — any integer width, any supported nesting.
//
// An unsupported target type leaves Codec<T> incomplete, so the failure is a
// compile error naming the type at the call site, never a silent fallback.
class JsonArg {
public:
JsonArg(const nlohmann::json& value, std::string path)
: m_value(value), m_path(std::move(path)) {}
template <class T>
operator T() const // NOLINT(google-explicit-constructor) — that is the point
{
return detail::Codec<std::decay_t<T>>::from(m_value, m_path);
}
private:
const nlohmann::json& m_value;
std::string m_path;
};
} // namespace logos
#endif // LOGOS_CODEC_H