logos-blockchain/lez-indexer-module
1
0
Fork 0
mirror of https://github.com/logos-blockchain/lez-indexer-module.git synced 2026-07-29 22:53:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #11 from logos-blockchain/erhant/migr-to-logos-module-builder-update-LEZ

Browse Source 
refactor!: adopt reworked indexer FFI (drop port, pointer frees, init_logger)
This commit is contained in:
erhant 2026-06-23 18:52:58 +03:00 committed by GitHub
commit 62e0f0baed
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
9 changed files with 6827 additions and 1469 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

41
README.md
View File

@ -3,7 +3,7 @@
A Logos Core **service module** (`type: core`, universal authoring model) that runs the Logos Execution Zone (L2) indexer and exposes it to the Logos ecosystem. It is a thin Qt-free plugin around the `indexer_ffi` library from
[`logos-execution-zone`](https://github.com/logos-blockchain/logos-execution-zone): it starts the indexer, which connects to an L1/bedrock node and indexes the zone's channel, and exposes **query methods over the Logos protocol** (Qt Remote Objects).
Registered module name: **`lez_indexer_module`**. Its public methods *are* its API — other modules call them in-process over the Logos protocol. It pairs with the
Registered module name: **`lez_indexer_module`**. Its public methods _are_ its API — other modules call them in-process over the Logos protocol. It pairs with the
[`lez-explorer-ui`](https://github.com/logos-co/lez-explorer-ui) block explorer, which reads from it **in-process over the Logos protocol** (typed `modules().lez_indexer_module.*` calls) — no RPC endpoint or socket in between.
> [!TIP]
@ -30,32 +30,36 @@ This will reduce friction when working on the project.
The module does **not** start the indexer on load — something must invoke its `start_indexer` method (via the module-viewer's invoke panel, or another module / basecamp over the Logos protocol):
```c
start_indexer(config_path, port)
start_indexer(config_path)
```
- `config_path` — **absolute** path to a JSON config (see
[`config/indexer_config.json`](config/indexer_config.json)). It must be absolute: the module runs inside the `logos_host` subprocess, whose working directory is not your shell's.
- `port` — **legacy** parameter, still required by the pinned `indexer_ffi`. Logos-protocol consumers do not use it; pass any value as a **string** (e.g. `"8779"`). It is slated for removal once the FFI drops it.
On success it returns `0`; a non-zero return is the FFI `OperationStatus` (e.g. `2 = InitializationError`). Once started, consume the indexer through the query methods below.
> [!CAUTION]
> [!TIP]
>
> The FFI does not log, so a non-zero `OperationStatus` code is all you get on failure.
> By default the indexer is silent. Call `init_logger(level)` once (e.g.
> `init_logger("info")`; accepts `off`/`error`/`warn`/`info`/`debug`/`trace`) to
> surface the indexer's own `log` output — useful since a failed call otherwise
> only reports a numeric `OperationStatus`. Logging is scoped to the indexer
> crates, and the first call wins.
### Query methods (the Logos-protocol API)
Each returns a compact JSON string (an **empty** string means not-found / failed query); all ids/hashes are 32-byte hex, all numeric args/ids are decimal strings.
| Method | Returns |
| --- | --- |
| `getLastFinalizedBlockId()` | tip block id (bare decimal string) |
| `getBlockById(block_id)` | block JSON |
| `getBlockByHash(hash)` | block JSON |
| `getBlocks(before, limit)` | JSON array of blocks; `before` = `""` for the tip, else a block id to page back from |
| `getTransaction(hash)` | transaction JSON |
| `getAccount(account_id)` | account JSON (the payload omits the id; callers inject the queried id) |
| `getTransactionsByAccount(account_id, offset, limit)` | JSON array of transactions touching the account |
| Method | Returns |
| ----------------------------------------------------- | ------------------------------------------------------------------------------------ |
| `getStatus()` | indexer status JSON (schema owned by `indexer_core`) |
| `getLastFinalizedBlockId()` | tip block id (bare decimal string) |
| `getBlockById(block_id)` | block JSON |
| `getBlockByHash(hash)` | block JSON |
| `getBlocks(before, limit)` | JSON array of blocks; `before` = `""` for the tip, else a block id to page back from |
| `getTransaction(hash)` | transaction JSON |
| `getAccount(account_id)` | account JSON (the payload omits the id; callers inject the queried id) |
| `getTransactionsByAccount(account_id, offset, limit)` | JSON array of transactions touching the account |
Because the module uses the universal authoring model (`interface: "universal"`), it publishes a typed LIDL contract, so universal consumers get Qt-typed wrappers (`modules().lez_indexer_module.getBlockById(...)`) rather than dynamic by-name calls.
@ -63,10 +67,9 @@ Because the module uses the universal authoring model (`interface: "universal"`)
`config/indexer_config.json` is deserialized into the indexer's `IndexerConfig`. Key fields:
| Field | Meaning | Default |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- |
| `bedrock_config.addr` | L1/bedrock node URL the indexer reads from; it must be reachable. | `http://localhost:8080` |
| `home` | Directory for the indexer's RocksDB state, relative to the host's working directory. Use an absolute path for a predictable location. | `"."` |
| `channel_id` | The zone channel the indexer consumes; must match what the sequencer inscribes. | |
| Field | Meaning | Default |
| --------------------- | ------------------------------------------------------------------------------- | ----------------------- |
| `bedrock_config.addr` | L1/bedrock node URL the indexer reads from; it must be reachable. | `http://localhost:8080` |
| `channel_id` | The zone channel the indexer consumes; must match what the sequencer inscribes. | |
The config keys must match the `IndexerConfig` schema of the `logos-execution-zone` rev pinned in `flake.nix`/`flake.lock`; bumping that rev may require re-syncing this file. Unknown keys are ignored.

209
config/indexer_config.json
View File

@ -1,160 +1,53 @@
{
"home": ".",
"consensus_info_polling_interval": "1s",
"bedrock_config": {
"addr": "http://localhost:8080",
"backoff": {
"start_delay": "100ms",
"max_retries": 5
}
"consensus_info_polling_interval": "1s",
"bedrock_config": {
"addr": "http://localhost:8080",
"backoff": {
"start_delay": "100ms",
"max_retries": 5
}
},
"channel_id": "0101010101010101010101010101010101010101010101010101010101010101",
"initial_accounts": [
{
"account_id": "CbgR6tj5kWx5oziiFptM7jMvrQeYY3Mzaao6ciuhSr2r",
"balance": 10000
},
"channel_id": "0101010101010101010101010101010101010101010101010101010101010101",
"initial_accounts": [
{
"account_id": "CbgR6tj5kWx5oziiFptM7jMvrQeYY3Mzaao6ciuhSr2r",
"balance": 10000
},
{
"account_id": "2RHZhw9h534Zr3eq2RGhQete2Hh667foECzXPmSkGni2",
"balance": 20000
}
],
"initial_commitments": [
{
"npk": [
139,
19,
158,
11,
155,
231,
85,
206,
132,
228,
220,
114,
145,
89,
113,
156,
238,
142,
242,
74,
182,
91,
43,
100,
6,
190,
31,
15,
31,
88,
96,
204
],
"account": {
"program_owner": [
0,
0,
0,
0,
0,
0,
0,
0
],
"balance": 10000,
"data": [],
"nonce": 0
}
},
{
"npk": [
173,
134,
33,
223,
54,
226,
10,
71,
215,
254,
143,
172,
24,
244,
243,
208,
65,
112,
118,
70,
217,
240,
69,
100,
129,
3,
121,
25,
213,
132,
42,
45
],
"account": {
"program_owner": [
0,
0,
0,
0,
0,
0,
0,
0
],
"balance": 20000,
"data": [],
"nonce": 0
}
}
],
"signing_key": [
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37,
37
]
}
{
"account_id": "2RHZhw9h534Zr3eq2RGhQete2Hh667foECzXPmSkGni2",
"balance": 20000
}
],
"initial_commitments": [
{
"npk": [
139, 19, 158, 11, 155, 231, 85, 206, 132, 228, 220, 114, 145, 89, 113,
156, 238, 142, 242, 74, 182, 91, 43, 100, 6, 190, 31, 15, 31, 88, 96,
204
],
"account": {
"program_owner": [0, 0, 0, 0, 0, 0, 0, 0],
"balance": 10000,
"data": [],
"nonce": 0
}
},
{
"npk": [
173, 134, 33, 223, 54, 226, 10, 71, 215, 254, 143, 172, 24, 244, 243,
208, 65, 112, 118, 70, 217, 240, 69, 100, 129, 3, 121, 25, 213, 132, 42,
45
],
"account": {
"program_owner": [0, 0, 0, 0, 0, 0, 0, 0],
"balance": 20000,
"data": [],
"nonce": 0
}
}
],
"signing_key": [
37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37,
37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37
]
}

7890
flake.lock generated
View File

File diff suppressed because it is too large Load Diff

2
flake.nix
View File

@ -7,7 +7,7 @@
# The LEZ indexer Rust FFI lib + header come from this flake's `indexer`
# package output (/lib/libindexer_ffi.* + /include/indexer_ffi.h). It is a
# prebuilt Nix derivation, so mkExternalLib consumes it directly (no build).
logos-execution-zone.url = "github:logos-blockchain/logos-execution-zone?ref=main";
logos-execution-zone.url = "git+https://github.com/logos-blockchain/logos-execution-zone?ref=main";
};
outputs =

4
metadata.json
View File

@ -8,9 +8,7 @@
"main": "lez_indexer_module_plugin",
"dependencies": [],
"nix": {
"external_libraries": [
{ "name": "indexer_ffi" }
],
"external_libraries": [{ "name": "indexer_ffi" }],
"cmake": {
"extra_include_dirs": ["lib"]
}

2
src/lez_ffi_marshalling.h
View File

@ -1,6 +1,6 @@
#pragma once
#include "lez_indexer_ffi.h"
#include <indexer_ffi.h>
#include <nlohmann/json.hpp>
#include <string>

14
src/lez_indexer_ffi.h
View File

@ -1,14 +0,0 @@
#pragma once
// The generated indexer_ffi.h carries no include guard, so including it from
// more than one header in the same translation unit redefines every type.
// Wrap it once here (behind #pragma once) and include THIS header instead of
// <indexer_ffi.h> directly, so the FFI types can be shared by both
// lez_indexer_module.h and lez_ffi_marshalling.h.
#ifdef __cplusplus
extern "C" {
#endif
#include <indexer_ffi.h>
#ifdef __cplusplus
}
#endif

102
src/lez_indexer_module_impl.cpp
View File

@ -1,7 +1,7 @@
#include "lez_indexer_module_impl.h"
#include "lez_ffi_marshalling.h"
#include "lez_indexer_ffi.h"
#include <indexer_ffi.h>
#include <cstdint>
#include <cstdio>
@ -21,28 +21,20 @@ namespace {
} // namespace
LezIndexerModuleImpl::~LezIndexerModuleImpl() {
if (indexer_service_ffi) {
OperationStatus operation_result = stop_indexer(handle(indexer_service_ffi));
if (is_error(&operation_result)) {
warn("destructor", "indexer FFI error on stop");
}
indexer_service_ffi = nullptr;
if (stop_indexer() != 0) {
warn("destructor", "indexer FFI error on stop");
}
}
// === Indexer Lifecycle ===
int64_t LezIndexerModuleImpl::start_indexer(const std::string& config_path, const std::string& port) {
int64_t LezIndexerModuleImpl::start_indexer(const std::string& config_path) {
if (!indexer_service_ffi) {
char* end = nullptr;
const unsigned long parsed = std::strtoul(port.c_str(), &end, 10);
if (end == port.c_str() || *end != '\0' || parsed > 0xFFFF) {
warn("start_indexer", "invalid port");
return -1;
}
const uint16_t port_num = static_cast<uint16_t>(parsed);
InitializedIndexerServiceFFIResult res = ::start_indexer(config_path.c_str(), port_num);
// Null runtime: the FFI creates and owns its own tokio runtime. Storage
// goes under this module's instance persistence path (host-owned, stable
// per Basecamp --user-dir) so RocksDB never lands in the process CWD.
InitializedIndexerServiceFFIResult res =
::start_indexer(nullptr, config_path.c_str(), instancePersistencePath().c_str());
if (is_error(&res.error)) {
warn("start_indexer", "indexer FFI error");
return static_cast<int64_t>(res.error);
@ -54,19 +46,31 @@ int64_t LezIndexerModuleImpl::start_indexer(const std::string& config_path, cons
return 0;
}
int64_t LezIndexerModuleImpl::stop_indexer() {
if (!indexer_service_ffi) {
return 0; // not running
}
// stop_indexer frees the handle; null ours before returning so a later start
// (or the destructor) doesn't double-free or operate on a dead pointer.
OperationStatus operation_result = ::stop_indexer(handle(indexer_service_ffi));
indexer_service_ffi = nullptr;
if (is_error(&operation_result)) {
warn("stop_indexer", "indexer FFI error on stop");
return static_cast<int64_t>(operation_result);
}
return 0;
}
void LezIndexerModuleImpl::init_logger(const std::string& level) {
::init_logger(level.c_str());
}
// === Indexer Queries ===
//
// Each method calls the matching query_* FFI function on the handle we already
// hold (no Runtime needed — the pinned FFI carries its own runtime inside the
// handle), marshals the returned C struct into compact JSON, then frees the
// FFI's deep allocations with the matching free_ffi_* function.
//
// Known leak: query_* returns its payload in a `Box::into_raw` *outer* box
// (PointerResult.value), and the free_ffi_* functions free only the inner
// boxes/vectors (they take the struct by value). The pinned FFI provides no
// free for the outer box, and libc free() would be UB if Rust's global
// allocator differs from the system one, so we deliberately leak the small
// (~8-32 byte) outer wrapper per query. Tracked for an upstream fix.
// Each method calls the matching query_* FFI function on the handle we hold,
// marshals the returned C struct into compact JSON, then frees the FFI
// allocation with the matching free_ffi_* function. The frees take the outer
// pointer (PointerResult.value) and reclaim the whole allocation.
std::string LezIndexerModuleImpl::getAccount(const std::string& account_id) {
if (!indexer_service_ffi) {
@ -87,7 +91,7 @@ std::string LezIndexerModuleImpl::getAccount(const std::string& account_id) {
}
const std::string out = jsonToCompactString(ffiAccountToJson(*res.value));
::free_ffi_account(*res.value);
::free_ffi_account(res.value);
return out;
}
@ -114,7 +118,7 @@ std::string LezIndexerModuleImpl::getBlockById(const std::string& block_id) {
if (res.value->is_some && res.value->value) {
out = jsonToCompactString(ffiBlockToJson(*res.value->value));
}
::free_ffi_block_opt(*res.value);
::free_ffi_block_opt(res.value);
return out;
}
@ -140,7 +144,7 @@ std::string LezIndexerModuleImpl::getBlockByHash(const std::string& hash) {
if (res.value->is_some && res.value->value) {
out = jsonToCompactString(ffiBlockToJson(*res.value->value));
}
::free_ffi_block_opt(*res.value);
::free_ffi_block_opt(res.value);
return out;
}
@ -167,7 +171,7 @@ std::string LezIndexerModuleImpl::getTransaction(const std::string& hash) {
if (res.value->is_some && res.value->value) {
out = jsonToCompactString(ffiTransactionToJson(*res.value->value));
}
::free_ffi_transaction_opt(*res.value);
::free_ffi_transaction_opt(res.value);
return out;
}
@ -213,7 +217,7 @@ std::string LezIndexerModuleImpl::getBlocks(const std::string& before, const std
for (uintptr_t i = 0; i < res.value->len; ++i) {
arr.push_back(ffiBlockToJson(res.value->entries[i]));
}
::free_ffi_block_vec(*res.value);
::free_ffi_block_vec(res.value);
return jsonToCompactString(arr);
}
@ -223,15 +227,33 @@ std::string LezIndexerModuleImpl::getLastFinalizedBlockId() {
return {};
}
PointerResult_u64__OperationStatus res = ::query_last_block(handle(indexer_service_ffi));
if (is_error(&res.error) || !res.value) {
warn("getLastFinalizedBlockId", "indexer FFI error");
LastBlockIdResult res = ::query_last_block(handle(indexer_service_ffi));
if (is_error(&res.error) || !res.is_some) {
warn("getLastFinalizedBlockId", "indexer FFI error or no finalized block yet");
return {};
}
// Bare decimal string; the FFI provides no free for the boxed u64 (leaked,
// see the note above).
return u64ToString(*res.value);
// Bare decimal string; the id is returned inline, nothing to free.
return u64ToString(res.block_id);
}
std::string LezIndexerModuleImpl::getStatus() {
if (!indexer_service_ffi) {
warn("getStatus", "indexer not started");
return {};
}
// The FFI builds the status JSON itself (schema owned by indexer_core), so
// there is nothing to marshal — copy the C string out and free it.
char* json = ::query_status(handle(indexer_service_ffi));
if (!json) {
warn("getStatus", "indexer FFI error");
return {};
}
const std::string out(json);
::free_cstring(json);
return out;
}
std::string LezIndexerModuleImpl::getTransactionsByAccount(
@ -270,6 +292,6 @@ std::string LezIndexerModuleImpl::getTransactionsByAccount(
for (uintptr_t i = 0; i < res.value->len; ++i) {
arr.push_back(ffiTransactionToJson(res.value->entries[i]));
}
::free_ffi_transaction_vec(*res.value);
::free_ffi_transaction_vec(res.value);
return jsonToCompactString(arr);
}

32
src/lez_indexer_module_impl.h
View File

@ -27,12 +27,19 @@ public:
~LezIndexerModuleImpl();
/// Boot ingestion against the indexer config at `config_path` (must be an
/// ABSOLUTE path — the module runs in a logos_host subprocess), listening on
/// `port`. Idempotent: a second call while already running is a no-op.
/// Returns 0 on success, else the FFI OperationStatus code (-1 on bad port).
/// ABSOLUTE path — the module runs in a logos_host subprocess). RocksDB state
/// is stored under this module's instance persistence path (host-owned),
/// independent of the config. Idempotent: a second call while already running
/// is a no-op. Returns 0 on success, else the FFI OperationStatus code.
/// int64_t (not int): the universal codegen marshals int64_t/std::string/bool
/// as scalar wire types; a plain `int` return is treated as a JSON payload.
int64_t start_indexer(const std::string& config_path, const std::string& port);
int64_t start_indexer(const std::string& config_path);
/// Stop ingestion and release the FFI handle. No-op (returns 0) when the
/// indexer isn't running. Pair with start_indexer to apply a new config:
/// stop, then start (start_indexer stays idempotent and won't restart on its
/// own). Returns 0 on success, else the FFI OperationStatus code.
int64_t stop_indexer();
/// Account by 32-byte hex id. The returned JSON omits the id; callers inject
/// the queried id themselves.
@ -48,6 +55,10 @@ public:
std::string getBlocks(const std::string& before, const std::string& limit);
/// Tip block id as a bare decimal string.
std::string getLastFinalizedBlockId();
/// Current ingestion status as a compact JSON object so a UI can tell
/// "catching up" from "failed": { state (starting/syncing/caught_up/error),
/// indexedBlockId, lastError }. Empty string if the indexer isn't running.
std::string getStatus();
/// Transactions touching `account_id` (32-byte hex), paginated by decimal
/// `offset`/`limit`.
//
@ -60,15 +71,10 @@ public:
std::string getTransactionsByAccount(const std::string& account_id, const std::string& offset, const std::string& limit);
// clang-format on
// Indexer Logging (opt-in)
//
// Installs the indexer FFI's logger (env_logger) so the indexer's Rust `log`
// output surfaces in the host process. Commented out because the underlying
// `::init_logger()` export does not yet exist in the pinned
// logos-execution-zone FFI. Uncomment once the export lands upstream and the
// pin is bumped.
//
// void init_logger();
/// Enable the indexer's logging at `level` (off/error/warn/info/debug/trace;
/// null or unparseable falls back to info). Scoped to the indexer crates
/// only; the first call wins (subsequent calls are no-ops).
void init_logger(const std::string& level);
private:
// IndexerServiceFFI* — opaque here (see class comment); cast in the .cpp.