943 Commits

Author SHA1 Message Date
Marcin Czenko
6acb78b6ef
feat: libstorage metrics potential fixes (#1458)
## Root Cause

The most likely failure is in the C test harness, not `storage_space`
itself.

The callback writes `r->ret` before it finishes writing `r->msg`, and
the polling thread treats `ret != -1` as “response is complete”. That
can make `is_resp_ok` return with `res == NULL`, after which
`check_space` calls `strstr(res, ...)` and segfaults.

The segfault is very likely a race in `tests/cbindings/storage.c`.

In `callback`:

```c
r->ret = ret;
...
r->msg = malloc(len + 1);
memcpy(r->msg, msg, len);
```

But `wait_resp` treats `r->ret != -1` as “the response is ready”:

```c
while (get_ret(r) == -1 && retries < MAX_RETRIES)
```

So this can happen:

1. Storage worker thread enters `callback`.
2. Callback sets `r->ret = RET_OK`.
3. Main C test thread sees `ret != -1` and exits `wait_resp`.
4. `is_resp_ok` observes `r->msg == NULL`, so `res` remains `NULL`.
5. `check_space` calls:

```c
strstr(res, "totalBlocks")
```

6. `strstr(NULL, ...)` segfaults.

It can also be worse: `is_resp_ok` may call `free_resp(r)` while the
callback is still writing into `r`, causing use-after-free.

## Why It Appears At `check_space`

This is probably timing. Earlier responses happened to finish copying
before the polling thread observed `ret`; `check_space` just hit the
race.

The new metrics PR did not reach `check_get_metrics`; the crash happens
before it. But since the PR now makes `make testLibstorage` run the C
test consistently, it exposes this existing test harness race.

## Minimal Fix

1. In `callback`, set `r->ret = ret` last, after `msg`, `chunk`, and
`len` are fully populated.
2. Add defensive `res == NULL` checks in tests like `check_space`,
similar to `check_get_metrics`.

## Longer-Term Fix

Ideally, replace polling shared fields with a mutex/condition variable
or atomics, because the current cross-thread access is technically a C
data race.
```

For point 3: the current fix improves ordering, but it still relies on unsynchronized reads/writes between threads. In C, one thread writing `r->ret`, `r->msg`, and `r->len` while another thread reads them without atomics or a lock is undefined behavior. Practically, it may work most of the time, but the compiler/CPU are not required to make those writes visible in the order we expect.

A proper fix would make `Resp` thread-safe. The cleanest approach is usually:

```c
typedef struct
{
    pthread_mutex_t mutex;
    pthread_cond_t cond;
    bool done;
    int ret;
    char *msg;
    char *chunk;
    size_t len;
} Resp;
```

Then:

- `callback` locks the mutex.
- It writes `msg`, `chunk`, `len`, and `ret`.
- It sets `done = true`.
- It signals the condition variable.
- It unlocks.

And:

- `wait_resp` locks the mutex.
- It waits on the condition variable until `done == true` or timeout.
- It unlocks.

This removes the sleep-based polling and gives the test a real happens-before relationship between callback writes and test-thread reads.
2026-06-18 20:24:46 +02:00
Marcin Czenko
ebd8a42755
feat: include metric type and help 2026-06-18 19:47:34 +02:00
Marcin Czenko
94e0abd579
build: minor fixes 2026-06-18 18:55:35 +02:00
Marcin Czenko
ccb6ed1303
fix: better (??) thread synchronization 2026-06-18 18:20:24 +02:00
Marcin Czenko
ce200ee0a2
fix: seg fault while running libstorage tests and adds some defensive checks 2026-06-18 18:09:55 +02:00
Giuliano Mega
b6306003d1
Merge branch 'master' into feat/libstorage-metrics 2026-06-18 13:00:46 -03:00
Eric
647fbb84a5
fix: allFinishedFailed -> allDone to retain cancellations (#1451) 2026-06-18 15:26:00 +00:00
Giuliano Mega
11f03b77b7
Merge branch 'master' into feat/libstorage-metrics 2026-06-18 12:25:36 -03:00
gmega
4885c5e9ba
address copilots comments 2026-06-18 12:25:24 -03:00
Chrysostomos Nanakos
7931aba01a
fix(tools/mix): raise mix_relay_dht libp2p connection limit (--max-connections, default 160) (#1457)
Signed-off-by: Chrysostomos Nanakos <chris@include.gr>
2026-06-18 15:06:44 +00:00
gmega
23e2178fde
add comment 2026-06-18 10:59:59 -03:00
gmega
8959700cf3
drop nim_runtime_info collector, handle badly behaved collectors 2026-06-18 10:46:56 -03:00
gmega
98c89b98b3
add libstorage call and tests 2026-06-18 09:34:28 -03:00
gmega
60ddfc241e
add nim-metrics -> logos openmetrics serializer 2026-06-18 09:34:27 -03:00
Giuliano Mega
6a6482c217
feat: add toggle_private_query to libstorage (#1448)
Signed-off-by: Chrysostomos Nanakos <chris@include.gr>
Co-authored-by: Chrysostomos Nanakos <chris@include.gr>
2026-06-18 12:13:51 +03:00
Chrysostomos Nanakos
3022b876bc
feat: run DHT queries over Mix (#1452)
Signed-off-by: Chrysostomos Nanakos <chris@include.gr>
2026-06-18 12:13:38 +03:00
Giuliano Mega
8f9eceaa19
feat: simple test runner for libstorage tests (#1450) 2026-06-17 11:02:27 +00:00
Giuliano Mega
aee1398836
fix: Merkle tree proof length check (#1432) 2026-06-16 18:14:17 +00:00
Eric
c4a92b3d11
refactor(release): emptyDir volumes are now used over PVCs, so no need to clean up (#1444) 2026-06-16 12:18:26 +00:00
Eric
58f8f2f63d
fix(blockexchange): bounds per-message work (#1447)
Signed-off-by: Chrysostomos Nanakos <chris@include.gr>
Co-authored-by: Chrysostomos Nanakos <chris@include.gr>
2026-06-15 13:32:21 +00:00
Eric
cb928aacdd
feat(testing): Release tests -- force pod spread to one pod per node (#1445) 2026-06-15 06:17:51 +00:00
richΛrd
d65f32f819
chore: update libp2p to 2.0.0 (#1443)
Co-authored-by: Arnaud <arnaud@status.im>
2026-06-12 12:32:33 +00:00
Arnaud
85d8822704
chore: include cancelled futures in failures (#1442) 2026-06-11 06:28:23 +00:00
Eric
d61512a5b7
feat: end-to-end release test pipeline on GKE with structured logging (#1439)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-05 06:27:55 +00:00
Eric
b892379ff1
chore: add nim mcp server and CLAUDE.md (#1441) 2026-06-03 19:07:19 +00:00
Arnaud
0ad13ca897
chore: update nimbus build system (#1433) 2026-06-03 08:27:04 +00:00
Arnaud
abce4f68c5
chore: remove unused cache and make discovery IP limits configurable (#1435) 2026-06-03 06:39:07 +00:00
Giuliano Mega
3e0e143270
feat: add testnet presets and no-bootstrap option (#1438) 2026-05-29 14:19:39 +00:00
Giuliano Mega
ebb1ae2599
feat: network presets (#1437) 2026-05-22 22:20:53 +00:00
Chrysostomos Nanakos
1218a7edbe
fix: validate WantBlocks request ranges to prevent DoS (#1434)
Signed-off-by: Chrysostomos Nanakos <chris@include.gr>
2026-05-13 09:11:30 +00:00
Arnaud
cf2f40f559
chore: update nim 2.2.10 (#1425) 2026-05-06 10:51:30 +00:00
Giuliano Mega
4b82d7f82d
fix: Remove unused (and incorrect) verify call from Merkle tree (#1416) 2026-04-30 21:21:47 +00:00
Chrysostomos Nanakos
76c4b23c62
fix: re-seed DHT routing table when empty (#1428)
Signed-off-by: Chrysostomos Nanakos <chris@include.gr>
2026-04-29 17:00:32 +00:00
Chrysostomos Nanakos
bb6ab1befa
chore: Block exchange protocol rewrite (#1411)
Signed-off-by: Chrysostomos Nanakos <chris@include.gr>
2026-04-25 00:37:42 +00:00
Arnaud
2dd97631ed
chore: update nim 2.2.8 (#1424)
Signed-off-by: Arnaud <arno.deville@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2026-04-22 13:57:14 +00:00
Arnaud
4c9ecd0766
feat: add a warning when private address only (#1423)
Signed-off-by: Arnaud <arno.deville@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-04-21 11:36:02 +00:00
Arnaud
8dd2356000
chore: update chronicles (#1422) 2026-04-21 06:05:00 +00:00
Arnaud
79d59dc66c
fix: cleanup chronicles writer in close function (#1413)
Signed-off-by: Arnaud <arno.deville@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-04-13 08:40:14 +00:00
Arnaud
ffc86c23d7
fix: nph build (#1419) 2026-04-13 08:20:00 +00:00
Arnaud
e8bc5c46e9
chore: cleanup dependencies (#1415)
Signed-off-by: Arnaud <arno.deville@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-04-13 06:07:38 +00:00
Eric
4ea8350612
fix(repostore): prevent underflow when deleting datasets (#1412) 2026-03-31 05:25:08 +00:00
5ecd82eeaa
mit-license: add copyright owner Logos
Signed-off-by: markoburcul <marko@status.im>
2026-03-24 18:19:06 +01:00
Eric
3bc6aa8b8d
feat(cli)!: change listen addr to listen port (#1409) v0.3.2 2026-02-24 10:03:48 +00:00
Eric
ab413bdfcf
chore!: Finish renaming Codex to Logos Storage (#1399) 2026-02-19 04:59:15 +00:00
Eric
4068bcb2ed
fix: bump nph and refactor build process in makefile (#1410) 2026-02-19 02:12:45 +00:00
Eric
fef46aee35
feat(testing): local libstorage c bindings test (#1407) 2026-02-18 04:29:59 +00:00
e375223500
fix: nix /tmp permission errors on MacOS
Because on MacOS Nix does not have proper filesystem namespacing like on
linux the /tmp filesystem is the real /tmp on the machine.

This can cause permission issues when different Nix build users creat it:

> cannot create directory: /tmp/nim/koch_d

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2026-02-13 12:28:50 +01:00
Eric
d02f44ffd6
feat(libstorage): enable rest api in libstorage in debug builds (#1406) 2026-02-11 08:25:53 +00:00
Giuliano Mega
3830ecc9e6
chore: bump nim-datastore (#1405) 2026-02-09 13:59:25 -03:00
Giuliano Mega
61791536e8
fix: add callback call to storage_destroy to keep consistency (#1401) 2026-02-09 14:03:55 +00:00