Chrysostomos Nanakos
f7a3e90414
refactor: remove makeRandomDataset helper function
...
Remove makeRandomDataset in favor of direct composition with
makeRandomBlocks and makeDataset.
Part of https://github.com/codex-storage/nim-codex/issues/974
Signed-off-by: Chrysostomos Nanakos <chris@include.gr>
2025-11-03 14:10:53 +02:00
Chrysostomos Nanakos
8812d98271
fix: assign selectPeer field in BlockExcEngine ctor
...
Part of https://github.com/codex-storage/nim-codex/issues/974
Signed-off-by: Chrysostomos Nanakos <chris@include.gr>
2025-11-03 14:10:53 +02:00
Chrysostomos Nanakos
6efdd4a068
chore: apply nph formatting
...
Part of https://github.com/codex-storage/nim-codex/issues/974
Signed-off-by: Chrysostomos Nanakos <chris@include.gr>
2025-11-03 14:10:52 +02:00
Chrysostomos Nanakos
0636adf5e8
refactor: make markRequested idempotent
...
Returns false on duplicate marking attempts instead of logging errors,
eliminating duplicate marking loop in blockPresenceHandler and
preventing duplicate block requests across concurrent flows.
Part of https://github.com/codex-storage/nim-codex/issues/974
2025-11-03 14:10:51 +02:00
Chrysostomos Nanakos
dddf7424b4
fix: resolve stuck peer refresh state preventing block discovery
...
This prevents peers from becoming permanently invisible to block discovery when
they fail to respond to WantHave requests.
Part of https://github.com/codex-storage/nim-codex/issues/974
2025-11-03 14:10:50 +02:00
Chrysostomos Nanakos
4abe8c4d97
feat: add strategic runtime metrics for block exchange monitoring
...
- Add codex_block_exchange_discovery_requests_total counter to track peer
discovery frequency
- Add codex_block_exchange_peer_timeouts_total counter to monitor peer
reliability issues
- Add codex_block_exchange_requests_failed_total counter to track request
failure rates
Part of https://github.com/codex-storage/nim-codex/issues/974
2025-11-03 14:10:49 +02:00
Chrysostomos Nanakos
9af510316a
perf: optimize block batch size from 500 to 50 blocks per message
...
Achieves significant memory reduction with equivalent network
performance. The reduced batch size prevents memory pressure
while preserving transfer efficiency, improving overall system
resource utilization.
Part of https://github.com/codex-storage/nim-codex/issues/974
2025-11-03 14:10:49 +02:00
Chrysostomos Nanakos
b9fe7dce58
feat: add peer count limits to discovery engine
...
This prevents unbounded peer accumulation while maintaining redundancy.
When peer count exceeds the maximum, the least active peers are removed from
tracking to free resources.
Part of https://github.com/codex-storage/nim-codex/issues/974
2025-11-03 14:10:48 +02:00
Chrysostomos Nanakos
2fdf234c9d
perf: optimize further fetchBatched
...
- Reduce DefaultFetchBatch to prevent blockData explosion
- Add chunked onBatch processing to prevent memory accumulation
- Implement memory buffer reuse to reduce GC allocation churn
- Fix short-circuit evaluation
Part of https://github.com/codex-storage/nim-codex/issues/974
2025-11-03 14:10:47 +02:00
Chrysostomos Nanakos
0fbca17269
feat: implement weighted random peer selection for load balancing
...
Use probabilistic distribution based on peer quality scores, giving all peers
opportunity while favoring better-performing ones. Selection probability is
inversely proportional to score.
Part of https://github.com/codex-storage/nim-codex/issues/974
2025-11-03 14:10:46 +02:00
gmega
8e8d9f8e60
fix: randomize block refresh time, optimize context store checks
2025-11-03 14:10:46 +02:00
gmega
1fdb14f092
feat: add block knowledge request mechanism, implement tests
2025-11-03 14:10:45 +02:00
gmega
9114a620a3
feat: add stopgap "adaptive" refresh
2025-11-03 14:10:44 +02:00
gmega
908d527dc9
fix: fix testdiscovery so it works with stricter block protocol
2025-11-03 14:10:44 +02:00
gmega
544ec123c7
fix: fix block exchange test to stricter protocol; minor refactor
2025-11-03 14:10:43 +02:00
gmega
50ab785662
feat: drop peer on activity timeout
2025-11-03 14:10:42 +02:00
gmega
d91fd053e7
feat: modify retry mechanism; add DHT guard rails; improve block cancellation handling
2025-11-03 14:10:42 +02:00
gmega
38a0656cab
chore: remove file committed by accident
2025-11-03 14:10:41 +02:00
gmega
d875022ec3
feat: allow futures to be returned out-of-order to decrease memory consumption
2025-11-03 14:10:40 +02:00
gmega
d0466ccf80
feat: remove quadratic joins in cancelBlocks; use SafeAsyncIterator for getBlocks; limit memory usage for fetchBatched when used as prefetcher
2025-11-03 14:10:39 +02:00
gmega
cc44052cf5
feat: add SafeAsyncIter chaining
2025-11-03 14:10:39 +02:00
gmega
b0b1c45376
fix: refresh timestamp before issuing request to prevent flood of knowledge updates
2025-11-03 14:10:38 +02:00
Giuliano Mega
b58a92d1f9
Update codex/stores/networkstore.nim
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
2025-11-03 14:10:37 +02:00
gmega
5a983347cb
adapt existing tests to new data structures, remove vestigial tests
2025-11-03 14:10:36 +02:00
gmega
3c43d57497
optimize remaining list joins so they're not quadratic
2025-11-03 14:10:36 +02:00
gmega
096eb118f9
replace list operations with sets
2025-11-03 14:10:35 +02:00
gmega
9088566632
update engine tests; add BlockAddress hashing tests
2025-11-03 14:10:34 +02:00
gmega
1135a513d4
feat: cap how many blocks we can pack in a single message
2025-11-03 14:10:33 +02:00
gmega
475d31bef2
feat: add dataset request batching
2025-11-03 14:10:31 +02:00
Eric
f791a960f2
fix(ci): Windows SIGILL in CI ( #1339 )
2025-11-03 11:45:02 +00:00
Arnaud
db8f866db4
feat: check if CID exists in local store ( #1331 )
2025-11-02 04:32:47 +00:00
Eric
7aca2f0e61
fix(ci): Move conventional commits job to workflow ( #1340 )
2025-11-02 04:00:55 +00:00
Eric
072bff5cab
fix: ci integration tests ( #1335 )
2025-10-30 19:38:11 +11:00
Arnaud
af55a761e6
chore: skip marketplace and long integration tests ( #1326 )
2025-10-22 19:22:33 +11:00
Adam Uhlíř
e3d8d195c3
chore: update nim-libp2p ( #1323 )
2025-10-01 13:19:15 +02:00
Slava
d1f2e2399b
ci: validate pr title to adhere conventional commits ( #1254 )
2025-08-12 08:51:41 +00:00
Slava
8cd10edb69
ci: auto deploy codex on devnet ( #1302 )
2025-07-28 10:02:19 +00:00
Slava
6cf99e255c
ci: release master builds and upload them to the cloud ( #1298 )
2025-07-10 11:17:11 +00:00
Dmitriy Ryajov
7eb2fb12cc
make default dirs runtime, not compile time. ( #1292 )
2025-06-26 18:44:24 +00:00
Slava
352273ff81
chore: bump codex-contracts-eth ( #1293 )
2025-06-26 18:09:48 +00:00
Slava
9ef9258720
chore(ci): bump node to v22 ( #1285 )
2025-06-26 01:11:00 +00:00
markspanbroek
7927afe715
chore: update nph dependency ( #1279 )
...
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2025-06-25 10:30:48 +00:00
markspanbroek
01615354af
refactor(ci): run integration tests in parallel by spinning up more runners ( #1287 )
2025-06-25 08:56:16 +00:00
Chrysostomos Nanakos
baff902137
fix: resolve shared block request cancellation conflicts ( #1284 )
2025-06-24 15:05:25 +00:00
markspanbroek
4d44154a40
fix(ci): remove "update" to gcc-14 on windows ( #1288 )
2025-06-24 09:00:56 +00:00
markspanbroek
e1c397e112
fix(tests): auto import all tests files and fix forgotten tests ( #1281 )
2025-06-23 11:18:59 +00:00
Arnaud
7b660e3554
chore(marketplace): use hardhat ignition ( #1195 )
2025-06-20 15:55:00 +00:00
Arnaud
c5e424ff1b
feat(marketplace) - add status l2 (Linea) network ( #1160 )
2025-06-20 12:30:40 +00:00
Slava
36f64ad3e6
chore: update testnet marketplace address ( #1283 )
2025-06-20 06:13:58 +00:00
Ben Bierens
235c0ec842
chore: updates codex-contracts-eth submodule ( #1278 )
...
Co-authored-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2025-06-19 10:31:52 +00:00
