422 Commits

Author SHA1 Message Date
Chrysostomos Nanakos
dddf7424b4
fix: resolve stuck peer refresh state preventing block discovery
This prevents peers from becoming permanently invisible to block discovery when
they fail to respond to WantHave requests.

Part of https://github.com/codex-storage/nim-codex/issues/974
2025-11-03 14:10:50 +02:00
Chrysostomos Nanakos
4abe8c4d97
feat: add strategic runtime metrics for block exchange monitoring
- Add codex_block_exchange_discovery_requests_total counter to track peer
  discovery frequency
- Add codex_block_exchange_peer_timeouts_total counter to monitor peer
  reliability issues
- Add codex_block_exchange_requests_failed_total counter to track request
  failure rates

Part of https://github.com/codex-storage/nim-codex/issues/974
2025-11-03 14:10:49 +02:00
Chrysostomos Nanakos
9af510316a
perf: optimize block batch size from 500 to 50 blocks per message
Achieves significant memory reduction with equivalent network
performance. The reduced batch size prevents memory pressure
while preserving transfer efficiency, improving overall system
resource utilization.

Part of https://github.com/codex-storage/nim-codex/issues/974
2025-11-03 14:10:49 +02:00
Chrysostomos Nanakos
b9fe7dce58
feat: add peer count limits to discovery engine
This prevents unbounded peer accumulation while maintaining redundancy.
When peer count exceeds the maximum, the least active peers are removed from
tracking to free resources.

Part of https://github.com/codex-storage/nim-codex/issues/974
2025-11-03 14:10:48 +02:00
Chrysostomos Nanakos
2fdf234c9d
perf: optimize further fetchBatched
- Reduce DefaultFetchBatch to prevent blockData explosion
- Add chunked onBatch processing to prevent memory accumulation
- Implement memory buffer reuse to reduce GC allocation churn
- Fix short-circuit evaluation

Part of https://github.com/codex-storage/nim-codex/issues/974
2025-11-03 14:10:47 +02:00
Chrysostomos Nanakos
0fbca17269
feat: implement weighted random peer selection for load balancing
Use probabilistic distribution based on peer quality scores, giving all peers
opportunity while favoring better-performing ones. Selection probability is
inversely proportional to score.

Part of https://github.com/codex-storage/nim-codex/issues/974
2025-11-03 14:10:46 +02:00
gmega
8e8d9f8e60
fix: randomize block refresh time, optimize context store checks 2025-11-03 14:10:46 +02:00
gmega
1fdb14f092
feat: add block knowledge request mechanism, implement tests 2025-11-03 14:10:45 +02:00
gmega
9114a620a3
feat: add stopgap "adaptive" refresh 2025-11-03 14:10:44 +02:00
gmega
908d527dc9
fix: fix testdiscovery so it works with stricter block protocol 2025-11-03 14:10:44 +02:00
gmega
544ec123c7
fix: fix block exchange test to stricter protocol; minor refactor 2025-11-03 14:10:43 +02:00
gmega
50ab785662
feat: drop peer on activity timeout 2025-11-03 14:10:42 +02:00
gmega
d91fd053e7
feat: modify retry mechanism; add DHT guard rails; improve block cancellation handling 2025-11-03 14:10:42 +02:00
gmega
d875022ec3
feat: allow futures to be returned out-of-order to decrease memory consumption 2025-11-03 14:10:40 +02:00
gmega
d0466ccf80
feat: remove quadratic joins in cancelBlocks; use SafeAsyncIterator for getBlocks; limit memory usage for fetchBatched when used as prefetcher 2025-11-03 14:10:39 +02:00
gmega
cc44052cf5
feat: add SafeAsyncIter chaining 2025-11-03 14:10:39 +02:00
gmega
b0b1c45376
fix: refresh timestamp before issuing request to prevent flood of knowledge updates 2025-11-03 14:10:38 +02:00
Giuliano Mega
b58a92d1f9
Update codex/stores/networkstore.nim
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
2025-11-03 14:10:37 +02:00
gmega
3c43d57497
optimize remaining list joins so they're not quadratic 2025-11-03 14:10:36 +02:00
gmega
096eb118f9
replace list operations with sets 2025-11-03 14:10:35 +02:00
gmega
9088566632
update engine tests; add BlockAddress hashing tests 2025-11-03 14:10:34 +02:00
gmega
1135a513d4
feat: cap how many blocks we can pack in a single message 2025-11-03 14:10:33 +02:00
gmega
475d31bef2
feat: add dataset request batching 2025-11-03 14:10:31 +02:00
Arnaud
db8f866db4
feat: check if CID exists in local store (#1331) 2025-11-02 04:32:47 +00:00
Dmitriy Ryajov
7eb2fb12cc
make default dirs runtime, not compile time. (#1292) 2025-06-26 18:44:24 +00:00
Chrysostomos Nanakos
baff902137
fix: resolve shared block request cancellation conflicts (#1284) 2025-06-24 15:05:25 +00:00
Arnaud
c5e424ff1b
feat(marketplace) - add status l2 (Linea) network (#1160) 2025-06-20 12:30:40 +00:00
Slava
36f64ad3e6
chore: update testnet marketplace address (#1283) 2025-06-20 06:13:58 +00:00
Arnaud
e35aec7870
chore: increase gas limits (#1272) 2025-06-18 12:18:56 +00:00
Chrysostomos Nanakos
b305e00160
Add support for slot reconstruction on unavailable slot detection (#1235)
Co-authored-by: Arnaud <arnaud@status.im>
2025-06-12 22:19:42 +00:00
Slava
3d2d8273e6
chore: update testnet marketplace address (#1270) 2025-06-12 08:10:22 +00:00
markspanbroek
27d807a841
fix(sales): fix marketplace block expiry (#1258) 2025-06-11 11:27:09 +00:00
Ben Bierens
85823342e9
Improves logging in maintenance module and erasure. (#1264) 2025-06-10 13:27:52 +00:00
Adam Uhlíř
7502b9ad2c
feat(cirdl): auto-discovery of marketplace contract (#1259) 2025-06-09 10:04:15 +00:00
Arnaud
3e17207a0b
feat(marketplace) - add command line arg for maxPriorityFeePerGas (#1189) 2025-06-05 07:47:39 +00:00
markspanbroek
ffbbee01b1
fix(purchasing): fix crash completing future more than once (#1249) 2025-06-04 14:15:07 +00:00
markspanbroek
2dd436bfb7
fix(sales): do not crash when retrieving request fails (#1248) 2025-06-04 11:22:14 +00:00
Arnaud
2e1306ac2d
chore: fix custom error handling when simulating invalid proofs (#1217)
* Fix custom error handling when simulating invalid proofs

* Update error message
2025-06-03 12:11:18 +00:00
Arnaud
45ade0e3c1
chore(marketplace): use canMarkProofAsMissing (#1188)
* Add canProofBeMarkedAsMissing

* Add more tests

* Update contracts submodule
2025-06-03 09:08:57 +00:00
Arnaud
ca869f6dce
fix(availabilities): use totalRemainingCollateral instead of totalCollateral for comparaison (#1229)
* Use totalRemainingCollateral instead of totalCollateral to compare the availability changes

* Update test to use totalRemainingCollateral instead of totalCollateral when testing OnAvailabilitySaved

* Reduce poll interval

* Fix flaky test

* Fix format
2025-06-02 16:47:12 +00:00
Slava
e43872d0b8
chore: update testnet marketplace address (#1245)
https://github.com/codex-storage/nim-codex/issues/1241
2025-05-30 09:12:55 +00:00
Giuliano Mega
d59c5b023c
chore: bump Nim to 2.2.4 (#1242)
* chore: bump Nim to 2.2.4

* fix: resolve symbol ambiguity and drop auto type

* fix: use reference to task instead of pointer or the compiler will deallocate `task` before the encoding/decoding is done

* fix: convention that maxCollateralPerByte equals totalRemainingCollateral when freeSize is 0 to avoid DivByZeroDefect

* fix: bump compiler version in CI pipeline as well
2025-05-29 16:37:38 -07:00
Arnaud
28a83db69e
chore: returns the collateral when a slot is reserved but not filled (#1216)
* Change token allowance method because increaseAllowance does not exist anymore

* Returns collateral when a reservation is deleted and not only a slot is filled

* Remove the returnedCollateral when the slot is not filled by the host

* Add returnedCollateral when the sale is ignored

* Add returnsCollateral variable for ignored state

* Rebase the contracts submodule on the master

* Add integration test

* Fix duration

* Remove unnecessary teardown function

* Remove misleading comment

* Get returned collateral from the request

* Enable logs to debug on CI

* Fix test

* Increase test timeout

* Fix typo

* Fix rebase
2025-05-29 14:47:37 +00:00
Arnaud
827d9ccccf
Update contracts (#1238) 2025-05-29 08:27:41 +00:00
Arnaud
c689542579
fix: sales cleanup cancellation (#1234)
* fix(sales): handle cancellation of slot cleanup

Ensures that processing slots from the slot queue
continues even when cleanup of a slot is cancelled.

Co-Authored-By: Eric <5089238+emizzle@users.noreply.github.com>

* chore(reservations): add more `raises` annotations

* Fix cleanup cancellation

* Add remove-agent to trackedfutures instead of the cleanup function

* Increase the timeout to match the request expiry

* Enable logs to debug on CI

* Remove useless except and do not return when add item back to slot queue fails

* Reduce poll interval to detect sale cancelled state

* Avoid cancelling cleanup routine

* Do not cancel creating reservation in order to avoid inconsistent state

* Remove useless try except

---------

Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2025-05-29 06:57:05 +00:00
munna0908
71422f0d3d
fix: Support for mapping multiple listener address (#1236)
* multi address mapping support

* fix thread issues

* fix local thread var issue

* chore: rename stopNatThread to stopNatThreads

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2025-05-27 09:05:43 +00:00
Ben Bierens
bfbd7264df
Adds missing async-raises for prover.verify (#1237) 2025-05-26 15:48:59 +00:00
Marcin Czenko
748830570a
checked exceptions in stores (#1179)
* checked exceptions in stores

* makes asynciter as much exception safe as it gets

* introduce "SafeAsyncIter" that uses Results and limits exceptions to cancellations

* adds {.push raises: [].} to errors

* uses SafeAsyncIter in "listBlocks" and in "getBlockExpirations"

* simplifies safeasynciter (magic of auto)

* gets rid of ugly casts

* tiny fix in hte way we create raising futures in tests of safeasynciter

* Removes two more casts caused by using checked exceptions

* adds an extended explanation of one more complex SafeAsyncIter test

* adds missing "finishOnErr" param in slice constructor of SafeAsyncIter

* better fix for "Error: Exception can raise an unlisted exception: Exception" error.

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2025-05-21 21:17:04 +00:00
markspanbroek
bde98738c2
fix(slotqueue): simplify slot queue workers (#1224)
* fix(slotqueue): simplify slot queue workers

- worker is now just an async running loop
- instead of passing a "done" Future, use an
  AsyncEvent to signal completion

* chore(slotqueue): address review comments

Co-Authored-By: Eric <5089238+emizzle@users.noreply.github.com>
Co-Authored-By: Dmitriy Ryajov <dryajov@gmail.com>

---------

Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2025-05-15 13:02:04 +00:00
Adam Uhlíř
f144099377
fix(api): availability creation validation (#1212) 2025-05-14 08:46:16 +00:00