845 Commits

Author SHA1 Message Date
Chrysostomos Nanakos
9e2f973b8a
refactor: make markRequested idempotent
Returns false on duplicate marking attempts instead of logging errors,
eliminating duplicate marking loop in blockPresenceHandler and
preventing duplicate block requests across concurrent flows.
2025-09-30 11:59:27 +03:00
Chrysostomos Nanakos
5efd08fb0e
fix: resolve stuck peer refresh state preventing block discovery
This prevents peers from becoming permanently invisible to block discovery when
they fail to respond to WantHave requests.
2025-09-30 11:59:27 +03:00
Chrysostomos Nanakos
25a2b3e9ed
feat: add strategic runtime metrics for block exchange monitoring
- Add codex_block_exchange_discovery_requests_total counter to track peer
  discovery frequency
- Add codex_block_exchange_peer_timeouts_total counter to monitor peer
  reliability issues
- Add codex_block_exchange_requests_failed_total counter to track request
  failure rates
2025-09-30 11:59:27 +03:00
Chrysostomos Nanakos
7f3004b5c0
perf: optimize block batch size from 500 to 50 blocks per message
Achieves significant memory reduction with equivalent network
performance. The reduced batch size prevents memory pressure
while preserving transfer efficiency, improving overall system
resource utilization.
2025-09-30 11:59:27 +03:00
Chrysostomos Nanakos
a7a384a66f
feat: add peer count limits to discovery engine
This prevents unbounded peer accumulation while maintaining redundancy.
When peer count exceeds the maximum, the least active peers are removed from
tracking to free resources.
2025-09-30 11:59:27 +03:00
Chrysostomos Nanakos
3a36e38b1f
perf: optimize further fetchBatched
- Reduce DefaultFetchBatch to prevent blockData explosion
- Add chunked onBatch processing to prevent memory accumulation
- Implement memory buffer reuse to reduce GC allocation churn
- Fix short-circuit evaluation
2025-09-30 11:59:27 +03:00
Chrysostomos Nanakos
524e93090d
feat: implement weighted random peer selection for load balancing
Use probabilistic distribution based on peer quality scores, giving all peers
opportunity while favoring better-performing ones. Selection probability is
inversely proportional to score.
2025-09-30 11:43:32 +03:00
Chrysostomos Nanakos
7cb4b31557
Implement load-balanced peer selection for block requests 2025-09-15 12:28:50 +03:00
gmega
6891824226
fix: randomize block refresh time, optimize context store checks 2025-07-09 14:53:06 -03:00
gmega
2ada4b5eb3
feat: add block knowledge request mechanism, implement tests 2025-07-08 15:23:54 -03:00
gmega
d7c403edfe
feat: add stopgap "adaptive" refresh 2025-07-03 18:43:11 -03:00
gmega
b274b29b83
fix: fix testdiscovery so it works with stricter block protocol 2025-07-03 12:06:02 -03:00
gmega
572b44856b
fix: fix block exchange test to stricter protocol; minor refactor 2025-07-03 10:18:32 -03:00
gmega
97fd68e4a3
feat: drop peer on activity timeout 2025-07-02 20:17:06 -03:00
gmega
ad8e250605
feat: modify retry mechanism; add DHT guard rails; improve block cancellation handling 2025-07-01 21:20:14 -03:00
gmega
cdab172bae
chore: remove file committed by accident 2025-06-30 20:19:32 -03:00
gmega
83fd83ffa5
feat: allow futures to be returned out-of-order to decrease memory consumption 2025-06-30 20:10:56 -03:00
gmega
db279d8fa9
feat: remove quadratic joins in cancelBlocks; use SafeAsyncIterator for getBlocks; limit memory usage for fetchBatched when used as prefetcher 2025-06-30 17:31:03 -03:00
gmega
d94bfe60f6
feat: add SafeAsyncIter chaining 2025-06-27 16:04:49 -03:00
gmega
313d6bac1f
fix: refresh timestamp before issuing request to prevent flood of knowledge updates 2025-06-10 19:55:51 -03:00
Giuliano Mega
03a1cc70eb
Update codex/stores/networkstore.nim
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
2025-06-09 17:13:10 -03:00
gmega
fbd378ec18
adapt existing tests to new data structures, remove vestigial tests 2025-06-09 15:15:08 -03:00
gmega
e18996eed1
optimize remaining list joins so they're not quadratic 2025-06-09 11:26:16 -03:00
gmega
a53f8da855
replace list operations with sets 2025-06-06 15:09:41 -03:00
gmega
b6e120394a
update engine tests; add BlockAddress hashing tests 2025-06-06 14:01:51 -03:00
gmega
1dcb998be6
feat: cap how many blocks we can pack in a single message 2025-06-06 12:56:56 -03:00
gmega
d80b7ddbd0
switch nph branch to provisional which compiles in 2.2.x 2025-06-05 17:45:38 -03:00
gmega
41f94d7a73
feat: add dataset request batching 2025-06-05 15:30:08 -03:00
markspanbroek
2dd436bfb7
fix(sales): do not crash when retrieving request fails (#1248) 2025-06-04 11:22:14 +00:00
Arnaud
2e1306ac2d
chore: fix custom error handling when simulating invalid proofs (#1217)
* Fix custom error handling when simulating invalid proofs

* Update error message
2025-06-03 12:11:18 +00:00
Arnaud
45ade0e3c1
chore(marketplace): use canMarkProofAsMissing (#1188)
* Add canProofBeMarkedAsMissing

* Add more tests

* Update contracts submodule
2025-06-03 09:08:57 +00:00
Arnaud
ca869f6dce
fix(availabilities): use totalRemainingCollateral instead of totalCollateral for comparaison (#1229)
* Use totalRemainingCollateral instead of totalCollateral to compare the availability changes

* Update test to use totalRemainingCollateral instead of totalCollateral when testing OnAvailabilitySaved

* Reduce poll interval

* Fix flaky test

* Fix format
2025-06-02 16:47:12 +00:00
Slava
e43872d0b8
chore: update testnet marketplace address (#1245)
https://github.com/codex-storage/nim-codex/issues/1241
v0.2.3
2025-05-30 09:12:55 +00:00
Giuliano Mega
d59c5b023c
chore: bump Nim to 2.2.4 (#1242)
* chore: bump Nim to 2.2.4

* fix: resolve symbol ambiguity and drop auto type

* fix: use reference to task instead of pointer or the compiler will deallocate `task` before the encoding/decoding is done

* fix: convention that maxCollateralPerByte equals totalRemainingCollateral when freeSize is 0 to avoid DivByZeroDefect

* fix: bump compiler version in CI pipeline as well
2025-05-29 16:37:38 -07:00
Arnaud
28a83db69e
chore: returns the collateral when a slot is reserved but not filled (#1216)
* Change token allowance method because increaseAllowance does not exist anymore

* Returns collateral when a reservation is deleted and not only a slot is filled

* Remove the returnedCollateral when the slot is not filled by the host

* Add returnedCollateral when the sale is ignored

* Add returnsCollateral variable for ignored state

* Rebase the contracts submodule on the master

* Add integration test

* Fix duration

* Remove unnecessary teardown function

* Remove misleading comment

* Get returned collateral from the request

* Enable logs to debug on CI

* Fix test

* Increase test timeout

* Fix typo

* Fix rebase
2025-05-29 14:47:37 +00:00
Slava
13811825b3
ci: use macos arm runners (#1174)
* ci: use inputs instead of matrix in a ccache key

* ci: switch to arm runners for macos

* ci: use node 20

* ci: pass cpu to a composite action
2025-05-29 10:17:46 +00:00
Arnaud
827d9ccccf
Update contracts (#1238) 2025-05-29 08:27:41 +00:00
Arnaud
c689542579
fix: sales cleanup cancellation (#1234)
* fix(sales): handle cancellation of slot cleanup

Ensures that processing slots from the slot queue
continues even when cleanup of a slot is cancelled.

Co-Authored-By: Eric <5089238+emizzle@users.noreply.github.com>

* chore(reservations): add more `raises` annotations

* Fix cleanup cancellation

* Add remove-agent to trackedfutures instead of the cleanup function

* Increase the timeout to match the request expiry

* Enable logs to debug on CI

* Remove useless except and do not return when add item back to slot queue fails

* Reduce poll interval to detect sale cancelled state

* Avoid cancelling cleanup routine

* Do not cancel creating reservation in order to avoid inconsistent state

* Remove useless try except

---------

Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2025-05-29 06:57:05 +00:00
munna0908
71422f0d3d
fix: Support for mapping multiple listener address (#1236)
* multi address mapping support

* fix thread issues

* fix local thread var issue

* chore: rename stopNatThread to stopNatThreads

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2025-05-27 09:05:43 +00:00
markspanbroek
25a8077e80
fix(integration): fix api validation test (#1201)
* integration: shutdown codex node at end of test

On Windows the codex node did not shut down properly after this test
finished.

* contracts: fix flaky test
2025-05-26 16:49:53 +00:00
Ben Bierens
bfbd7264df
Adds missing async-raises for prover.verify (#1237) 2025-05-26 15:48:59 +00:00
Arnaud
f7d06cd0e8
chore(marketplace): switch to websocket (#1166)
* Switch to websocket

* Create resubscribe future

* Resubscribe websocket events after 5 minutes

* Remove the subscribe workaround and use define the resubscribe symbol

* Use localhost for ws url

* Define 240 seconds for resubscription interval

* Ensute that updates are sync when using ws
2025-05-23 14:13:19 +00:00
Marcin Czenko
748830570a
checked exceptions in stores (#1179)
* checked exceptions in stores

* makes asynciter as much exception safe as it gets

* introduce "SafeAsyncIter" that uses Results and limits exceptions to cancellations

* adds {.push raises: [].} to errors

* uses SafeAsyncIter in "listBlocks" and in "getBlockExpirations"

* simplifies safeasynciter (magic of auto)

* gets rid of ugly casts

* tiny fix in hte way we create raising futures in tests of safeasynciter

* Removes two more casts caused by using checked exceptions

* adds an extended explanation of one more complex SafeAsyncIter test

* adds missing "finishOnErr" param in slice constructor of SafeAsyncIter

* better fix for "Error: Exception can raise an unlisted exception: Exception" error.

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2025-05-21 21:17:04 +00:00
markspanbroek
bde98738c2
fix(slotqueue): simplify slot queue workers (#1224)
* fix(slotqueue): simplify slot queue workers

- worker is now just an async running loop
- instead of passing a "done" Future, use an
  AsyncEvent to signal completion

* chore(slotqueue): address review comments

Co-Authored-By: Eric <5089238+emizzle@users.noreply.github.com>
Co-Authored-By: Dmitriy Ryajov <dryajov@gmail.com>

---------

Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
v0.2.2
2025-05-15 13:02:04 +00:00
Adam Uhlíř
28e87d06cc
docs(openapi): freeSize non-optional (#1211) 2025-05-14 10:14:40 +00:00
Adam Uhlíř
f144099377
fix(api): availability creation validation (#1212) 2025-05-14 08:46:16 +00:00
Adam Uhlíř
19a5e05c13
docs(openapi): add local data delete endpoint (#1214)
* docs(openapi): add local data delete endpoint

* chore: feedback

Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>

---------

Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2025-05-03 16:54:38 +00:00
Slava
b39d541227
chore: update testnet marketplace address (#1209)
https://github.com/codex-storage/nim-codex/issues/1203
v0.2.1
2025-04-23 06:18:38 +00:00
Adam Uhlíř
d220e53fe1
ci: trigger python generator upon release (#1208) 2025-04-22 14:46:03 +00:00
Ben Bierens
2eb83a0ebb
Codex-contracts hash in version information. (#1207)
* Adds revision hash of codex-contracts to version information.

* Update codex/conf.nim

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>

* Update codex/conf.nim

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>

* Update codex/rest/api.nim

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>

* simplified git command

* Remove space

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>

* Updates openapi.yaml

---------

Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
2025-04-22 14:32:32 +00:00