411 Commits

Author SHA1 Message Date
Chrysostomos Nanakos
9e2f973b8a
refactor: make markRequested idempotent
Returns false on duplicate marking attempts instead of logging errors,
eliminating duplicate marking loop in blockPresenceHandler and
preventing duplicate block requests across concurrent flows.
2025-09-30 11:59:27 +03:00
Chrysostomos Nanakos
5efd08fb0e
fix: resolve stuck peer refresh state preventing block discovery
This prevents peers from becoming permanently invisible to block discovery when
they fail to respond to WantHave requests.
2025-09-30 11:59:27 +03:00
Chrysostomos Nanakos
25a2b3e9ed
feat: add strategic runtime metrics for block exchange monitoring
- Add codex_block_exchange_discovery_requests_total counter to track peer
  discovery frequency
- Add codex_block_exchange_peer_timeouts_total counter to monitor peer
  reliability issues
- Add codex_block_exchange_requests_failed_total counter to track request
  failure rates
2025-09-30 11:59:27 +03:00
Chrysostomos Nanakos
7f3004b5c0
perf: optimize block batch size from 500 to 50 blocks per message
Achieves significant memory reduction with equivalent network
performance. The reduced batch size prevents memory pressure
while preserving transfer efficiency, improving overall system
resource utilization.
2025-09-30 11:59:27 +03:00
Chrysostomos Nanakos
a7a384a66f
feat: add peer count limits to discovery engine
This prevents unbounded peer accumulation while maintaining redundancy.
When peer count exceeds the maximum, the least active peers are removed from
tracking to free resources.
2025-09-30 11:59:27 +03:00
Chrysostomos Nanakos
3a36e38b1f
perf: optimize further fetchBatched
- Reduce DefaultFetchBatch to prevent blockData explosion
- Add chunked onBatch processing to prevent memory accumulation
- Implement memory buffer reuse to reduce GC allocation churn
- Fix short-circuit evaluation
2025-09-30 11:59:27 +03:00
Chrysostomos Nanakos
524e93090d
feat: implement weighted random peer selection for load balancing
Use probabilistic distribution based on peer quality scores, giving all peers
opportunity while favoring better-performing ones. Selection probability is
inversely proportional to score.
2025-09-30 11:43:32 +03:00
Chrysostomos Nanakos
7cb4b31557
Implement load-balanced peer selection for block requests 2025-09-15 12:28:50 +03:00
gmega
6891824226
fix: randomize block refresh time, optimize context store checks 2025-07-09 14:53:06 -03:00
gmega
2ada4b5eb3
feat: add block knowledge request mechanism, implement tests 2025-07-08 15:23:54 -03:00
gmega
d7c403edfe
feat: add stopgap "adaptive" refresh 2025-07-03 18:43:11 -03:00
gmega
b274b29b83
fix: fix testdiscovery so it works with stricter block protocol 2025-07-03 12:06:02 -03:00
gmega
572b44856b
fix: fix block exchange test to stricter protocol; minor refactor 2025-07-03 10:18:32 -03:00
gmega
97fd68e4a3
feat: drop peer on activity timeout 2025-07-02 20:17:06 -03:00
gmega
ad8e250605
feat: modify retry mechanism; add DHT guard rails; improve block cancellation handling 2025-07-01 21:20:14 -03:00
gmega
83fd83ffa5
feat: allow futures to be returned out-of-order to decrease memory consumption 2025-06-30 20:10:56 -03:00
gmega
db279d8fa9
feat: remove quadratic joins in cancelBlocks; use SafeAsyncIterator for getBlocks; limit memory usage for fetchBatched when used as prefetcher 2025-06-30 17:31:03 -03:00
gmega
d94bfe60f6
feat: add SafeAsyncIter chaining 2025-06-27 16:04:49 -03:00
gmega
313d6bac1f
fix: refresh timestamp before issuing request to prevent flood of knowledge updates 2025-06-10 19:55:51 -03:00
Giuliano Mega
03a1cc70eb
Update codex/stores/networkstore.nim
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
2025-06-09 17:13:10 -03:00
gmega
e18996eed1
optimize remaining list joins so they're not quadratic 2025-06-09 11:26:16 -03:00
gmega
a53f8da855
replace list operations with sets 2025-06-06 15:09:41 -03:00
gmega
b6e120394a
update engine tests; add BlockAddress hashing tests 2025-06-06 14:01:51 -03:00
gmega
1dcb998be6
feat: cap how many blocks we can pack in a single message 2025-06-06 12:56:56 -03:00
gmega
41f94d7a73
feat: add dataset request batching 2025-06-05 15:30:08 -03:00
markspanbroek
2dd436bfb7
fix(sales): do not crash when retrieving request fails (#1248) 2025-06-04 11:22:14 +00:00
Arnaud
2e1306ac2d
chore: fix custom error handling when simulating invalid proofs (#1217)
* Fix custom error handling when simulating invalid proofs

* Update error message
2025-06-03 12:11:18 +00:00
Arnaud
45ade0e3c1
chore(marketplace): use canMarkProofAsMissing (#1188)
* Add canProofBeMarkedAsMissing

* Add more tests

* Update contracts submodule
2025-06-03 09:08:57 +00:00
Arnaud
ca869f6dce
fix(availabilities): use totalRemainingCollateral instead of totalCollateral for comparaison (#1229)
* Use totalRemainingCollateral instead of totalCollateral to compare the availability changes

* Update test to use totalRemainingCollateral instead of totalCollateral when testing OnAvailabilitySaved

* Reduce poll interval

* Fix flaky test

* Fix format
2025-06-02 16:47:12 +00:00
Slava
e43872d0b8
chore: update testnet marketplace address (#1245)
https://github.com/codex-storage/nim-codex/issues/1241
2025-05-30 09:12:55 +00:00
Giuliano Mega
d59c5b023c
chore: bump Nim to 2.2.4 (#1242)
* chore: bump Nim to 2.2.4

* fix: resolve symbol ambiguity and drop auto type

* fix: use reference to task instead of pointer or the compiler will deallocate `task` before the encoding/decoding is done

* fix: convention that maxCollateralPerByte equals totalRemainingCollateral when freeSize is 0 to avoid DivByZeroDefect

* fix: bump compiler version in CI pipeline as well
2025-05-29 16:37:38 -07:00
Arnaud
28a83db69e
chore: returns the collateral when a slot is reserved but not filled (#1216)
* Change token allowance method because increaseAllowance does not exist anymore

* Returns collateral when a reservation is deleted and not only a slot is filled

* Remove the returnedCollateral when the slot is not filled by the host

* Add returnedCollateral when the sale is ignored

* Add returnsCollateral variable for ignored state

* Rebase the contracts submodule on the master

* Add integration test

* Fix duration

* Remove unnecessary teardown function

* Remove misleading comment

* Get returned collateral from the request

* Enable logs to debug on CI

* Fix test

* Increase test timeout

* Fix typo

* Fix rebase
2025-05-29 14:47:37 +00:00
Arnaud
827d9ccccf
Update contracts (#1238) 2025-05-29 08:27:41 +00:00
Arnaud
c689542579
fix: sales cleanup cancellation (#1234)
* fix(sales): handle cancellation of slot cleanup

Ensures that processing slots from the slot queue
continues even when cleanup of a slot is cancelled.

Co-Authored-By: Eric <5089238+emizzle@users.noreply.github.com>

* chore(reservations): add more `raises` annotations

* Fix cleanup cancellation

* Add remove-agent to trackedfutures instead of the cleanup function

* Increase the timeout to match the request expiry

* Enable logs to debug on CI

* Remove useless except and do not return when add item back to slot queue fails

* Reduce poll interval to detect sale cancelled state

* Avoid cancelling cleanup routine

* Do not cancel creating reservation in order to avoid inconsistent state

* Remove useless try except

---------

Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2025-05-29 06:57:05 +00:00
munna0908
71422f0d3d
fix: Support for mapping multiple listener address (#1236)
* multi address mapping support

* fix thread issues

* fix local thread var issue

* chore: rename stopNatThread to stopNatThreads

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2025-05-27 09:05:43 +00:00
Ben Bierens
bfbd7264df
Adds missing async-raises for prover.verify (#1237) 2025-05-26 15:48:59 +00:00
Marcin Czenko
748830570a
checked exceptions in stores (#1179)
* checked exceptions in stores

* makes asynciter as much exception safe as it gets

* introduce "SafeAsyncIter" that uses Results and limits exceptions to cancellations

* adds {.push raises: [].} to errors

* uses SafeAsyncIter in "listBlocks" and in "getBlockExpirations"

* simplifies safeasynciter (magic of auto)

* gets rid of ugly casts

* tiny fix in hte way we create raising futures in tests of safeasynciter

* Removes two more casts caused by using checked exceptions

* adds an extended explanation of one more complex SafeAsyncIter test

* adds missing "finishOnErr" param in slice constructor of SafeAsyncIter

* better fix for "Error: Exception can raise an unlisted exception: Exception" error.

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2025-05-21 21:17:04 +00:00
markspanbroek
bde98738c2
fix(slotqueue): simplify slot queue workers (#1224)
* fix(slotqueue): simplify slot queue workers

- worker is now just an async running loop
- instead of passing a "done" Future, use an
  AsyncEvent to signal completion

* chore(slotqueue): address review comments

Co-Authored-By: Eric <5089238+emizzle@users.noreply.github.com>
Co-Authored-By: Dmitriy Ryajov <dryajov@gmail.com>

---------

Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2025-05-15 13:02:04 +00:00
Adam Uhlíř
f144099377
fix(api): availability creation validation (#1212) 2025-05-14 08:46:16 +00:00
Adam Uhlíř
19a5e05c13
docs(openapi): add local data delete endpoint (#1214)
* docs(openapi): add local data delete endpoint

* chore: feedback

Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>

---------

Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2025-05-03 16:54:38 +00:00
Slava
b39d541227
chore: update testnet marketplace address (#1209)
https://github.com/codex-storage/nim-codex/issues/1203
2025-04-23 06:18:38 +00:00
Ben Bierens
2eb83a0ebb
Codex-contracts hash in version information. (#1207)
* Adds revision hash of codex-contracts to version information.

* Update codex/conf.nim

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>

* Update codex/conf.nim

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>

* Update codex/rest/api.nim

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>

* simplified git command

* Remove space

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>

* Updates openapi.yaml

---------

Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
2025-04-22 14:32:32 +00:00
Adam Uhlíř
acf81d0571
chore: add marketplace topic to reservations (#1193) 2025-04-17 04:05:53 +00:00
markspanbroek
b92f79a654
Increase gas estimates (#1192)
* update nim-ethers to version 2.0.0

To allow for gas estimation of contract calls

* contracts: add 10% extra gas to contract calls

These calls could otherwise run out of gas because
the on-chain state may have changed between the time
of the estimate and the time of processing the
transaction.
2025-04-15 10:31:06 +00:00
munna0908
e9c6d19873
use constantine sha256 for codex tree hashing (#1168) 2025-03-31 06:41:08 +00:00
Marcin Czenko
5ec3b2b027
make sure we do not call "get" on unverified Result while fetching in batches (#1169)
* makes sure we do not call "get" on unverified result

* make handling of failed blocks in fetchBatched even more explicit

* simplifies allFinishedValues and makes it independent from allFinishedFailed

* only sleep if not iter.finished in fetchBatched
2025-03-31 04:57:55 +00:00
Arnaud
0032e60398
fix(marketplace): catch Marketplace_SlotIsFree and continue the cancelled process (#1139)
* Catch Marketplace_SlotIsFree and continue the cancelled process

* Add log message when the slot if free during failed state

* Reduce log level to debug for slot free error

* Separate slot mock errors

* Initialize variable in setyp

* Improve tests

* Remove non-meaningful checks and rename test

* Remove the Option in the error setters

* Return collateral when the state is cancelled only if the slot is filled by the host

* Do not propagate AsyncLockError

* Wrap contract error into specific error type

* Remove debug message

* Catch only SlotStateMismatchError in cancelled

* Fix error

* Remove returnBytesWas

* Use MarketError after raises pragma were defined

* Fix typo

* Fix lint
2025-03-26 15:17:39 +00:00
Arnaud
7deeb7d2b3
feat(marketplace): persistent availabilities (#1099)
* Add availability enabled parameter

* Return bytes to availability when finished

* Add until parameter

* Remove debug message

* Clean up and fix tests

* Update documentations and cleanup

* Avoid swallowing CancelledError

* Move until validation to reservations module

* Call onAvailabilityAdded callabck when the availability is enabled in sales

* Remove until validation in restapi when creating an availability

* Add openapi documentation

* Use results instead of stew/results (#1112)

* feat: request duration limit (#1057)

* feat: request duration limit

* Fix tests and duration type

* Add custom error

* Remove merge issue

* Update codex contracts eth

* Update market config and fix test

* Fix SlotReservationsConfig syntax

* Update dependencies

* test: remove doubled test

* chore: update contracts repo

---------

Co-authored-by: Arnaud <arnaud@status.im>

* fix(statemachine): do not raise from state.run (#1115)

* fix(statemachine): do not raise from state.run

* fix rebase

* fix exception handling in SaleProvingSimulated.prove

- re-raise CancelledError
- don't return State on CatchableError
- expect the Proofs_InvalidProof custom error instead of checking a string

* asyncSpawn salesagent.onCancelled

This was swallowing a KeyError in one of the tests (fixed in the previous commit)

* remove error handling states in asyncstatemachine

* revert unneeded changes

* formatting

* PR feedback, logging updates

* chore(integration): simplify block expiration integration test (#1100)

* chore(integration): simplify block expiration integration test

* clean up

* fix after rebase

* perf: contract storage optimizations (#1094)

* perf: contract storage optimizations

* Apply optimization changes

* Apply optimizing parameters sizing

* Update codex-contracts-eth

* bump latest changes in contracts branch

* Change requestDurationLimit to uint64

* fix tests

* fix tests

---------

Co-authored-by: Arnaud <arnaud@status.im>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>

* bump contracts to master (#1122)

* Add availability enabled parameter

* Return bytes to availability when finished

* Add until parameter

* Clean up and fix tests

* Move until validation to reservations module

* Apply suggestion changes: return the reservation module error

* Apply suggestion changes for until dates

* Apply suggestion changes: reorganize tests

* Fix indent

* Remove test related to timing issue

* Add raises errors to async pragram and remove useless try except

* Update open api documentation

* Fix wording

* Remove the httpClient restart statements

* Use market.getRequestEnd to set validUntil

* Remove returnBytes

* Use clock.now in testing

* Move the api validation file to the right file

---------

Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2025-03-26 11:45:22 +00:00
Arnaud
60b6996eb0
chore(marketplace): define raises for async pragma (#1165)
* Define raises for async pragma

* Update nim ethers

* Replace CatchableError by MarketError
2025-03-26 08:06:37 +00:00
Arnaud
a0d6fbaf02
chore(marketplace) - fix the http error codes when validating the availability requests (#1104)
* Use 422 http code when there is a validation error

* Update the open api description

* Fix typo

* Add more tests for total size

* Catch CancelledError because TrackedFuture raise no error

* Split rest api validation test to a new file

* Change the way of testing negative numbers

* Rename client variable and fix test status code

* Try to reduce the number of requests in CI when asserting in tests

* Fix rebase and remove safeEventually
2025-03-24 15:47:05 +00:00