473 Commits

Author SHA1 Message Date
Eric
b701e08a56
Merge branch 'feat/simple-optimizations-rebase' into release/1.0.0-prototype.1 2025-11-07 14:42:41 +11:00
Chrysostomos Nanakos
f7a3e90414
refactor: remove makeRandomDataset helper function
Remove makeRandomDataset in favor of direct composition with
makeRandomBlocks and makeDataset.

Part of https://github.com/codex-storage/nim-codex/issues/974

Signed-off-by: Chrysostomos Nanakos <chris@include.gr>
2025-11-03 14:10:53 +02:00
Chrysostomos Nanakos
6efdd4a068
chore: apply nph formatting
Part of https://github.com/codex-storage/nim-codex/issues/974

Signed-off-by: Chrysostomos Nanakos <chris@include.gr>
2025-11-03 14:10:52 +02:00
gmega
8e8d9f8e60
fix: randomize block refresh time, optimize context store checks 2025-11-03 14:10:46 +02:00
gmega
1fdb14f092
feat: add block knowledge request mechanism, implement tests 2025-11-03 14:10:45 +02:00
gmega
908d527dc9
fix: fix testdiscovery so it works with stricter block protocol 2025-11-03 14:10:44 +02:00
gmega
544ec123c7
fix: fix block exchange test to stricter protocol; minor refactor 2025-11-03 14:10:43 +02:00
gmega
50ab785662
feat: drop peer on activity timeout 2025-11-03 14:10:42 +02:00
gmega
38a0656cab
chore: remove file committed by accident 2025-11-03 14:10:41 +02:00
gmega
d0466ccf80
feat: remove quadratic joins in cancelBlocks; use SafeAsyncIterator for getBlocks; limit memory usage for fetchBatched when used as prefetcher 2025-11-03 14:10:39 +02:00
gmega
cc44052cf5
feat: add SafeAsyncIter chaining 2025-11-03 14:10:39 +02:00
gmega
b0b1c45376
fix: refresh timestamp before issuing request to prevent flood of knowledge updates 2025-11-03 14:10:38 +02:00
gmega
5a983347cb
adapt existing tests to new data structures, remove vestigial tests 2025-11-03 14:10:36 +02:00
gmega
9088566632
update engine tests; add BlockAddress hashing tests 2025-11-03 14:10:34 +02:00
Arnaud
6bf3c8bd01
Reraise cancelled error 2025-11-03 23:06:52 +11:00
Arnaud
d9f5261b80
Make rest api server optional 2025-11-03 23:06:51 +11:00
Arnaud
34976c3c58
Fix mock clock 2025-11-03 23:06:51 +11:00
Eric
f791a960f2
fix(ci): Windows SIGILL in CI (#1339) 2025-11-03 11:45:02 +00:00
Arnaud
db8f866db4
feat: check if CID exists in local store (#1331) 2025-11-02 04:32:47 +00:00
Eric
072bff5cab
fix: ci integration tests (#1335) 2025-10-30 19:38:11 +11:00
Arnaud
af55a761e6
chore: skip marketplace and long integration tests (#1326) 2025-10-22 19:22:33 +11:00
markspanbroek
01615354af
refactor(ci): run integration tests in parallel by spinning up more runners (#1287) 2025-06-25 08:56:16 +00:00
markspanbroek
e1c397e112
fix(tests): auto import all tests files and fix forgotten tests (#1281) 2025-06-23 11:18:59 +00:00
Arnaud
7b660e3554
chore(marketplace): use hardhat ignition (#1195) 2025-06-20 15:55:00 +00:00
Arnaud
d443df441d
chore: improve marketplace integration tests (#1268) 2025-06-19 06:36:10 +00:00
Chrysostomos Nanakos
b305e00160
Add support for slot reconstruction on unavailable slot detection (#1235)
Co-authored-by: Arnaud <arnaud@status.im>
2025-06-12 22:19:42 +00:00
markspanbroek
27d807a841
fix(sales): fix marketplace block expiry (#1258) 2025-06-11 11:27:09 +00:00
Ben Bierens
85823342e9
Improves logging in maintenance module and erasure. (#1264) 2025-06-10 13:27:52 +00:00
Adam Uhlíř
7502b9ad2c
feat(cirdl): auto-discovery of marketplace contract (#1259) 2025-06-09 10:04:15 +00:00
Eric
1bea94c390
fix(tests): fetching past contract events test (#1255) 2025-06-04 20:36:09 -07:00
markspanbroek
2dd436bfb7
fix(sales): do not crash when retrieving request fails (#1248) 2025-06-04 11:22:14 +00:00
Arnaud
45ade0e3c1
chore(marketplace): use canMarkProofAsMissing (#1188)
* Add canProofBeMarkedAsMissing

* Add more tests

* Update contracts submodule
2025-06-03 09:08:57 +00:00
Arnaud
ca869f6dce
fix(availabilities): use totalRemainingCollateral instead of totalCollateral for comparaison (#1229)
* Use totalRemainingCollateral instead of totalCollateral to compare the availability changes

* Update test to use totalRemainingCollateral instead of totalCollateral when testing OnAvailabilitySaved

* Reduce poll interval

* Fix flaky test

* Fix format
2025-06-02 16:47:12 +00:00
Giuliano Mega
d59c5b023c
chore: bump Nim to 2.2.4 (#1242)
* chore: bump Nim to 2.2.4

* fix: resolve symbol ambiguity and drop auto type

* fix: use reference to task instead of pointer or the compiler will deallocate `task` before the encoding/decoding is done

* fix: convention that maxCollateralPerByte equals totalRemainingCollateral when freeSize is 0 to avoid DivByZeroDefect

* fix: bump compiler version in CI pipeline as well
2025-05-29 16:37:38 -07:00
Arnaud
28a83db69e
chore: returns the collateral when a slot is reserved but not filled (#1216)
* Change token allowance method because increaseAllowance does not exist anymore

* Returns collateral when a reservation is deleted and not only a slot is filled

* Remove the returnedCollateral when the slot is not filled by the host

* Add returnedCollateral when the sale is ignored

* Add returnsCollateral variable for ignored state

* Rebase the contracts submodule on the master

* Add integration test

* Fix duration

* Remove unnecessary teardown function

* Remove misleading comment

* Get returned collateral from the request

* Enable logs to debug on CI

* Fix test

* Increase test timeout

* Fix typo

* Fix rebase
2025-05-29 14:47:37 +00:00
Arnaud
c689542579
fix: sales cleanup cancellation (#1234)
* fix(sales): handle cancellation of slot cleanup

Ensures that processing slots from the slot queue
continues even when cleanup of a slot is cancelled.

Co-Authored-By: Eric <5089238+emizzle@users.noreply.github.com>

* chore(reservations): add more `raises` annotations

* Fix cleanup cancellation

* Add remove-agent to trackedfutures instead of the cleanup function

* Increase the timeout to match the request expiry

* Enable logs to debug on CI

* Remove useless except and do not return when add item back to slot queue fails

* Reduce poll interval to detect sale cancelled state

* Avoid cancelling cleanup routine

* Do not cancel creating reservation in order to avoid inconsistent state

* Remove useless try except

---------

Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2025-05-29 06:57:05 +00:00
markspanbroek
25a8077e80
fix(integration): fix api validation test (#1201)
* integration: shutdown codex node at end of test

On Windows the codex node did not shut down properly after this test
finished.

* contracts: fix flaky test
2025-05-26 16:49:53 +00:00
Arnaud
f7d06cd0e8
chore(marketplace): switch to websocket (#1166)
* Switch to websocket

* Create resubscribe future

* Resubscribe websocket events after 5 minutes

* Remove the subscribe workaround and use define the resubscribe symbol

* Use localhost for ws url

* Define 240 seconds for resubscription interval

* Ensute that updates are sync when using ws
2025-05-23 14:13:19 +00:00
Marcin Czenko
748830570a
checked exceptions in stores (#1179)
* checked exceptions in stores

* makes asynciter as much exception safe as it gets

* introduce "SafeAsyncIter" that uses Results and limits exceptions to cancellations

* adds {.push raises: [].} to errors

* uses SafeAsyncIter in "listBlocks" and in "getBlockExpirations"

* simplifies safeasynciter (magic of auto)

* gets rid of ugly casts

* tiny fix in hte way we create raising futures in tests of safeasynciter

* Removes two more casts caused by using checked exceptions

* adds an extended explanation of one more complex SafeAsyncIter test

* adds missing "finishOnErr" param in slice constructor of SafeAsyncIter

* better fix for "Error: Exception can raise an unlisted exception: Exception" error.

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2025-05-21 21:17:04 +00:00
markspanbroek
bde98738c2
fix(slotqueue): simplify slot queue workers (#1224)
* fix(slotqueue): simplify slot queue workers

- worker is now just an async running loop
- instead of passing a "done" Future, use an
  AsyncEvent to signal completion

* chore(slotqueue): address review comments

Co-Authored-By: Eric <5089238+emizzle@users.noreply.github.com>
Co-Authored-By: Dmitriy Ryajov <dryajov@gmail.com>

---------

Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2025-05-15 13:02:04 +00:00
Adam Uhlíř
f144099377
fix(api): availability creation validation (#1212) 2025-05-14 08:46:16 +00:00
Eric
0f152d333c
chore: bump contracts to master (#1197)
Bump contracts to master branch.

There was a change that allowed hardhat to have multiple blocks with the same timestamp, so this needed to be reflected in two tests.
2025-04-17 06:13:43 +00:00
Marcin Czenko
5ec3b2b027
make sure we do not call "get" on unverified Result while fetching in batches (#1169)
* makes sure we do not call "get" on unverified result

* make handling of failed blocks in fetchBatched even more explicit

* simplifies allFinishedValues and makes it independent from allFinishedFailed

* only sleep if not iter.finished in fetchBatched
2025-03-31 04:57:55 +00:00
Marcin Czenko
0ec52abc98
fixes RandomChunker not respecting padding (#1170) 2025-03-31 04:48:22 +00:00
Arnaud
0032e60398
fix(marketplace): catch Marketplace_SlotIsFree and continue the cancelled process (#1139)
* Catch Marketplace_SlotIsFree and continue the cancelled process

* Add log message when the slot if free during failed state

* Reduce log level to debug for slot free error

* Separate slot mock errors

* Initialize variable in setyp

* Improve tests

* Remove non-meaningful checks and rename test

* Remove the Option in the error setters

* Return collateral when the state is cancelled only if the slot is filled by the host

* Do not propagate AsyncLockError

* Wrap contract error into specific error type

* Remove debug message

* Catch only SlotStateMismatchError in cancelled

* Fix error

* Remove returnBytesWas

* Use MarketError after raises pragma were defined

* Fix typo

* Fix lint
2025-03-26 15:17:39 +00:00
Arnaud
7deeb7d2b3
feat(marketplace): persistent availabilities (#1099)
* Add availability enabled parameter

* Return bytes to availability when finished

* Add until parameter

* Remove debug message

* Clean up and fix tests

* Update documentations and cleanup

* Avoid swallowing CancelledError

* Move until validation to reservations module

* Call onAvailabilityAdded callabck when the availability is enabled in sales

* Remove until validation in restapi when creating an availability

* Add openapi documentation

* Use results instead of stew/results (#1112)

* feat: request duration limit (#1057)

* feat: request duration limit

* Fix tests and duration type

* Add custom error

* Remove merge issue

* Update codex contracts eth

* Update market config and fix test

* Fix SlotReservationsConfig syntax

* Update dependencies

* test: remove doubled test

* chore: update contracts repo

---------

Co-authored-by: Arnaud <arnaud@status.im>

* fix(statemachine): do not raise from state.run (#1115)

* fix(statemachine): do not raise from state.run

* fix rebase

* fix exception handling in SaleProvingSimulated.prove

- re-raise CancelledError
- don't return State on CatchableError
- expect the Proofs_InvalidProof custom error instead of checking a string

* asyncSpawn salesagent.onCancelled

This was swallowing a KeyError in one of the tests (fixed in the previous commit)

* remove error handling states in asyncstatemachine

* revert unneeded changes

* formatting

* PR feedback, logging updates

* chore(integration): simplify block expiration integration test (#1100)

* chore(integration): simplify block expiration integration test

* clean up

* fix after rebase

* perf: contract storage optimizations (#1094)

* perf: contract storage optimizations

* Apply optimization changes

* Apply optimizing parameters sizing

* Update codex-contracts-eth

* bump latest changes in contracts branch

* Change requestDurationLimit to uint64

* fix tests

* fix tests

---------

Co-authored-by: Arnaud <arnaud@status.im>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>

* bump contracts to master (#1122)

* Add availability enabled parameter

* Return bytes to availability when finished

* Add until parameter

* Clean up and fix tests

* Move until validation to reservations module

* Apply suggestion changes: return the reservation module error

* Apply suggestion changes for until dates

* Apply suggestion changes: reorganize tests

* Fix indent

* Remove test related to timing issue

* Add raises errors to async pragram and remove useless try except

* Update open api documentation

* Fix wording

* Remove the httpClient restart statements

* Use market.getRequestEnd to set validUntil

* Remove returnBytes

* Use clock.now in testing

* Move the api validation file to the right file

---------

Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2025-03-26 11:45:22 +00:00
Arnaud
60b6996eb0
chore(marketplace): define raises for async pragma (#1165)
* Define raises for async pragma

* Update nim ethers

* Replace CatchableError by MarketError
2025-03-26 08:06:37 +00:00
Arnaud
a0d6fbaf02
chore(marketplace) - fix the http error codes when validating the availability requests (#1104)
* Use 422 http code when there is a validation error

* Update the open api description

* Fix typo

* Add more tests for total size

* Catch CancelledError because TrackedFuture raise no error

* Split rest api validation test to a new file

* Change the way of testing negative numbers

* Rename client variable and fix test status code

* Try to reduce the number of requests in CI when asserting in tests

* Fix rebase and remove safeEventually
2025-03-24 15:47:05 +00:00
Arnaud
709a8648fd
chore: add request validations (#1144)
* Add request validations

* Define expiry as required field in storage request params and fix tests

* Fix error messages

* Enable logs to figure out the issue with recurring failing test on macos

* Add custom errors raised by contract

* Remove custom error non existing anymore

* Update asynctest module

* Update timer tests after updating asynctest
2025-03-24 11:53:34 +00:00
Arnaud
9d7b521519
chore: add missing custom errors (#1134)
* Add missing custom errors

* Separate mock state errors

* Remove the Option in the error setters

* Wrap the contract errors in MarketError

* Remove async raises (needs to address it in another PR)

* Wrap contract errors into specific error types

* Rename SlotNotFreeError to SlotStateMismatchError
2025-03-18 07:06:46 +00:00