837 Commits

Author SHA1 Message Date
gmega
6891824226
fix: randomize block refresh time, optimize context store checks 2025-07-09 14:53:06 -03:00
gmega
2ada4b5eb3
feat: add block knowledge request mechanism, implement tests 2025-07-08 15:23:54 -03:00
gmega
d7c403edfe
feat: add stopgap "adaptive" refresh 2025-07-03 18:43:11 -03:00
gmega
b274b29b83
fix: fix testdiscovery so it works with stricter block protocol 2025-07-03 12:06:02 -03:00
gmega
572b44856b
fix: fix block exchange test to stricter protocol; minor refactor 2025-07-03 10:18:32 -03:00
gmega
97fd68e4a3
feat: drop peer on activity timeout 2025-07-02 20:17:06 -03:00
gmega
ad8e250605
feat: modify retry mechanism; add DHT guard rails; improve block cancellation handling 2025-07-01 21:20:14 -03:00
gmega
cdab172bae
chore: remove file committed by accident 2025-06-30 20:19:32 -03:00
gmega
83fd83ffa5
feat: allow futures to be returned out-of-order to decrease memory consumption 2025-06-30 20:10:56 -03:00
gmega
db279d8fa9
feat: remove quadratic joins in cancelBlocks; use SafeAsyncIterator for getBlocks; limit memory usage for fetchBatched when used as prefetcher 2025-06-30 17:31:03 -03:00
gmega
d94bfe60f6
feat: add SafeAsyncIter chaining 2025-06-27 16:04:49 -03:00
gmega
313d6bac1f
fix: refresh timestamp before issuing request to prevent flood of knowledge updates 2025-06-10 19:55:51 -03:00
Giuliano Mega
03a1cc70eb
Update codex/stores/networkstore.nim
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
2025-06-09 17:13:10 -03:00
gmega
fbd378ec18
adapt existing tests to new data structures, remove vestigial tests 2025-06-09 15:15:08 -03:00
gmega
e18996eed1
optimize remaining list joins so they're not quadratic 2025-06-09 11:26:16 -03:00
gmega
a53f8da855
replace list operations with sets 2025-06-06 15:09:41 -03:00
gmega
b6e120394a
update engine tests; add BlockAddress hashing tests 2025-06-06 14:01:51 -03:00
gmega
1dcb998be6
feat: cap how many blocks we can pack in a single message 2025-06-06 12:56:56 -03:00
gmega
d80b7ddbd0
switch nph branch to provisional which compiles in 2.2.x 2025-06-05 17:45:38 -03:00
gmega
41f94d7a73
feat: add dataset request batching 2025-06-05 15:30:08 -03:00
markspanbroek
2dd436bfb7
fix(sales): do not crash when retrieving request fails (#1248) 2025-06-04 11:22:14 +00:00
Arnaud
2e1306ac2d
chore: fix custom error handling when simulating invalid proofs (#1217)
* Fix custom error handling when simulating invalid proofs

* Update error message
2025-06-03 12:11:18 +00:00
Arnaud
45ade0e3c1
chore(marketplace): use canMarkProofAsMissing (#1188)
* Add canProofBeMarkedAsMissing

* Add more tests

* Update contracts submodule
2025-06-03 09:08:57 +00:00
Arnaud
ca869f6dce
fix(availabilities): use totalRemainingCollateral instead of totalCollateral for comparaison (#1229)
* Use totalRemainingCollateral instead of totalCollateral to compare the availability changes

* Update test to use totalRemainingCollateral instead of totalCollateral when testing OnAvailabilitySaved

* Reduce poll interval

* Fix flaky test

* Fix format
2025-06-02 16:47:12 +00:00
Slava
e43872d0b8
chore: update testnet marketplace address (#1245)
https://github.com/codex-storage/nim-codex/issues/1241
v0.2.3
2025-05-30 09:12:55 +00:00
Giuliano Mega
d59c5b023c
chore: bump Nim to 2.2.4 (#1242)
* chore: bump Nim to 2.2.4

* fix: resolve symbol ambiguity and drop auto type

* fix: use reference to task instead of pointer or the compiler will deallocate `task` before the encoding/decoding is done

* fix: convention that maxCollateralPerByte equals totalRemainingCollateral when freeSize is 0 to avoid DivByZeroDefect

* fix: bump compiler version in CI pipeline as well
2025-05-29 16:37:38 -07:00
Arnaud
28a83db69e
chore: returns the collateral when a slot is reserved but not filled (#1216)
* Change token allowance method because increaseAllowance does not exist anymore

* Returns collateral when a reservation is deleted and not only a slot is filled

* Remove the returnedCollateral when the slot is not filled by the host

* Add returnedCollateral when the sale is ignored

* Add returnsCollateral variable for ignored state

* Rebase the contracts submodule on the master

* Add integration test

* Fix duration

* Remove unnecessary teardown function

* Remove misleading comment

* Get returned collateral from the request

* Enable logs to debug on CI

* Fix test

* Increase test timeout

* Fix typo

* Fix rebase
2025-05-29 14:47:37 +00:00
Slava
13811825b3
ci: use macos arm runners (#1174)
* ci: use inputs instead of matrix in a ccache key

* ci: switch to arm runners for macos

* ci: use node 20

* ci: pass cpu to a composite action
2025-05-29 10:17:46 +00:00
Arnaud
827d9ccccf
Update contracts (#1238) 2025-05-29 08:27:41 +00:00
Arnaud
c689542579
fix: sales cleanup cancellation (#1234)
* fix(sales): handle cancellation of slot cleanup

Ensures that processing slots from the slot queue
continues even when cleanup of a slot is cancelled.

Co-Authored-By: Eric <5089238+emizzle@users.noreply.github.com>

* chore(reservations): add more `raises` annotations

* Fix cleanup cancellation

* Add remove-agent to trackedfutures instead of the cleanup function

* Increase the timeout to match the request expiry

* Enable logs to debug on CI

* Remove useless except and do not return when add item back to slot queue fails

* Reduce poll interval to detect sale cancelled state

* Avoid cancelling cleanup routine

* Do not cancel creating reservation in order to avoid inconsistent state

* Remove useless try except

---------

Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2025-05-29 06:57:05 +00:00
munna0908
71422f0d3d
fix: Support for mapping multiple listener address (#1236)
* multi address mapping support

* fix thread issues

* fix local thread var issue

* chore: rename stopNatThread to stopNatThreads

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2025-05-27 09:05:43 +00:00
markspanbroek
25a8077e80
fix(integration): fix api validation test (#1201)
* integration: shutdown codex node at end of test

On Windows the codex node did not shut down properly after this test
finished.

* contracts: fix flaky test
2025-05-26 16:49:53 +00:00
Ben Bierens
bfbd7264df
Adds missing async-raises for prover.verify (#1237) 2025-05-26 15:48:59 +00:00
Arnaud
f7d06cd0e8
chore(marketplace): switch to websocket (#1166)
* Switch to websocket

* Create resubscribe future

* Resubscribe websocket events after 5 minutes

* Remove the subscribe workaround and use define the resubscribe symbol

* Use localhost for ws url

* Define 240 seconds for resubscription interval

* Ensute that updates are sync when using ws
2025-05-23 14:13:19 +00:00
Marcin Czenko
748830570a
checked exceptions in stores (#1179)
* checked exceptions in stores

* makes asynciter as much exception safe as it gets

* introduce "SafeAsyncIter" that uses Results and limits exceptions to cancellations

* adds {.push raises: [].} to errors

* uses SafeAsyncIter in "listBlocks" and in "getBlockExpirations"

* simplifies safeasynciter (magic of auto)

* gets rid of ugly casts

* tiny fix in hte way we create raising futures in tests of safeasynciter

* Removes two more casts caused by using checked exceptions

* adds an extended explanation of one more complex SafeAsyncIter test

* adds missing "finishOnErr" param in slice constructor of SafeAsyncIter

* better fix for "Error: Exception can raise an unlisted exception: Exception" error.

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2025-05-21 21:17:04 +00:00
markspanbroek
bde98738c2
fix(slotqueue): simplify slot queue workers (#1224)
* fix(slotqueue): simplify slot queue workers

- worker is now just an async running loop
- instead of passing a "done" Future, use an
  AsyncEvent to signal completion

* chore(slotqueue): address review comments

Co-Authored-By: Eric <5089238+emizzle@users.noreply.github.com>
Co-Authored-By: Dmitriy Ryajov <dryajov@gmail.com>

---------

Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
v0.2.2
2025-05-15 13:02:04 +00:00
Adam Uhlíř
28e87d06cc
docs(openapi): freeSize non-optional (#1211) 2025-05-14 10:14:40 +00:00
Adam Uhlíř
f144099377
fix(api): availability creation validation (#1212) 2025-05-14 08:46:16 +00:00
Adam Uhlíř
19a5e05c13
docs(openapi): add local data delete endpoint (#1214)
* docs(openapi): add local data delete endpoint

* chore: feedback

Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>

---------

Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2025-05-03 16:54:38 +00:00
Slava
b39d541227
chore: update testnet marketplace address (#1209)
https://github.com/codex-storage/nim-codex/issues/1203
v0.2.1
2025-04-23 06:18:38 +00:00
Adam Uhlíř
d220e53fe1
ci: trigger python generator upon release (#1208) 2025-04-22 14:46:03 +00:00
Ben Bierens
2eb83a0ebb
Codex-contracts hash in version information. (#1207)
* Adds revision hash of codex-contracts to version information.

* Update codex/conf.nim

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>

* Update codex/conf.nim

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>

* Update codex/rest/api.nim

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>

* simplified git command

* Remove space

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>

* Updates openapi.yaml

---------

Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
2025-04-22 14:32:32 +00:00
Slava
22f5150d1d
ci: add compatible contracts image for nim-codex dist-tests docker image (#1204) 2025-04-18 14:21:24 +00:00
Eric
0f152d333c
chore: bump contracts to master (#1197)
Bump contracts to master branch.

There was a change that allowed hardhat to have multiple blocks with the same timestamp, so this needed to be reflected in two tests.
2025-04-17 06:13:43 +00:00
Adam Uhlíř
acf81d0571
chore: add marketplace topic to reservations (#1193) 2025-04-17 04:05:53 +00:00
Adam Uhlíř
7c7871ac75
ci: add compatible contracts image for nim-codex docker image (#1186)
* ci: add compatible contracts image for nim-codex docker image

* ci: with submodules

* ci: with submodules on correct place

* ci: remove double dash

* ci: avoiding artifact conflicts

* ci: add labels to arch images

* ci: correct way to add label to arch images

* ci: correct contract label

* ci: avoid building contracts image and use contracts commit hash

* refactor: better way to get the hash
2025-04-15 13:52:19 +00:00
markspanbroek
b92f79a654
Increase gas estimates (#1192)
* update nim-ethers to version 2.0.0

To allow for gas estimation of contract calls

* contracts: add 10% extra gas to contract calls

These calls could otherwise run out of gas because
the on-chain state may have changed between the time
of the estimate and the time of processing the
transaction.
2025-04-15 10:31:06 +00:00
Arnaud
6f62afef75
Apply changes to the openapi file (#1187) 2025-04-04 12:58:23 +00:00
Arnaud
4e2a321ad5
chore(openapi): add required parameters (#1178)
* Update the openapi file

* Fix typo

* Remove SalesAvailabilityCREATE and add collateralPerByte

* Fix SalesAvailability reference

* chore: adding perf optimization tweaks to openapi (#1182)

* chore: adding perf optimization tweaks to openapi

* chore: slotsize integer

---------

Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
2025-04-02 14:09:23 +00:00
Slava
1213377ac4
ci: switch out from ubuntu 20.04 (#1184)
* ci: use ubuntu-latest for coverage (#1141)

* ci: pass --keep-going to lcov and genhtml (#1141)

* ci: use ubuntu-22.04 for release workflow (#1141)

* ci: install gcc-14 on linux (#1141)

* chore: bump nim-leveldbstatic to 0.2.1
2025-04-02 09:09:43 +00:00