773 Commits

Author SHA1 Message Date
Adam Uhlíř
d2d8f64989
perf: contract storage optimizations (#1094)
* perf: contract storage optimizations

* Apply optimization changes

* Apply optimizing parameters sizing

* Update codex-contracts-eth

* bump latest changes in contracts branch

* Change requestDurationLimit to uint64

* fix tests

* fix tests

---------

Co-authored-by: Arnaud <arnaud@status.im>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2025-02-24 11:24:47 +01:00
Eric
b294af57ba
chore(integration): simplify block expiration integration test (#1100)
* chore(integration): simplify block expiration integration test

* clean up

* fix after rebase
2025-02-24 11:01:38 +01:00
Eric
1220ada0d6
fix(statemachine): do not raise from state.run (#1115)
* fix(statemachine): do not raise from state.run

* fix rebase

* fix exception handling in SaleProvingSimulated.prove

- re-raise CancelledError
- don't return State on CatchableError
- expect the Proofs_InvalidProof custom error instead of checking a string

* asyncSpawn salesagent.onCancelled

This was swallowing a KeyError in one of the tests (fixed in the previous commit)

* remove error handling states in asyncstatemachine

* revert unneeded changes

* formatting

* PR feedback, logging updates
2025-02-24 11:01:38 +01:00
Adam Uhlíř
84b948278a
feat: request duration limit (#1057)
* feat: request duration limit

* Fix tests and duration type

* Add custom error

* Remove merge issue

* Update codex contracts eth

* Update market config and fix test

* Fix SlotReservationsConfig syntax

* Update dependencies

* test: remove doubled test

* chore: update contracts repo

---------

Co-authored-by: Arnaud <arnaud@status.im>
2025-02-24 10:56:56 +01:00
Arnaud
a93fa3b7bc
Use results instead of stew/results (#1112) 2025-02-24 10:56:56 +01:00
Arnaud
d045a7e6e6
Add openapi documentation 2025-02-21 17:42:45 +01:00
Arnaud
dba109fda1
Remove until validation in restapi when creating an availability 2025-02-21 17:09:34 +01:00
Arnaud
2b02d38b45
Call onAvailabilityAdded callabck when the availability is enabled in sales 2025-02-21 17:03:41 +01:00
Arnaud
ada1a6f865
Move until validation to reservations module 2025-02-21 17:03:09 +01:00
Arnaud
c62a023191
Merge branch 'feat/persistent-availabilities' of github.com:codex-storage/nim-codex into feat/persistent-availabilities 2025-02-20 10:09:51 +01:00
Arnaud
d2ccc4778a
Avoid swallowing CancelledError 2025-02-19 09:38:19 +01:00
Arnaud
f9e5342637
Merge branch 'master' into feat/persistent-availabilities 2025-02-18 11:19:16 +01:00
Arnaud
0107eb06fe
chore(marketplace): cid should be bytes (#1073)
* Change cid format from string to bytes for the marketplace

* refactor: marketplace custom errors handling

* chore: update contracts repo

* chore: update contracts submodule

* Update contracts submodule

* Initialize the Cid using init function

* Restorage serialize pragma

* Use Cid object instead of buffer

* Simplify cid usage

* Simplify cid usage

* bump codex-contracts-eth after PR merge, formatting

* fix rebase

* collateralPerByte => collateralPerSlot

---------

Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2025-02-18 05:47:47 +00:00
Arnaud
6e73338425
Remove deprecated function (#1111)
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2025-02-17 22:04:04 +00:00
Eric
5af3477793
chore(deps): bump ethers to propagate cancellations (#1116)
* chore(deps): bump ethers to propagate cancellations

Ethers was swallowing canellations and turning them into EthersErrors, which was causing the sales statemachine to error when it should have been simply cancelling the current state's run. Hopefully fixes the intermittently failing marketplace integration test.

* Add missing errors in async raises pragma

* bump to version of ethers that supports cancellations

---------

Co-authored-by: Arnaud <arnaud@status.im>
2025-02-17 22:00:52 +00:00
Arnaud
dc08ff8840
chore(marketplace): add a cache for storage requests (#1090)
* Add cache to for requests

* Change request cache description message and use const as default value

* Set request cache size configuration hidden
2025-02-17 10:34:42 +00:00
Giuliano Mega
25c84f4e0e
Fix/repostore deletes for non-orphan blocks (#1109)
* fix: fix deletion of non-orphan blocks

* feat: improve error feedback for illegal direct block deletes

* chore: minor rewording of test header
2025-02-14 13:34:17 +00:00
munna0908
c65148822e
feat: multithreading support for erasure coding (#1087)
* implement async encode

* implement async decode

* cleanup code

* add num-threads flag

* fix tests

* code cleanup

* improve return types and exception handling for async proc

* add validation check for numThreads flag

* modify encode method

* add new tests for aync encoding

* modify decode method

* cleanup test cases

* add new cli flag for threadCount

* test cleanup

* add new tests

* fix decodeAsync exception handling

* code cleanup

* chore: cosmetic changes
2025-02-12 17:56:26 +00:00
Dmitriy Ryajov
45e97513a7
remove uploadedAt from manifest (#1091)
* remove uploadedAt from manifest

* fix test
2025-02-12 10:48:58 +00:00
Slava
20f6fef7ab
fix: use ubuntu-24.04 runners for docker workflows (#1102)
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
2025-02-11 21:49:37 +00:00
Giuliano Mega
bbe1f09cd7
Purging of local datasets (#1103)
* feat(codex-node): add dataset deletion API to Codex node

* feat(api): add deletion of local datasets to API

* fix: logging, remove garbage, drop some CORS headers from DELETE request

* fix: change empty response return code to 204 instead of 200

* fix: add time-based idling to avoid locking up the node during deletes, fix API status code

* fix: uncomment commented tests committed by accident

* fix: return correct code when missing CID is a Manifest CID; add back CORS headers

* fix: remove lingering echo
2025-02-11 19:00:05 +00:00
Marcin Czenko
11888e78d7
deploy openapi spec only when tagged (#1106) 2025-02-11 15:16:45 +00:00
Marcin Czenko
8880ad9cd4
fix linting in "codex/blockexchange/engine/engine.nim" (#1107) 2025-02-11 10:47:25 +00:00
Eric
dfa90a9981
fix(build): compilation on macos when including nim-nat-traversal (#1084)
* fix(build): compilation on macos when including nim-nat-traversal

- removes the `VERSION` rename to `VERSION_temp` in the Makefile
- instead, relies on `-iqoute` to include the `nim-nat-traversal/vendor/libnatpmp-upstream` directory in the search paths. `-iquote` will match the `vendor/libnatpmp-upstream/VERSION` file for `#include "version"` and not `#include <version>`, the latter being what is included by the macos sdk and was causing issues with `-I`. The [gcc 14.2 docs](https://gcc.gnu.org/onlinedocs/gcc-14.2.0/cpp/Invocation.html#index-I) describe how `-iquote` alleviates this issue:
> Directories specified with -iquote apply only to the quote form of the directive, #include "file". Directories specified with -I, -isystem, or -idirafter apply to lookup for both the #include "file" and #include <file> directives.

For more info, please see https://github.com/status-im/nim-nat-traversal/pull/34.

* bump nim-nat-traversal

Now that https://github.com/status-im/nim-nat-traversal/pull/34 has been merged, change back to master commit
2025-02-07 01:18:00 +00:00
Dmitriy Ryajov
17d3f99f45
use a case-of instead of if for better readability (#1063) 2025-02-06 21:36:35 +00:00
Csaba Kiraly
e62a09d9b1
add ccache and sccache to speed up CI (#1074)
* add ccache and sccache to speed up CI

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* include testname and nim version in cache separation

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* Make sure ccache has precedence over custom clang/llvm

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* enable ccache for windows

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* ccache: evict old files

Make sure old unused cache files are not lingering around for long

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

---------

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
2025-02-06 21:36:14 +00:00
Marcin Czenko
c05eec422c
fix dataset and slot size calculations in integration tests (#1095)
* fixes datasetSize and slotSize helpers (and also RandomChunker.example)

* adds overload for <<upload>> for seq[byte]

* changes RandomChunker.example to return seq[byte]

* fixes restapi tests after correcting RandomChunker.example

* review: use string.fromBytes from nim-stew to convert seq[byte] to string
2025-02-06 15:21:12 +00:00
Arnaud
ffd149fba8
Update documentations and cleanup 2025-02-06 15:17:06 +01:00
Arnaud
9764d3f133
Clean up and fix tests 2025-02-06 14:54:04 +01:00
Arnaud
ac24f35250
Remove debug message 2025-02-06 08:30:29 +01:00
Arnaud
904d20c48f
Add until parameter 2025-02-05 20:52:38 +01:00
Slava
54d499be41
docker: add BOOTSTRAP_NODE_URL to Docker entrypoint (#1098) 2025-02-04 21:22:34 +00:00
Arnaud
7257f1768f
Return bytes to availability when finished 2025-02-04 19:29:07 +01:00
Giuliano Mega
0595723f66
Minor improvements to download API (#1092)
* chore: improve error messages in upload API

* chore: remove unreachable (dead) code

* fix: API integration test
2025-02-04 16:01:14 +00:00
Arnaud
b80b1f85be
Add availability enabled parameter 2025-01-29 17:20:02 +01:00
Arnaud
58a962add8
Update contracts and add custom errors (#1088) 2025-01-28 11:16:24 +00:00
Marcin Czenko
962fc1cd95
Feat: price per byte (#1078)
* openAPI: StorageRequestCreation: reward => pricePerByte, collateral => collateralPerByte

* purchasing: reward => pricePerByte, collateral => collateralPerByte

* Updates availabilities and reservations to use totalCollateral, minPricePerByte, and maxCollateralPerByte

* Uses correct div operator when operating on UInt256

* proposal updating totalCollateral in availability

* makes sure that reading currentCollateral happens before freeing slot

* Updates naming

* fixes tests: unit and contracts

* uses feat/price-per-byte branch for codex-contracts-eth

* temporarily disables integration tests on CI

* introduces high level <<totalCollateral>> property for a cleaner external interface

* updates integration tests

* Applies review comments

* Updates description of totalCollateral in SalesAvailability

* updates codex-contracts-eth (price-per-byte)
2025-01-24 17:18:00 +00:00
markspanbroek
f6c792de79
fix slotqueue worker starvation (#1081)
* fix slotqueue worker starvation

* improve slotqueue tests

Co-Authored-By: Marcin Czenko <marcin.czenko@pm.me>

* slotqueue nph formatting

---------

Co-authored-by: Marcin Czenko <marcin.czenko@pm.me>
2025-01-23 09:28:14 +00:00
Adam Uhlíř
1c4184f29c
chore: ignoring style formatting commit for git blame (#1085) 2025-01-22 09:05:47 +00:00
Adam Uhlíř
e5df8c50d3
style: nph formatting (#1067)
* style: nph setup

* chore: formates codex/ and tests/ folder with nph 0.6.1
2025-01-21 20:54:46 +00:00
d114e6e942 nix: build nim compiler
Using nimbus build system since the codex now uses version of Nim >2.
Tested with nim versions(>2) in nixpkgs but none of them work thus the
removal of USE_SYSTEM_NIM and its logic.
Signed-off-by: markoburcul <marko@status.im>
2025-01-21 16:45:33 +01:00
893f6d02ab jenkins: create initial pipeline
Referenced issue: https://github.com/codex-storage/nim-codex/issues/940

Signed-off-by: markoburcul <marko@status.im>
2025-01-21 16:45:33 +01:00
Arnaud
389ab59aa7
Use custom error message (#1079) 2025-01-20 16:04:50 +00:00
Giuliano Mega
ac12de37b2
chore: delete leftover code from old threaded erasure coding backend (#1080) 2025-01-20 14:50:33 +00:00
Slava
833e253baa
feat: switch to github arm runners (#1077)
* feat: switch to github arm runners

* chore: fabiocaccamo/create-matrix-action
2025-01-20 14:42:01 +00:00
Arnaud
2ad7c31c85
Remove duplicate header (#971)
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
2025-01-16 13:51:07 +00:00
Arnaud
4606726e27
Add default Content-Disposition header for download (#981) 2025-01-16 13:25:26 +00:00
Adam Uhlíř
68ad804f9e
refactor: marketplace custom errors handling (#1061)
* refactor: marketplace custom errors handling

* chore: update contracts repo

* chore: update contracts submodule
2025-01-16 09:34:44 +00:00
Marko Burčul
39e8e6e6fa
nix: update readme (#1064)
Include the instructions for running nim-codex as a systemd service on
NixOS.

Signed-off-by: markoburcul <marko@status.im>
2025-01-14 08:21:11 +00:00
0cffa02748
nix: add codex service definition
Referenced issue: https://github.com/codex-storage/nim-codex/issues/940

Signed-off-by: markoburcul <marko@status.im>
2025-01-10 17:49:20 +01:00