761 Commits

Author SHA1 Message Date
Eric
f6aee4ff6e
bump contracts to master (#1122) 2025-02-21 11:02:36 +00:00
Adam Uhlíř
44981d24d0
perf: contract storage optimizations (#1094)
* perf: contract storage optimizations

* Apply optimization changes

* Apply optimizing parameters sizing

* Update codex-contracts-eth

* bump latest changes in contracts branch

* Change requestDurationLimit to uint64

* fix tests

* fix tests

---------

Co-authored-by: Arnaud <arnaud@status.im>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2025-02-20 07:11:06 +00:00
Eric
04327a3986
chore(integration): simplify block expiration integration test (#1100)
* chore(integration): simplify block expiration integration test

* clean up

* fix after rebase
2025-02-20 05:52:51 +00:00
Eric
87590f43ce
fix(statemachine): do not raise from state.run (#1115)
* fix(statemachine): do not raise from state.run

* fix rebase

* fix exception handling in SaleProvingSimulated.prove

- re-raise CancelledError
- don't return State on CatchableError
- expect the Proofs_InvalidProof custom error instead of checking a string

* asyncSpawn salesagent.onCancelled

This was swallowing a KeyError in one of the tests (fixed in the previous commit)

* remove error handling states in asyncstatemachine

* revert unneeded changes

* formatting

* PR feedback, logging updates
2025-02-19 00:18:45 +00:00
Adam Uhlíř
1052dad30c
feat: request duration limit (#1057)
* feat: request duration limit

* Fix tests and duration type

* Add custom error

* Remove merge issue

* Update codex contracts eth

* Update market config and fix test

* Fix SlotReservationsConfig syntax

* Update dependencies

* test: remove doubled test

* chore: update contracts repo

---------

Co-authored-by: Arnaud <arnaud@status.im>
2025-02-18 19:41:54 +00:00
Arnaud
2298a0bf81
Use results instead of stew/results (#1112) 2025-02-18 10:17:05 +00:00
Arnaud
0107eb06fe
chore(marketplace): cid should be bytes (#1073)
* Change cid format from string to bytes for the marketplace

* refactor: marketplace custom errors handling

* chore: update contracts repo

* chore: update contracts submodule

* Update contracts submodule

* Initialize the Cid using init function

* Restorage serialize pragma

* Use Cid object instead of buffer

* Simplify cid usage

* Simplify cid usage

* bump codex-contracts-eth after PR merge, formatting

* fix rebase

* collateralPerByte => collateralPerSlot

---------

Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2025-02-18 05:47:47 +00:00
Arnaud
6e73338425
Remove deprecated function (#1111)
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2025-02-17 22:04:04 +00:00
Eric
5af3477793
chore(deps): bump ethers to propagate cancellations (#1116)
* chore(deps): bump ethers to propagate cancellations

Ethers was swallowing canellations and turning them into EthersErrors, which was causing the sales statemachine to error when it should have been simply cancelling the current state's run. Hopefully fixes the intermittently failing marketplace integration test.

* Add missing errors in async raises pragma

* bump to version of ethers that supports cancellations

---------

Co-authored-by: Arnaud <arnaud@status.im>
2025-02-17 22:00:52 +00:00
Arnaud
dc08ff8840
chore(marketplace): add a cache for storage requests (#1090)
* Add cache to for requests

* Change request cache description message and use const as default value

* Set request cache size configuration hidden
2025-02-17 10:34:42 +00:00
Giuliano Mega
25c84f4e0e
Fix/repostore deletes for non-orphan blocks (#1109)
* fix: fix deletion of non-orphan blocks

* feat: improve error feedback for illegal direct block deletes

* chore: minor rewording of test header
2025-02-14 13:34:17 +00:00
munna0908
c65148822e
feat: multithreading support for erasure coding (#1087)
* implement async encode

* implement async decode

* cleanup code

* add num-threads flag

* fix tests

* code cleanup

* improve return types and exception handling for async proc

* add validation check for numThreads flag

* modify encode method

* add new tests for aync encoding

* modify decode method

* cleanup test cases

* add new cli flag for threadCount

* test cleanup

* add new tests

* fix decodeAsync exception handling

* code cleanup

* chore: cosmetic changes
2025-02-12 17:56:26 +00:00
Dmitriy Ryajov
45e97513a7
remove uploadedAt from manifest (#1091)
* remove uploadedAt from manifest

* fix test
2025-02-12 10:48:58 +00:00
Slava
20f6fef7ab
fix: use ubuntu-24.04 runners for docker workflows (#1102)
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
2025-02-11 21:49:37 +00:00
Giuliano Mega
bbe1f09cd7
Purging of local datasets (#1103)
* feat(codex-node): add dataset deletion API to Codex node

* feat(api): add deletion of local datasets to API

* fix: logging, remove garbage, drop some CORS headers from DELETE request

* fix: change empty response return code to 204 instead of 200

* fix: add time-based idling to avoid locking up the node during deletes, fix API status code

* fix: uncomment commented tests committed by accident

* fix: return correct code when missing CID is a Manifest CID; add back CORS headers

* fix: remove lingering echo
2025-02-11 19:00:05 +00:00
Marcin Czenko
11888e78d7
deploy openapi spec only when tagged (#1106) 2025-02-11 15:16:45 +00:00
Marcin Czenko
8880ad9cd4
fix linting in "codex/blockexchange/engine/engine.nim" (#1107) 2025-02-11 10:47:25 +00:00
Eric
dfa90a9981
fix(build): compilation on macos when including nim-nat-traversal (#1084)
* fix(build): compilation on macos when including nim-nat-traversal

- removes the `VERSION` rename to `VERSION_temp` in the Makefile
- instead, relies on `-iqoute` to include the `nim-nat-traversal/vendor/libnatpmp-upstream` directory in the search paths. `-iquote` will match the `vendor/libnatpmp-upstream/VERSION` file for `#include "version"` and not `#include <version>`, the latter being what is included by the macos sdk and was causing issues with `-I`. The [gcc 14.2 docs](https://gcc.gnu.org/onlinedocs/gcc-14.2.0/cpp/Invocation.html#index-I) describe how `-iquote` alleviates this issue:
> Directories specified with -iquote apply only to the quote form of the directive, #include "file". Directories specified with -I, -isystem, or -idirafter apply to lookup for both the #include "file" and #include <file> directives.

For more info, please see https://github.com/status-im/nim-nat-traversal/pull/34.

* bump nim-nat-traversal

Now that https://github.com/status-im/nim-nat-traversal/pull/34 has been merged, change back to master commit
2025-02-07 01:18:00 +00:00
Dmitriy Ryajov
17d3f99f45
use a case-of instead of if for better readability (#1063) 2025-02-06 21:36:35 +00:00
Csaba Kiraly
e62a09d9b1
add ccache and sccache to speed up CI (#1074)
* add ccache and sccache to speed up CI

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* include testname and nim version in cache separation

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* Make sure ccache has precedence over custom clang/llvm

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* enable ccache for windows

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* ccache: evict old files

Make sure old unused cache files are not lingering around for long

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

---------

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
2025-02-06 21:36:14 +00:00
Marcin Czenko
c05eec422c
fix dataset and slot size calculations in integration tests (#1095)
* fixes datasetSize and slotSize helpers (and also RandomChunker.example)

* adds overload for <<upload>> for seq[byte]

* changes RandomChunker.example to return seq[byte]

* fixes restapi tests after correcting RandomChunker.example

* review: use string.fromBytes from nim-stew to convert seq[byte] to string
2025-02-06 15:21:12 +00:00
Slava
54d499be41
docker: add BOOTSTRAP_NODE_URL to Docker entrypoint (#1098) 2025-02-04 21:22:34 +00:00
Giuliano Mega
0595723f66
Minor improvements to download API (#1092)
* chore: improve error messages in upload API

* chore: remove unreachable (dead) code

* fix: API integration test
2025-02-04 16:01:14 +00:00
Arnaud
58a962add8
Update contracts and add custom errors (#1088) 2025-01-28 11:16:24 +00:00
Marcin Czenko
962fc1cd95
Feat: price per byte (#1078)
* openAPI: StorageRequestCreation: reward => pricePerByte, collateral => collateralPerByte

* purchasing: reward => pricePerByte, collateral => collateralPerByte

* Updates availabilities and reservations to use totalCollateral, minPricePerByte, and maxCollateralPerByte

* Uses correct div operator when operating on UInt256

* proposal updating totalCollateral in availability

* makes sure that reading currentCollateral happens before freeing slot

* Updates naming

* fixes tests: unit and contracts

* uses feat/price-per-byte branch for codex-contracts-eth

* temporarily disables integration tests on CI

* introduces high level <<totalCollateral>> property for a cleaner external interface

* updates integration tests

* Applies review comments

* Updates description of totalCollateral in SalesAvailability

* updates codex-contracts-eth (price-per-byte)
2025-01-24 17:18:00 +00:00
markspanbroek
f6c792de79
fix slotqueue worker starvation (#1081)
* fix slotqueue worker starvation

* improve slotqueue tests

Co-Authored-By: Marcin Czenko <marcin.czenko@pm.me>

* slotqueue nph formatting

---------

Co-authored-by: Marcin Czenko <marcin.czenko@pm.me>
2025-01-23 09:28:14 +00:00
Adam Uhlíř
1c4184f29c
chore: ignoring style formatting commit for git blame (#1085) 2025-01-22 09:05:47 +00:00
Adam Uhlíř
e5df8c50d3
style: nph formatting (#1067)
* style: nph setup

* chore: formates codex/ and tests/ folder with nph 0.6.1
2025-01-21 20:54:46 +00:00
markoburcul
d114e6e942 nix: build nim compiler
Using nimbus build system since the codex now uses version of Nim >2.
Tested with nim versions(>2) in nixpkgs but none of them work thus the
removal of USE_SYSTEM_NIM and its logic.
Signed-off-by: markoburcul <marko@status.im>
2025-01-21 16:45:33 +01:00
markoburcul
893f6d02ab jenkins: create initial pipeline
Referenced issue: https://github.com/codex-storage/nim-codex/issues/940

Signed-off-by: markoburcul <marko@status.im>
2025-01-21 16:45:33 +01:00
Arnaud
389ab59aa7
Use custom error message (#1079) 2025-01-20 16:04:50 +00:00
Giuliano Mega
ac12de37b2
chore: delete leftover code from old threaded erasure coding backend (#1080) 2025-01-20 14:50:33 +00:00
Slava
833e253baa
feat: switch to github arm runners (#1077)
* feat: switch to github arm runners

* chore: fabiocaccamo/create-matrix-action
2025-01-20 14:42:01 +00:00
Arnaud
2ad7c31c85
Remove duplicate header (#971)
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
2025-01-16 13:51:07 +00:00
Arnaud
4606726e27
Add default Content-Disposition header for download (#981) 2025-01-16 13:25:26 +00:00
Adam Uhlíř
68ad804f9e
refactor: marketplace custom errors handling (#1061)
* refactor: marketplace custom errors handling

* chore: update contracts repo

* chore: update contracts submodule
2025-01-16 09:34:44 +00:00
Marko Burčul
39e8e6e6fa
nix: update readme (#1064)
Include the instructions for running nim-codex as a systemd service on
NixOS.

Signed-off-by: markoburcul <marko@status.im>
2025-01-14 08:21:11 +00:00
markoburcul
0cffa02748
nix: add codex service definition
Referenced issue: https://github.com/codex-storage/nim-codex/issues/940

Signed-off-by: markoburcul <marko@status.im>
2025-01-10 17:49:20 +01:00
markoburcul
3dc7224330
nix: update nix packages and dependencies in flake
Signed-off-by: markoburcul <marko@status.im>
2025-01-10 17:49:18 +01:00
Arnaud
f25c555d59
Chore/update nim version (#1052)
* Move to version 2.0.6

* Update nim-confutils submodule to latest version

* Update dependencies

* Update Nim version to 2.0.12

* Add gcsafe pragma

* Add missing import

* Update specific conf for Nim 2.x

* Fix method signatures

* Revert erasure coding attempt to fix bug

* More gcsafe pragma

* Duplicate code from libp2p because it is not exported anymore

* Fix camelcase function names

* Use alreadySeen because need is not a bool anymore

* newLPStreamReadError does not exist anymore so use another error

* Replace ValidIpAddress by IpAddress

* Add gcsafe pragma

* Restore maintenance parameter deleted by mistake when removing esasure coding fix attempt code

* Update method signatures

* Copy LPStreamReadError code from libp2p which was removed

* Fix camel case

* Fix enums in tests

* Fix camel case

* Extract node components to a variable to make Nim 2 happy

* Update the tests using ValidIpAddress to IpAddress

* Fix cast for value which is already an option

* Set nim version to 2.0.x for CI

* Set nim version to 2.0.x for CI

* Move to miniupnp version 2.2.4 to avoid symlink error

* Set core.symlinks to false for Windows for miniupnp >= 2.2.5 support

* Update to Nim 2.0.14

* Update CI nim versions to 2.0.14

* Try with GCC 14

* Replace apt-fast by apt-get

* Update ubuntu runner to latest

* Use Ubuntu 20.04 for coverage

* Disable CI cache for coverage

* Add coverage property description

* Remove commented test

* Check the node value of seen instead of using alreadySeen

* Fix the merge. The taskpool work was reverted.

* Update nim-ethers submodule

* Remove deprecated ValidIpAddress. Fix missing case and imports.

* Fix a weird issue where nim-confutils cannot find NatAny

* Fix tests and remove useless static keyword
2025-01-10 14:12:37 +00:00
Ben Bierens
caed3c07a3
Fix sending of WantBlocks messages and tracking of peerWants (#1019)
* sends wantBlock to peers with block. wantHave to everyone else

* Cleanup cheapestPeer. Fixes test for peers lists

* Fixes issue where peerWants are only stored for type wantBlock.

* Review comments by Dmitriy

* consistent logging of addresses

* prevents duplicate scheduling. Fixes cancellation

* fast

* Marks cancel-presence situation with todo comment.

* fixtest: testsales enable logging

* Review by Dmitriy: Remember peerWants only if we don't have them.

* rework `wantListHandler` handling

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2025-01-09 22:44:02 +00:00
munna0908
74c46b3651
network: support for uPnP and PMP nat traversal (#1050)
* add support for uPnP and PMP nat traversal

* update cli flags

* merge with master

* use listener address from switch

* update build script to rename version file

* fix styling issues

* update docker files

- remove Disc_ip env
- update CODE_NAT parsing logic

* code cleanup

* move nat flag parsing logic to conf.nim
2025-01-09 18:11:22 +00:00
Adam Uhlíř
407f77871f
chore: warning cleanup (#1055)
* chore: warning cleanup

* chore: fix proper disabling of warning

* chore: ignore the import when not needed
2025-01-08 11:30:54 +00:00
Arnaud
145aa5d84a
chore: remove old fixme (#1060)
* Remove FIXME

* Fix indentation
2025-01-08 07:45:15 +00:00
Arnaud
0badcb662a
chore(openapi): add announceAddresses for debug (#994)
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
2025-01-08 07:34:54 +00:00
Adam Uhlíř
4b99b58645
fix: unhide log-format cli flag (#1049) 2024-12-19 12:03:50 +00:00
Eric
6ff4d30b43
fix(tests/integration): fix bootstrap node spr (#1048)
After a change in PR #1031, bootstrap node sprs may not work when Hardhat nodes are started with the tests. This fixes it by appending all started client's and provider's SPR to a sequence, and using that sequence of SPRs to start the next node. This means all subsequently started nodes will be connected to its previously started peers.

This also fixes the case when bootstrap SPRs would not be present if no clients were started.
2024-12-18 08:10:36 +00:00
Eric
8645d336ff
refactor(trackedfutures): remove return of future from tracked futures api (#1046)
- cleans up all instances of `.track` to use the `module.trackedfutures.track(future)` procedure, for better readability
- removes the `track` override that is no longer used in the codebase
2024-12-18 07:39:03 +00:00
Ben Bierens
20bb5e5a38
Applies multinodessuite in twonodessuite (#1031)
* Sets timeout for codexclient httpClient. Adds reliable transfer test.

* disable new test to check timeout setting in CI

* restores new test

* adds heartbeat log and logfile to ci output

* fixes suite

* fixes blocked output stream by switching to multinode fixture

* new twonodessuite based on multinodesuite

* Applies updated twonodessuite

* removes heartbeat log

* applies multinodesuite in testsales

* applies multinodesuite in testmarketplace

* fixes account fetch for host and client in testmarketplace

* adds waitTillNextPeriod at end of marketplace test

* Uses marketplacesuite in testmarketplace
2024-12-17 13:01:41 +00:00
Eric
c498e2f53b
fix(nodeprocess): asyncspawn capture output (#1045)
- Ensures no exceptions are raised from `captureOutput`
- Asyncspawns the future to ensure errors are not silently swallowed
2024-12-17 09:51:38 +00:00