366 Commits

Author SHA1 Message Date
Arnaud
e558fd8f94
Enable http pipeline 2025-02-27 10:48:59 +01:00
Arnaud
f06b7ffcae
Add raises errors to async pragram and remove useless try except 2025-02-25 13:58:31 +01:00
Arnaud
d851b3a9ca
Fix indent 2025-02-24 15:07:50 +01:00
Arnaud
a18e879705
Apply suggestion changes for until dates 2025-02-24 11:48:37 +01:00
Arnaud
b64aa5042b
Apply suggestion changes: return the reservation module error 2025-02-24 11:48:36 +01:00
Arnaud
c437ccf9a7
Move until validation to reservations module 2025-02-24 11:48:32 +01:00
Arnaud
5b564b14e1
Clean up and fix tests 2025-02-24 11:46:36 +01:00
Arnaud
ff9b01a640
Add until parameter 2025-02-24 11:45:37 +01:00
Adam Uhlíř
d2d8f64989
perf: contract storage optimizations (#1094)
* perf: contract storage optimizations

* Apply optimization changes

* Apply optimizing parameters sizing

* Update codex-contracts-eth

* bump latest changes in contracts branch

* Change requestDurationLimit to uint64

* fix tests

* fix tests

---------

Co-authored-by: Arnaud <arnaud@status.im>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2025-02-24 11:24:47 +01:00
Eric
1220ada0d6
fix(statemachine): do not raise from state.run (#1115)
* fix(statemachine): do not raise from state.run

* fix rebase

* fix exception handling in SaleProvingSimulated.prove

- re-raise CancelledError
- don't return State on CatchableError
- expect the Proofs_InvalidProof custom error instead of checking a string

* asyncSpawn salesagent.onCancelled

This was swallowing a KeyError in one of the tests (fixed in the previous commit)

* remove error handling states in asyncstatemachine

* revert unneeded changes

* formatting

* PR feedback, logging updates
2025-02-24 11:01:38 +01:00
Adam Uhlíř
84b948278a
feat: request duration limit (#1057)
* feat: request duration limit

* Fix tests and duration type

* Add custom error

* Remove merge issue

* Update codex contracts eth

* Update market config and fix test

* Fix SlotReservationsConfig syntax

* Update dependencies

* test: remove doubled test

* chore: update contracts repo

---------

Co-authored-by: Arnaud <arnaud@status.im>
2025-02-24 10:56:56 +01:00
Arnaud
a93fa3b7bc
Use results instead of stew/results (#1112) 2025-02-24 10:56:56 +01:00
Arnaud
dba109fda1
Remove until validation in restapi when creating an availability 2025-02-21 17:09:34 +01:00
Arnaud
2b02d38b45
Call onAvailabilityAdded callabck when the availability is enabled in sales 2025-02-21 17:03:41 +01:00
Arnaud
ada1a6f865
Move until validation to reservations module 2025-02-21 17:03:09 +01:00
Arnaud
c62a023191
Merge branch 'feat/persistent-availabilities' of github.com:codex-storage/nim-codex into feat/persistent-availabilities 2025-02-20 10:09:51 +01:00
Arnaud
d2ccc4778a
Avoid swallowing CancelledError 2025-02-19 09:38:19 +01:00
Arnaud
f9e5342637
Merge branch 'master' into feat/persistent-availabilities 2025-02-18 11:19:16 +01:00
Arnaud
0107eb06fe
chore(marketplace): cid should be bytes (#1073)
* Change cid format from string to bytes for the marketplace

* refactor: marketplace custom errors handling

* chore: update contracts repo

* chore: update contracts submodule

* Update contracts submodule

* Initialize the Cid using init function

* Restorage serialize pragma

* Use Cid object instead of buffer

* Simplify cid usage

* Simplify cid usage

* bump codex-contracts-eth after PR merge, formatting

* fix rebase

* collateralPerByte => collateralPerSlot

---------

Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2025-02-18 05:47:47 +00:00
Arnaud
6e73338425
Remove deprecated function (#1111)
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2025-02-17 22:04:04 +00:00
Eric
5af3477793
chore(deps): bump ethers to propagate cancellations (#1116)
* chore(deps): bump ethers to propagate cancellations

Ethers was swallowing canellations and turning them into EthersErrors, which was causing the sales statemachine to error when it should have been simply cancelling the current state's run. Hopefully fixes the intermittently failing marketplace integration test.

* Add missing errors in async raises pragma

* bump to version of ethers that supports cancellations

---------

Co-authored-by: Arnaud <arnaud@status.im>
2025-02-17 22:00:52 +00:00
Arnaud
dc08ff8840
chore(marketplace): add a cache for storage requests (#1090)
* Add cache to for requests

* Change request cache description message and use const as default value

* Set request cache size configuration hidden
2025-02-17 10:34:42 +00:00
Giuliano Mega
25c84f4e0e
Fix/repostore deletes for non-orphan blocks (#1109)
* fix: fix deletion of non-orphan blocks

* feat: improve error feedback for illegal direct block deletes

* chore: minor rewording of test header
2025-02-14 13:34:17 +00:00
munna0908
c65148822e
feat: multithreading support for erasure coding (#1087)
* implement async encode

* implement async decode

* cleanup code

* add num-threads flag

* fix tests

* code cleanup

* improve return types and exception handling for async proc

* add validation check for numThreads flag

* modify encode method

* add new tests for aync encoding

* modify decode method

* cleanup test cases

* add new cli flag for threadCount

* test cleanup

* add new tests

* fix decodeAsync exception handling

* code cleanup

* chore: cosmetic changes
2025-02-12 17:56:26 +00:00
Dmitriy Ryajov
45e97513a7
remove uploadedAt from manifest (#1091)
* remove uploadedAt from manifest

* fix test
2025-02-12 10:48:58 +00:00
Giuliano Mega
bbe1f09cd7
Purging of local datasets (#1103)
* feat(codex-node): add dataset deletion API to Codex node

* feat(api): add deletion of local datasets to API

* fix: logging, remove garbage, drop some CORS headers from DELETE request

* fix: change empty response return code to 204 instead of 200

* fix: add time-based idling to avoid locking up the node during deletes, fix API status code

* fix: uncomment commented tests committed by accident

* fix: return correct code when missing CID is a Manifest CID; add back CORS headers

* fix: remove lingering echo
2025-02-11 19:00:05 +00:00
Marcin Czenko
8880ad9cd4
fix linting in "codex/blockexchange/engine/engine.nim" (#1107) 2025-02-11 10:47:25 +00:00
Dmitriy Ryajov
17d3f99f45
use a case-of instead of if for better readability (#1063) 2025-02-06 21:36:35 +00:00
Arnaud
ffd149fba8
Update documentations and cleanup 2025-02-06 15:17:06 +01:00
Arnaud
9764d3f133
Clean up and fix tests 2025-02-06 14:54:04 +01:00
Arnaud
ac24f35250
Remove debug message 2025-02-06 08:30:29 +01:00
Arnaud
904d20c48f
Add until parameter 2025-02-05 20:52:38 +01:00
Arnaud
7257f1768f
Return bytes to availability when finished 2025-02-04 19:29:07 +01:00
Giuliano Mega
0595723f66
Minor improvements to download API (#1092)
* chore: improve error messages in upload API

* chore: remove unreachable (dead) code

* fix: API integration test
2025-02-04 16:01:14 +00:00
Arnaud
b80b1f85be
Add availability enabled parameter 2025-01-29 17:20:02 +01:00
Arnaud
58a962add8
Update contracts and add custom errors (#1088) 2025-01-28 11:16:24 +00:00
Marcin Czenko
962fc1cd95
Feat: price per byte (#1078)
* openAPI: StorageRequestCreation: reward => pricePerByte, collateral => collateralPerByte

* purchasing: reward => pricePerByte, collateral => collateralPerByte

* Updates availabilities and reservations to use totalCollateral, minPricePerByte, and maxCollateralPerByte

* Uses correct div operator when operating on UInt256

* proposal updating totalCollateral in availability

* makes sure that reading currentCollateral happens before freeing slot

* Updates naming

* fixes tests: unit and contracts

* uses feat/price-per-byte branch for codex-contracts-eth

* temporarily disables integration tests on CI

* introduces high level <<totalCollateral>> property for a cleaner external interface

* updates integration tests

* Applies review comments

* Updates description of totalCollateral in SalesAvailability

* updates codex-contracts-eth (price-per-byte)
2025-01-24 17:18:00 +00:00
markspanbroek
f6c792de79
fix slotqueue worker starvation (#1081)
* fix slotqueue worker starvation

* improve slotqueue tests

Co-Authored-By: Marcin Czenko <marcin.czenko@pm.me>

* slotqueue nph formatting

---------

Co-authored-by: Marcin Czenko <marcin.czenko@pm.me>
2025-01-23 09:28:14 +00:00
Adam Uhlíř
e5df8c50d3
style: nph formatting (#1067)
* style: nph setup

* chore: formates codex/ and tests/ folder with nph 0.6.1
2025-01-21 20:54:46 +00:00
Arnaud
389ab59aa7
Use custom error message (#1079) 2025-01-20 16:04:50 +00:00
Giuliano Mega
ac12de37b2
chore: delete leftover code from old threaded erasure coding backend (#1080) 2025-01-20 14:50:33 +00:00
Arnaud
2ad7c31c85
Remove duplicate header (#971)
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
2025-01-16 13:51:07 +00:00
Arnaud
4606726e27
Add default Content-Disposition header for download (#981) 2025-01-16 13:25:26 +00:00
Adam Uhlíř
68ad804f9e
refactor: marketplace custom errors handling (#1061)
* refactor: marketplace custom errors handling

* chore: update contracts repo

* chore: update contracts submodule
2025-01-16 09:34:44 +00:00
Arnaud
f25c555d59
Chore/update nim version (#1052)
* Move to version 2.0.6

* Update nim-confutils submodule to latest version

* Update dependencies

* Update Nim version to 2.0.12

* Add gcsafe pragma

* Add missing import

* Update specific conf for Nim 2.x

* Fix method signatures

* Revert erasure coding attempt to fix bug

* More gcsafe pragma

* Duplicate code from libp2p because it is not exported anymore

* Fix camelcase function names

* Use alreadySeen because need is not a bool anymore

* newLPStreamReadError does not exist anymore so use another error

* Replace ValidIpAddress by IpAddress

* Add gcsafe pragma

* Restore maintenance parameter deleted by mistake when removing esasure coding fix attempt code

* Update method signatures

* Copy LPStreamReadError code from libp2p which was removed

* Fix camel case

* Fix enums in tests

* Fix camel case

* Extract node components to a variable to make Nim 2 happy

* Update the tests using ValidIpAddress to IpAddress

* Fix cast for value which is already an option

* Set nim version to 2.0.x for CI

* Set nim version to 2.0.x for CI

* Move to miniupnp version 2.2.4 to avoid symlink error

* Set core.symlinks to false for Windows for miniupnp >= 2.2.5 support

* Update to Nim 2.0.14

* Update CI nim versions to 2.0.14

* Try with GCC 14

* Replace apt-fast by apt-get

* Update ubuntu runner to latest

* Use Ubuntu 20.04 for coverage

* Disable CI cache for coverage

* Add coverage property description

* Remove commented test

* Check the node value of seen instead of using alreadySeen

* Fix the merge. The taskpool work was reverted.

* Update nim-ethers submodule

* Remove deprecated ValidIpAddress. Fix missing case and imports.

* Fix a weird issue where nim-confutils cannot find NatAny

* Fix tests and remove useless static keyword
2025-01-10 14:12:37 +00:00
Ben Bierens
caed3c07a3
Fix sending of WantBlocks messages and tracking of peerWants (#1019)
* sends wantBlock to peers with block. wantHave to everyone else

* Cleanup cheapestPeer. Fixes test for peers lists

* Fixes issue where peerWants are only stored for type wantBlock.

* Review comments by Dmitriy

* consistent logging of addresses

* prevents duplicate scheduling. Fixes cancellation

* fast

* Marks cancel-presence situation with todo comment.

* fixtest: testsales enable logging

* Review by Dmitriy: Remember peerWants only if we don't have them.

* rework `wantListHandler` handling

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2025-01-09 22:44:02 +00:00
munna0908
74c46b3651
network: support for uPnP and PMP nat traversal (#1050)
* add support for uPnP and PMP nat traversal

* update cli flags

* merge with master

* use listener address from switch

* update build script to rename version file

* fix styling issues

* update docker files

- remove Disc_ip env
- update CODE_NAT parsing logic

* code cleanup

* move nat flag parsing logic to conf.nim
2025-01-09 18:11:22 +00:00
Adam Uhlíř
407f77871f
chore: warning cleanup (#1055)
* chore: warning cleanup

* chore: fix proper disabling of warning

* chore: ignore the import when not needed
2025-01-08 11:30:54 +00:00
Arnaud
145aa5d84a
chore: remove old fixme (#1060)
* Remove FIXME

* Fix indentation
2025-01-08 07:45:15 +00:00
Adam Uhlíř
4b99b58645
fix: unhide log-format cli flag (#1049) 2024-12-19 12:03:50 +00:00