* checked exceptions in stores
* makes asynciter as much exception safe as it gets
* introduce "SafeAsyncIter" that uses Results and limits exceptions to cancellations
* adds {.push raises: [].} to errors
* uses SafeAsyncIter in "listBlocks" and in "getBlockExpirations"
* simplifies safeasynciter (magic of auto)
* gets rid of ugly casts
* tiny fix in hte way we create raising futures in tests of safeasynciter
* Removes two more casts caused by using checked exceptions
* adds an extended explanation of one more complex SafeAsyncIter test
* adds missing "finishOnErr" param in slice constructor of SafeAsyncIter
* better fix for "Error: Exception can raise an unlisted exception: Exception" error.
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* makes sure we do not call "get" on unverified result
* make handling of failed blocks in fetchBatched even more explicit
* simplifies allFinishedValues and makes it independent from allFinishedFailed
* only sleep if not iter.finished in fetchBatched
* cleanup imports and logs
* add BlockHandle type
* revert deps
* refactor: async error handling and future tracking improvements
- Update async procedures to use explicit raises annotation
- Modify TrackedFutures to handle futures with no raised exceptions
- Replace `asyncSpawn` with explicit future tracking
- Update test suites to use `unittest2`
- Standardize error handling across network and async components
- Remove deprecated error handling patterns
This commit introduces a more robust approach to async error handling and future management, improving type safety and reducing potential runtime errors.
* bump nim-serde
* remove asyncSpawn
* rework background downloads and prefetch
* imporove logging
* refactor: enhance async procedures with error handling and raise annotations
* misc cleanup
* misc
* refactor: implement allFinishedFailed to aggregate future results with success and failure tracking
* refactor: update error handling in reader procedures to raise ChunkerError and CancelledError
* refactor: improve error handling in wantListHandler and accountHandler procedures
* refactor: simplify LPStreamReadError creation by consolidating parameters
* refactor: enhance error handling in AsyncStreamWrapper to catch unexpected errors
* refactor: enhance error handling in advertiser and discovery loops to improve resilience
* misc
* refactor: improve code structure and readability
* remove cancellation from addSlotToQueue
* refactor: add assertion for unexpected errors in local store checks
* refactor: prevent tracking of finished futures and improve test assertions
* refactor: improve error handling in local store checks
* remove usage of msgDetail
* feat: add initial implementation of discovery engine and related components
* refactor: improve task scheduling logic by removing unnecessary break statement
* break after scheduling a task
* make taskHandler cancelable
* refactor: update async handlers to raise CancelledError
* refactor(advertiser): streamline error handling and improve task flow in advertise loops
* fix: correct spelling of "divisible" in error messages and comments
* refactor(discovery): simplify discovery task loop and improve error handling
* refactor(engine): filter peers before processing in cancelBlocks procedure
* Indicate that slot is being repaired when trying to download
* Fix tests
* Apply nph
* Calculate the repair collateral when adding the item into the queue
* Add slotCollateral calculation with getRequest cache and remove populationItem function
* Update with pricePerByte
* Simplify StorageAsk parameter
* Minor fixes
* Move cache request to another PR
* Rename SlotQueueItem collateral and required in init
* Use override func to optimise calls when the slot state is known
* Remove unused code
* Cosmetic change
* Use raiseMarketError helper
* Add exceptions to async pragma
* Cosmetic change
* Use raiseMarketError helper
* Let slotCollateral determines the slot sate
* Use configSync to avoid async pragma in onStorageRequested
* Add loadConfig function
* Add CatchableError to async pragma
* Add missing pragma raises errors
* Move loadConfig
* Avoid swallow CancelledError
* Avoid swallowing CancelledError
* Avoid swallowing CancelledError
* Update error messages
* Except MarketError instead of CatchableError
* Fix merge issue
* Log fatal when configuration cannot be loaded
* Propagate MarketError in slotCollateral
* Remove useless configSync
* Use result with explicit error
* Fix syntax
---------
Signed-off-by: Arnaud <arnaud@status.im>
* adding basic retry functionality
* avoid duplicate requests and batch them
* fix cancelling blocks
* properly resolve blocks
* minor cleanup - use `self`
* avoid useless asyncSpawn
* track retries
* limit max inflight and set libp2p maxIncomingStreams
* cleanup
* add basic yield in readLoop
* use tuple instead of object
* cleanup imports and logs
* increase defaults
* wip
* fix prefetch batching
* cleanup
* decrease timeouts to speedup tests
* remove outdated test
* add retry tests
* should track retries
* remove useless test
* use correct block address (index was off by 1)
* remove duplicate noop proc
* add BlockHandle type
* Use BlockHandle type
* add fetchLocal to control batching from local store
* add format target
* revert deps
* adjust quotaMaxBytes
* cleanup imports and logs
* revert deps
* cleanup blocks on cancelled
* terminate erasure and prefetch jobs on stream end
* split storing and retrieving data into separate tests
* track `b.discoveryLoop` future
* misc
* remove useless check
* feat(codex-node): add dataset deletion API to Codex node
* feat(api): add deletion of local datasets to API
* fix: logging, remove garbage, drop some CORS headers from DELETE request
* fix: change empty response return code to 204 instead of 200
* fix: add time-based idling to avoid locking up the node during deletes, fix API status code
* fix: uncomment commented tests committed by accident
* fix: return correct code when missing CID is a Manifest CID; add back CORS headers
* fix: remove lingering echo
* openAPI: StorageRequestCreation: reward => pricePerByte, collateral => collateralPerByte
* purchasing: reward => pricePerByte, collateral => collateralPerByte
* Updates availabilities and reservations to use totalCollateral, minPricePerByte, and maxCollateralPerByte
* Uses correct div operator when operating on UInt256
* proposal updating totalCollateral in availability
* makes sure that reading currentCollateral happens before freeing slot
* Updates naming
* fixes tests: unit and contracts
* uses feat/price-per-byte branch for codex-contracts-eth
* temporarily disables integration tests on CI
* introduces high level <<totalCollateral>> property for a cleaner external interface
* updates integration tests
* Applies review comments
* Updates description of totalCollateral in SalesAvailability
* updates codex-contracts-eth (price-per-byte)
* Move to version 2.0.6
* Update nim-confutils submodule to latest version
* Update dependencies
* Update Nim version to 2.0.12
* Add gcsafe pragma
* Add missing import
* Update specific conf for Nim 2.x
* Fix method signatures
* Revert erasure coding attempt to fix bug
* More gcsafe pragma
* Duplicate code from libp2p because it is not exported anymore
* Fix camelcase function names
* Use alreadySeen because need is not a bool anymore
* newLPStreamReadError does not exist anymore so use another error
* Replace ValidIpAddress by IpAddress
* Add gcsafe pragma
* Restore maintenance parameter deleted by mistake when removing esasure coding fix attempt code
* Update method signatures
* Copy LPStreamReadError code from libp2p which was removed
* Fix camel case
* Fix enums in tests
* Fix camel case
* Extract node components to a variable to make Nim 2 happy
* Update the tests using ValidIpAddress to IpAddress
* Fix cast for value which is already an option
* Set nim version to 2.0.x for CI
* Set nim version to 2.0.x for CI
* Move to miniupnp version 2.2.4 to avoid symlink error
* Set core.symlinks to false for Windows for miniupnp >= 2.2.5 support
* Update to Nim 2.0.14
* Update CI nim versions to 2.0.14
* Try with GCC 14
* Replace apt-fast by apt-get
* Update ubuntu runner to latest
* Use Ubuntu 20.04 for coverage
* Disable CI cache for coverage
* Add coverage property description
* Remove commented test
* Check the node value of seen instead of using alreadySeen
* Fix the merge. The taskpool work was reverted.
* Update nim-ethers submodule
* Remove deprecated ValidIpAddress. Fix missing case and imports.
* Fix a weird issue where nim-confutils cannot find NatAny
* Fix tests and remove useless static keyword
* chore: bump dependencies, including nim-ethers with chronos v4 support
Bumps the following dependencies:
- nim-ethers to commit 507ac6a4cc71cec9be7693fa393db4a49b52baf9 which contains a pinned nim-eth version. This is to be replaced by a versioned library, so it will be pinned to a particular version. There is a crucial fix in this version of ethers that fixes nonce management which is causing issues in the Codex testnet.
- nim-json-rpc to v0.4.4
- nim-json-serialization to v0.2.8
- nim-serde to v1.2.2
- nim-serialization to v0.2.4
Currently, one of the integration tests is failing.
* fix integration test
- When a state's run was cancelled, it was being caught as an error due to catching all CatchableErrors. This caused a state transition to SaleErrored, however cancellation of run was not actually an error. Handling this correctly fixed the issue.
- Stopping of the clock was moved to after `HostInteractions` (sales) which avoided an assertion around getting time when the clock was not started.
* bump ethers to include nonce fix and filter not found fix
* bump ethers: fixes missing symbol not exported in ethers
* Fix cirdl test imports/exports
* Debugging in ci
* Handle CancelledErrors for state.run in one place only
* Rename `config` to `configuration`
There was a symbol clash preventing compilation and it was easiest to rename `config` to `configuration` in the contracts. Not even remotely ideal, but it was the only way.
* bump ethers to latest
Prevents an issue were `JsonNode.items` symbol could not be found
* More changes to support `config` > `configuration`
* cleanup
* testing to see if this fixes failure in ci
* bumps contracts
- ensures slot is free before allowing reservation
- renames config to configuration to avoid symbol clash
* Add metadata to the manifest
* Remove useless import
* Fix the openapi documentation
* Use optional fields instead of default values
* Remove testRestApi target
* Return failure when the protobuf cannot get the field
* Set download headers and fix cors headers when an error is returned
* Add tests to verify the download headers
* Try to adjust the content length header
* Fix convertion to string
* Remove the content length header
* Remove testRestApi target
* Removing debug messages
* add prover bandaid
* Improve error message text
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
---------
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* fix verifiable manifest initialization
* fix linearstrategy, use verifiableStrategy to select blocks for slots
* check for both strategies in attribute inheritance test
* Rework AsyncIter
* Add tests for finishing iter on error
* Improved error handling for and additional tests
* Use new style of constructors
* Handle future cancellation
* Docs for constructors
* integration: move REST API tests into their own module
* integration: move upload and download tests into their own module
* integration: move purchasing tests into their own module
* integration: move marketplace tests to the right module
* integration: mine a block *after* starting nodes
To ensure that tests involving multiple nodes do
not start with out-of-sync clocks
* Fix: do not swallow CancellationErrors
* integration: avoid underflow in UInt256
* network: remove unnecessary error handling
No Exceptions can occur, only Defects, because everything
is asyncSpawned.
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* network: do not raise in asyncSpawned proc
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* wire prover into node
* stricter case object checks
* return correct proof
* misc renames
* adding usefull traces
* fix nodes and tolerance to match expected params
* format challenges in logs
* add circom compat to solidity groth16 convertion
* update
* bump time to give nodes time to load with all circom artifacts
* misc
* misc
* use correct dataset geometry in erasure
* make errors more searchable
* use parens around `=? (await...)` calls
* styling
* styling
* use push raises
* fix to match constructor arguments
* merge master
* merge master
* integration: fix proof parameters for a test
Increased times due to ZK proof generation.
Increased storage requirement because we're now hosting
5 slots instead of 1.
* sales: calculate initial proof at start of period
reason: this ensures that the period (and therefore
the challenge) doesn't change while we're calculating
the proof
* integration: fix proof parameters for tests
Increased times due to waiting on next period.
Fixed data to be of right size.
Updated expected payout due to hosting 5 slots.
* sales: wait for stable proof challenge
When the block pointer is nearing the
wrap-around point, we wait another period
before calculating a proof.
* fix merge conflict
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* don't pass erasure
* use correct stores and construct erasure inside the node
* fix tests to match new constructor
* remove prover argument
* review commets
* revert failing on no-prover for now
* small cleanup
* comment out invalid proofs broken test
* refactor multi node test suite
Refactor the multinode test suite into the marketplace test suite.
- Arbitrary number of nodes can be started with each test: clients, providers, validators
- Hardhat can also be started locally with each test, usually for the purpose of saving and inspecting its log file.
- Log files for all nodes can be persisted on disk, with configuration at the test-level
- Log files, if persisted (as specified in the test), will be persisted to a CI artifact
- Node config is specified at the test-level instead of the suite-level
- Node/Hardhat process starting/stopping is now async, and runs much faster
- Per-node config includes:
- simulating proof failures
- logging to file
- log level
- log topics
- storage quota
- debug (print logs to stdout)
- Tests find next available ports when starting nodes, as closing ports on Windows can lag
- Hardhat is no longer required to be running prior to starting the integration tests (as long as Hardhat is configured to run in the tests).
- If Hardhat is already running, a snapshot will be taken and reverted before and after each test, respectively.
- If Hardhat is not already running and configured to run at the test-level, a Hardhat process will be spawned and torn down before and after each test, respectively.
* additional logging for debug purposes
* address PR feedback
- fix spelling
- revert change from catching ProviderError to SignerError -- this should be handled more consistently in the Market abstraction, and will be handled in another PR.
- remove method label from raiseAssert
- remove unused import
* Use API instead of command exec to test for free port
Use chronos `createStreamServer` API to test for free port by binding localhost address and port. Use `ServerFlags.ReuseAddr` to enable reuse of same IP/Port on multiple test runs.
* clean up
* remove upraises annotations from tests
* Update tests to work with updated erasure coding slot sizes
* update dataset size, nodes, tolerance to match valid ec params
Integration tests now have valid dataset sizes (blocks), tolerances, and number of nodes, to work with valid ec params. These values are validated when requested storage.
Print the rest api failure message (via doAssert) when a rest api call fails (eg the rest api may validate some ec params).
All integration tests pass when the async `clock.now` changes are reverted.
* dont use async clock for now
* fix workflow
* move integration logs uplod to reusable
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* Smart contracts update: Groth16Proof instead of bytes
* Use dummy verifier for now, until we can create ZK proofs
* Fix tests: submit proof only when slot is filled
* Submit dummy proofs for now
* More detailed log when proof submission failed
* Use dummy verifier for integration tests
For now at least
* Fix mistake in blanket renaming to ethProvider
* Update to latest codex-contracts-eth
* feat: zkey-hash from chain
* Fix zkeyHash
---------
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
* implement a logging proxy
The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks
* Rename logging -> logutils to avoid ambiguity with common names
* clean up
* add setProperty for JsonRecord, remove nim-json-serialization conflict
* Allow specifying textlines and json format separately
Not specifying a LogFormat will apply the formatting to both textlines and json sinks.
Specifying a LogFormat will apply the formatting to only that sink.
* remove unneeded usages of std/json
We only need to import utils/json instead of std/json
* move serialization from rest/json to utils/json so it can be shared
* fix NoColors ambiguity
Was causing unit tests to fail on Windows.
* Remove nre usage to fix Windows error
Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)
* Add logutils module doc
* Shorten logutils.formatIt for `NBytes`
Both json and textlines formatIt were not needed, and could be combined into one formatIt
* remove debug integration test config
debug output and logformat of json for integration test logs
* Use ## module doc to support docgen
* bump nim-poseidon2 to export fromBytes
Before the changes in this branch, fromBytes was likely being resolved by nim-stew, or other dependency. With the changes in this branch, that dependency was removed and fromBytes could no longer be resolved. By exporting fromBytes from nim-poseidon, the correct resolution is now happening.
* fixes to get compiling after rebasing master
* Add support for Result types being logged using formatIt
* Setting up testfixture for proof datasampler
* Sets up calculating number of cells in a slot
* Sets up tests for bitwise modulo
* Implements cell index collection
* setting up slot blocks module
* Implements getting treeCID from slot
* implements getting slot blocks by index
* Implements out-of-range check for slot index
* cleanup
* Sets up getting sample from block
* Implements selecting a cell sample from a block
* Implements building a minitree for block cells
* Adds method to get dataset block index from slot block index
* It's running
* splits up indexing
* almost there
* Fixes test. Implementation is now functional
* Refactoring to object-oriented
* Cleanup
* Lining up output type with updated reference code.
* setting up
* Updates expected samples
* Updates proof checking test to match new format
* move builder to own dir
* move sampler to own dir
* fix paths
* various changes to add support for the sampler
* wip sampler implementation
* don't use upraises
* wip sampler integration
* misc
* move tests around
* Various fixes to select correct slot and block index
* removing old tests
* cleanup
* misc
fix tests that work with correct cell indices
* remove unused file
* fixup logging
* add logscope
* truncate entropy to 31 bytes, otherwise it might be > than mod
* forwar getCidAndProof to local store
* misc
* Adds missing test for initial-proving state
* reverting back to correct slot/block indexing
* fix tests for revert
* misc
* misc
---------
Co-authored-by: benbierens <thatbenbierens@gmail.com>
* rework merkle tree support
* rename merkletree -> codexmerkletree
* treed and proof encoding/decoding
* style
* adding codex merkle and coders tests
* use default hash codec
* proof size changed
* add from nodes test
* shorte file names
* wip poseidon tree
* shorten file names
* root returns a result
* import poseidon tests
* fix merge issues and cleanup a few warnings
* setting up slot builder
* Getting cids in slot
* ensures blocks are devisable by number of slots
* wip
* Implements indexing strategies
* Swaps in indexing strategy into erasure.
* wires slot and indexing tests up
* Fixes issue where indexing strategy stepped gives wrong values for smallest of ranges
* debugs indexing strategies
* Can select slot blocks
* finding number of pad cells
* Implements building slot tree
* finishes implementing slot builder
* Adds check that block size is a multiple of cell size
* Cleanup slotbuilder
* Review comments by Tomasz
* Fixes issue where ecK was used as numberOfSlots.
* rework merkle tree support
* deps
* rename merkletree -> codexmerkletree
* treed and proof encoding/decoding
* style
* adding codex merkle and coders tests
* remove new codecs for now
* proof size changed
* add from nodes test
* shorte file names
* wip poseidon tree
* shorten file names
* fix bad `elements` iter
* bump
* bump
* wip
* reworking slotbuilder
* move out of manifest
* expose getCidAndProof
* import index strat...
* remove getMHash
* remove unused artifacts
* alias zero
* add digest for multihash
* merge issues
* remove unused hashes
* add option to result converter
* misc
* fix tests
* add helper to derive EC block count
* rename method
* misc
* bump
* extract slot root building into own proc
* revert to manifest to accessor
---------
Co-authored-by: benbierens <thatbenbierens@gmail.com>
* implement a logging proxy
The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks
* Rename logging -> logutils to avoid ambiguity with common names
* clean up
* add setProperty for JsonRecord, remove nim-json-serialization conflict
* Allow specifying textlines and json format separately
Not specifying a LogFormat will apply the formatting to both textlines and json sinks.
Specifying a LogFormat will apply the formatting to only that sink.
* remove unneeded usages of std/json
We only need to import utils/json instead of std/json
* move serialization from rest/json to utils/json so it can be shared
* fix NoColors ambiguity
Was causing unit tests to fail on Windows.
* Remove nre usage to fix Windows error
Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)
* Add logutils module doc
* Shorten logutils.formatIt for `NBytes`
Both json and textlines formatIt were not needed, and could be combined into one formatIt
* remove debug integration test config
debug output and logformat of json for integration test logs
* Use ## module doc to support docgen
