- cleans up all instances of `.track` to use the `module.trackedfutures.track(future)` procedure, for better readability
- removes the `track` override that is no longer used in the codebase
* adds a new overload of queryPastEvents allowing to query past events based on timestamp in the past
* adds state restoration to validator
* refactors a bit to get the tests back to work
* replaces deprecated generic methods from Market with methods for specific event types
* Refactors binary search
* adds market tests for querying past SlotFilled events and binary search
* Takes into account that <<earliest>> block available is not necessarily the genesis block
* Adds more logging and makes testing earliest block boundary more reliable
* adds validation tests for historical state restoration
* adds mockprovider to simplify and improve testing of the edge conditions
* adds slot reservation to the new tests after rebasing
* adds validation groups and group index in logs of validator
* adds integration test with two validators
* adds comment on how to enable logging in integration test executable itself
* testIntegration: makes list is running nodes injected and available in the body of the test
* validation: adds integration test for historical state
* adds more logging to validator
* integration test: validator only looks 30 days back for historical state
* adds logging of the slotState when removing slots during validation
* review and refactor validator integration tests
* adds validation to the set of integration tests
* Fixes mistyped name of the mock provider module in testMarket
* Fixes a typo in the name of the validation suite in integration tests
* Makes validation unit test a bit easier to follow
* better use of logScopes to reduce duplication
* improves timing and clarifies the test conditions
* uses http as default RPC provider for nodes running in integration tests as a workaround for dropped subscriptions
* simplifies the validation integration tests by waiting for failed request instead of tracking slots
* adds config option allowing selectively to set different provider url
* Brings back the default settings for RPC provider in integration tests
* use http RPC provider for clients in validation integration tests
* fine-tune the tests
* Makes validator integration test more robust - adds extra tracking
* brings tracking of marketplace event back to validator integration test
* refactors integration tests
* deletes tmp file
* adds <<return>> after forcing integration test to fail preliminarily
* re-enables all integration tests and matrix
* stops debug output in CI
* allows to choose a different RPC provider for a given integration test suite
* fixes signature of <<getBlock>> method in mockProvider
* adds missing import which seem to be braking integration tests on windows
* makes sure that clients, SPs, and validators use the same provider url
* makes validator integration tests using http at 127.0.0.1:8545
* testvalidator: stop resubscribing as we are now using http polling as rpc provider
* applying review comments
* groups queryPastStorage overrides together (review comment)
* groups the historical validation tests into a sub suite
* removes the temporary extensions in marketplacesuite and multinodesuite allowing to specify provider url
* simplifies validation integration tests
* Removes debug logs when waiting for request to fail
* Renaming waitForRequestFailed => waitForRequestToFail
* renames blockNumberForBlocksAgo to pastBlockTag and makes it private
* removes redundant debugging logs
* refines logging in validation
* removes dev logging from mockmarket
* improves exception handling in provider helper procs and prepares for extraction to a separate module
* Uses chronos instead of std/times for Duration
* extracts provider and binary search helpers to a separate module
* removes redundant log entry params from validator
* unifies the notation to consistently use method call syntax
* reuses ProviderError from nim-ethers in the provider extension
* clarifies the comment in multinodesuite
* uses == operator to check the predefined tags and raises exception when `BlockTag.pending` is requested.
* when waiting for request to fail, we break on any request state that is not Started
* removes tests that were moved to testProvider from testMarket
* extracts tests that use MockProvider to a separate async suite
* improves performance of the historical state restoration
* removing redundant log messages in validator (groupIndex and groups)
* adds testProvider to testContracts group
* removes unused import in testMarket
* feat: repair is rewarded
* chore: update contracts repo
* feat: proving loop handles repair case
* test: assert repair state
* chore: update contracts repo
* fix: upon unknown state of repair go to error
* Use http subscriptions instead of websocket for tests
To work around this issue when subscriptions are
inactive for more than 5 minutes:
https://github.com/NomicFoundation/hardhat/issues/2053
Use 100 millisecond polling; default polling interval
of 4 seconds is too close to the 5 second timeout for
`check eventually`.
* use .confirm(1) instead of confirm(0)
confirm(0) doesn't wait at all, confirm(1) waits
for the transaction to be mined
* speed up partial payout integration test
* update nim-ethers to version 0.10.0
includes fixes for http polling and .confirm()
* fix timing of marketplace tests
allow for a bit more time to withdraw funds
* use .confirm(1) in marketplace tests
to ensure that the transaction has been processed
before continuing with the test
* fix timing issue in validation unit test
* fix proof integration test
there were two logic errors in this test:
- a slot is freed anyway at the end of the contract
- when starting the request takes a long time, the
first slot can already be freed because there were
too many missing proofs
* fix intermittent error in contract tests
currentTime() doesn't always correctly reflect
the time of the next transaction
* reduce number of slots in integration test
otherwise the windows runner in the CI won't
be able to start the request before it expires
* fix timing in purchasing test
allow for a bit more time for a request to
be submitted
* fix timing of request submission in test
windows ci is so slow, it can take up to 40 seconds
just to submit a storage request to hardhat
* increase proof period to 90 seconds
* adjust timing of integration tests
reason: with the increased period length of 90 seconds, it
can take longer to wait for a stable challenge at the
beginning of a period.
* increase CI timeout to 2 hours
* Fix slow builds on windows
apparently it takes windows 2-3 seconds to
resolve "localhost" to 127.0.0.1 for every
json-rpc connection that we make 🤦
* Add metadata to the manifest
* Remove useless import
* Fix the openapi documentation
* Use optional fields instead of default values
* Remove testRestApi target
* Return failure when the protobuf cannot get the field
* Set download headers and fix cors headers when an error is returned
* Add tests to verify the download headers
* Try to adjust the content length header
* Fix convertion to string
* Remove the content length header
* Remove testRestApi target
* Removing debug messages
* Avoid cancelling states when slot is filled
* improve logging
Improves logging for situations where a Sale should be ignored instead of being considered an error, including when reservation is not allowed and when a slot was filled by another host.
* remove onSlotFilled unit tests from states
* convert EthersError to MarketError
* change `canReserveSlot` and `reserveSlot` parameters
Parameters for `canReserveSlot` and `reserveSlot` were changed from `SlotId` to `RequestId` and `UInt256 slotIndex`.
* Add SaleSlotReserving
Adds a new state, SaleSlotReserving, that attempts to reserve a slot before downloading.
If the slot cannot be reserved, the state moves to SaleIgnored.
On error, the state moves to SaleErrored.
SaleIgnored is also updated to pass in `reprocessSlot` and `returnBytes`, controlling the behaviour in the Sales module after the slot is ignored. This is because previously it was assumed that SaleIgnored was only reached when there was no Availability. This is no longer the case, since SaleIgnored can now be reached when a slot cannot be reserved.
* Update SalePreparing
Specify `reprocessSlot` and `returnBytes` when moving to `SaleIgnored` from `SalePreparing`.
Update tests to include test for a raised CatchableError.
* Fix unit test
* Modify `canReserveSlot` and `reverseSlot` params after rebase
* Update MockMarket with new `canReserveSlot` and `reserveSlot` params
* fix after rebase
also bump codex-contracts-eth to master
* Adds validatorPartitionSize and validatorPartitionIndex config options
* adds partitioning options to the validation type
* adds partitioning logic to the validator
* ignores partitionIndex when partitionSize is either 0 or 1
* clips the partition index to <<partitionIndex mod partitionSize>>
* handles negative values for the validation partition index
* updates long description of the new validator cli options
* makes default partitionSize to be 0 for better backward compatibility
* Improving formatting on validator CLI
* reactors validation params into a separate type and simplifies validation of validation params
* removes suspected duplication
* fixes typo in validator CLI help
* updates README
* Applies review comments - using optionals and range types to handle validation params
* Adds initializer to the configFactory for validatorMaxSlots
* [Review] update validator CLI description and README
* [Review]: renaming validationParams to validationConfig (config)
* [Review]: move validationconfig.nim to a higher level (next to validation.nim)
* changes backing type of MaxSlots to be int and makes sure slots are validated without limit when maxSlots is set to 0
* adds more end-to-end test for the validator and the groups
* fixes typo in README and conf.nim
* makes `maxSlotsConstraintRespected` and `shouldValidateSlot` private + updates the tests
* fixes public address of the signer account in the marketplace tutorial
* applies review comments - removes two tests
* Introduces a start method to prover
* Moves backend creation into start method
* sets up three paths for backend initialization
* Extracts backend initialization to backend-factory
* Implements loading backend from cli files or previously downloaded local files
* Wires up downloading and unzipping
* functional implementation
* Fixes testprover.nim
* Sets up tests for backendfactory
* includes libzip-dev
* pulls in updated contracts
* removes integration cli tests for r1cs, wasm, and zkey file arguments.
* Fixes issue where inner-scope values are lost before returning
* sets local proof verification for dist-test images
* Adds two traces and bumps nim-ethers
* Adds separate path for circuit files
* Create circuit dir if not exists
* fix: make sure requestStorage is mined
* fix: correct place to plug confirm
* test: fixing contracts tests
* Restores gitmodules
* restores nim-datastore reference
* Sets up downloader exe
* sets up tool skeleton
* implements getting of circuit hash
* Implements downloader tool
* sets up test skeleton
* Implements test for cirdl
* includes testTools in testAll
* Cleanup building.md
* cleans up previous downloader implementation
* cleans up testbackendfactory
* moves start of prover into node.nim
* Fills in arguments in example command
* Initializes backend in prover constructor
* Restores tests
* Restores tests for cli instructions
* Review comments by Dmitriy, part 1
* Quotes path in download instruction.
* replaces curl with chronos http session
* Moves cirdl build output to 'build' folder.
* Fixes chronicles log output
* Add cirdl support to the codex Dockerfile
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Add cirdl support to the docker entrypoint
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Add cirdl support to the release workflow
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Disable verify_circuit flag for releases
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Removes backendFactory placeholder type
* wip
* Replaces zip library with status-im/zippy library (which supports zip and tar)
* Updates cirdl to not change circuitdir folder
* Switches from zip to tar.gz
* Review comments by Dmitriy
* updates codex-contracts-eth
* Adds testTools to CI
* Adds check for access to config.circuitdir
* Update fixture circuit zkey
* Update matrix to run tools tests on Windows
* Adds 'deps' dependency for cirdl
* Adjust docker-entrypoint.sh to use CODEX_CIRCUIT_DIR env var
* Review comments by Giuliano
---------
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: Veaceslav Doina <20563034+veaceslavdoina@users.noreply.github.com>
* refactor(marketplace): move marketplace events to the Market abstraction
Move marketplace contract events to the Market abstraction so the types can be shared across all modules that call the Market abstraction.
* Remove unneeded conversion
* Switch to generic implementation of event querying
* change parent type to MarketplaceEvent
* add changes to use chronos v4 in compat mode
* switch chronos to compat fix branch
* use nimbus-build-system with configurable Nim repo
* add missing imports
* add missing await
* bump compat
* pin nim version in Makefile
* add await instead of asyncSpawn to advertisement queue loop
* bump DHT to v0.5.0
* allow error state of `onBatch` to propagate upwards in test code
* pin Nim compiler commit to avoid fetching stale branch
* make CI build against branch head instead of merge
* fix handling of return values in testslotqueue
* add logs and test
* add Merkle proof checks
* factor out Circom input normalization, fix proof input serialization
* add test and update existing ones
* update circuit assets
* add back trace message
* switch contracts to fix branch
* update codex-contracts-eth to latest
* do not expose prove with prenormalized inputs
* Fix verifiable manifest constructor
* Add integration test for verifiable manifest download
Add integration test for testing download of verifiable dataset after creating request for storage
* add missing import
* add testecbug to integration suite
* Remove hardhat instance from integration test
* change description, drop echo
---------
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: gmega <giuliano.mega@gmail.com>
* fix verifiable manifest initialization
* fix linearstrategy, use verifiableStrategy to select blocks for slots
* check for both strategies in attribute inheritance test
* Rework AsyncIter
* Add tests for finishing iter on error
* Improved error handling for and additional tests
* Use new style of constructors
* Handle future cancellation
* Docs for constructors
* pulls in datastore-leveldb update
* bump
* Applies LevelDb as metadata store. Adds option for repostore.
* Sets submodule to main branch
* I can do syntax, me
* Removes wildcard from metadata query key
* Applies leveldb instead of sqlite-in-memory for tests
* Restores query key wildcard.
* Pins nim-datastore to latest master
* bumps leveldb to 0.1.4
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* add seen flag
* Add MockSlotQueueItem and better prioritisation tests
* Update seen priority, and include in SlotQueueItem.init
* Re-add processed slots to queue
Re-add processed slots to queue if the sale was ignored or errored
* add pausing of queue
- when processing slots in queue, pause queue if item was marked seen
- if availability size is increased, trigger onAvailabilityAdded callback
- in sales, on availability added, clear 'seen' flags, then unpause the queue
- when items pushed to the queue, unpause the queue
* remove unused NoMatchingAvailabilityError from slotqueue
The slot queue should also have nothing to do with availabilities
* when all availabilities are empty, pause the queue
An empty availability is defined as size < DefaultBlockSize as this means even the smallest possible request could not be served. However, this is up for discussion.
* remove availability from onAvailabilitiesEmptied callback
* refactor onAvailabilityAdded and onAvailabilitiesEmptied
onAvailabilityAdded and onAvailabilitiesEmptied are now only called from reservations.update (and eventually reservations.delete once implemented).
- Add empty routine for Availability and Reservation
- Add allEmpty routine for Availability and Reservation, which returns true when all all Availability or Reservation objects in the datastore are empty.
* SlotQueue test support updates
* Sales module test support updates
* Reservations module tests for queue pausing
* Sales module tests for queue pausing
Includes tests for sales states cancelled, errored, ignored to ensure onCleanUp is called with correct parameters
* SlotQueue module tests for queue pausing
* fix existing sales test
* PR feedback
- indent `self.unpause`
- update comment for `clearSeenFlags`
* reprocessSlot in SaleErrored only when coming from downloading
* remove pausing of queue when availabilities are "emptied"
Queue pausing when all availiabilies are "emptied" is not necessary, given that the node would not be able to service slots once all its availabilities' freeSize are too small for the slots in the queue, and would then be paused anyway.
Add test that asserts the queue is paused once the freeSpace of availabilities drops too low to fill slots in the queue.
* Update clearing of seen flags
The asyncheapqueue update overload would need to check index bounds and ultimately a different solution was found using the mitems iterator.
* fix test
request.id was different before updating request.ask.slots, and that id was used to set the state in mockmarket.
* Change filled/cleanup future to nil, so no await is needed
* add wait to allow items to be added to queue
* do not unpause queue when seen items are pushed
* re-add seen item back to queue once paused
Previously, when a seen item was processed, it was first popped off the queue, then the queue was paused waiting to process that item once the queue was unpaused. Now, when a seen item is processed, it is popped off the queue, the queue is paused, then the item is re-added to the queue and the queue will wait until unpaused before it will continue popping items off the queue. If the item was not re-added to the queue, it would have been processed immediately once unpaused, however there may have been other items with higher priority pushed to the queue in the meantime. The queue would not be unpaused if those added items were already seen. In particular, this may happen when ignored items due to lack of availability are re-added to a paused queue. Those ignored items will likely have a higher priority than the item that was just seen (due to it having been processed first), causing the queue to the be paused.
* address PR comments
* json > nim-serde bump
Should wait until serde is integrated into nim-ethers before making these changes as there will be less import exceptions required.
* bump nim-serde
* change func to proc due to chronicles side effects
* import serde into utils/json, use as proxy
import nim-serde into utils/json and use utils/json as a proxy for serde functions, including overloading `%` and `fromJson` for application types.
* update tests to use serde
* bump serde to latest
* remove testjson -- no longer needed
* bump serde in nimble
* updates to reconcile rebase with master
* wire prover into node
* stricter case object checks
* return correct proof
* misc renames
* adding usefull traces
* fix nodes and tolerance to match expected params
* format challenges in logs
* add circom compat to solidity groth16 convertion
* update
* bump time to give nodes time to load with all circom artifacts
* misc
* misc
* use correct dataset geometry in erasure
* make errors more searchable
* use parens around `=? (await...)` calls
* styling
* styling
* use push raises
* fix to match constructor arguments
* merge master
* merge master
* integration: fix proof parameters for a test
Increased times due to ZK proof generation.
Increased storage requirement because we're now hosting
5 slots instead of 1.
* sales: calculate initial proof at start of period
reason: this ensures that the period (and therefore
the challenge) doesn't change while we're calculating
the proof
* integration: fix proof parameters for tests
Increased times due to waiting on next period.
Fixed data to be of right size.
Updated expected payout due to hosting 5 slots.
* sales: wait for stable proof challenge
When the block pointer is nearing the
wrap-around point, we wait another period
before calculating a proof.
* fix merge conflict
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* clock: add 1 second leeway before acting on timeouts
* sales: do not raise in proving loop when slot is cancelled
Allow the onCancelled callback to handle cancellation, and
the onFailed callback to handle failed requests.
* sales: cleanup proving tests
* sales: fix sales agent tests
* sales: stop cancellation loop when request started, finished or failed
* sales: fix flaky test
* sales: fix another flaky test
* clock: add comment explaining the + 1 second
Co-Authored-By: benbierens <thatbenbierens@gmail.com>
---------
Co-authored-by: benbierens <thatbenbierens@gmail.com>
* Applies peer-scoped lock to peer task handler.
* Replace async lock with delete-first approach.
* Cleanup some logging
* Adds inFlight flag to WantListEntry
* Clears inflight flag when local retrieval fails.
* Adds test for setting of in-flight
* Adds test for clearing in-flight when lookup fails
* Review comments by Tomasz
---------
Co-authored-by: gmega <giuliano.mega@gmail.com>
* add block cancellation support + tests
* tie issueCancellations into resolveBlocks for proper exception tracking, address comments
* pull cancellation as separate primitive in BlockExcNetwork
* use allFutures, rename issueBlockCancellations -> cancelBlocks
* use trc instead of wrn to register send error
* do not log peer IDs
* rework backend to instantiate key at initialization
* add groth16 convertes for solidity
* prover taks num samples on construction
* add zkey file
* rework helpers
* rename types
* update tests
* reworked test helpers
* rename types
* rework test
* test all slots artifacts
* bump to latest version
* don't pass erasure
* use correct stores and construct erasure inside the node
* fix tests to match new constructor
* remove prover argument
* review commets
* revert failing on no-prover for now
* small cleanup
* comment out invalid proofs broken test
Eric
Marcin Czenko
Adam Uhlíř
Ben Bierens
markspanbroek
Arnaud
Giuliano Mega
Tomasz Bekas
Dmitriy Ryajov