- add a TestManager property to IntegrationTest, so manager does not need to be passed into all functions
- cleanup:
- remove unneeded stopHardhat function
- add hardhat instance to manager outside of startHardhat
- need to test with longer tests to ensure the parallelisation is truly happening
- is the +10 hardhat port needed?
- try with more integration tests
# Conflicts:
# tests/integration/hardhatprocess.nim
# tests/integration/multinodes.nim
# tests/integration/testcli.nim
# tests/testIntegration.nim
* Move to version 2.0.6
* Update nim-confutils submodule to latest version
* Update dependencies
* Update Nim version to 2.0.12
* Add gcsafe pragma
* Add missing import
* Update specific conf for Nim 2.x
* Fix method signatures
* Revert erasure coding attempt to fix bug
* More gcsafe pragma
* Duplicate code from libp2p because it is not exported anymore
* Fix camelcase function names
* Use alreadySeen because need is not a bool anymore
* newLPStreamReadError does not exist anymore so use another error
* Replace ValidIpAddress by IpAddress
* Add gcsafe pragma
* Restore maintenance parameter deleted by mistake when removing esasure coding fix attempt code
* Update method signatures
* Copy LPStreamReadError code from libp2p which was removed
* Fix camel case
* Fix enums in tests
* Fix camel case
* Extract node components to a variable to make Nim 2 happy
* Update the tests using ValidIpAddress to IpAddress
* Fix cast for value which is already an option
* Set nim version to 2.0.x for CI
* Set nim version to 2.0.x for CI
* Move to miniupnp version 2.2.4 to avoid symlink error
* Set core.symlinks to false for Windows for miniupnp >= 2.2.5 support
* Update to Nim 2.0.14
* Update CI nim versions to 2.0.14
* Try with GCC 14
* Replace apt-fast by apt-get
* Update ubuntu runner to latest
* Use Ubuntu 20.04 for coverage
* Disable CI cache for coverage
* Add coverage property description
* Remove commented test
* Check the node value of seen instead of using alreadySeen
* Fix the merge. The taskpool work was reverted.
* Update nim-ethers submodule
* Remove deprecated ValidIpAddress. Fix missing case and imports.
* Fix a weird issue where nim-confutils cannot find NatAny
* Fix tests and remove useless static keyword
* sends wantBlock to peers with block. wantHave to everyone else
* Cleanup cheapestPeer. Fixes test for peers lists
* Fixes issue where peerWants are only stored for type wantBlock.
* Review comments by Dmitriy
* consistent logging of addresses
* prevents duplicate scheduling. Fixes cancellation
* fast
* Marks cancel-presence situation with todo comment.
* fixtest: testsales enable logging
* Review by Dmitriy: Remember peerWants only if we don't have them.
* rework `wantListHandler` handling
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
After a change in PR #1031, bootstrap node sprs may not work when Hardhat nodes are started with the tests. This fixes it by appending all started client's and provider's SPR to a sequence, and using that sequence of SPRs to start the next node. This means all subsequently started nodes will be connected to its previously started peers.
This also fixes the case when bootstrap SPRs would not be present if no clients were started.
- cleans up all instances of `.track` to use the `module.trackedfutures.track(future)` procedure, for better readability
- removes the `track` override that is no longer used in the codebase
* Sets timeout for codexclient httpClient. Adds reliable transfer test.
* disable new test to check timeout setting in CI
* restores new test
* adds heartbeat log and logfile to ci output
* fixes suite
* fixes blocked output stream by switching to multinode fixture
* new twonodessuite based on multinodesuite
* Applies updated twonodessuite
* removes heartbeat log
* applies multinodesuite in testsales
* applies multinodesuite in testmarketplace
* fixes account fetch for host and client in testmarketplace
* adds waitTillNextPeriod at end of marketplace test
* Uses marketplacesuite in testmarketplace
* adds a new overload of queryPastEvents allowing to query past events based on timestamp in the past
* adds state restoration to validator
* refactors a bit to get the tests back to work
* replaces deprecated generic methods from Market with methods for specific event types
* Refactors binary search
* adds market tests for querying past SlotFilled events and binary search
* Takes into account that <<earliest>> block available is not necessarily the genesis block
* Adds more logging and makes testing earliest block boundary more reliable
* adds validation tests for historical state restoration
* adds mockprovider to simplify and improve testing of the edge conditions
* adds slot reservation to the new tests after rebasing
* adds validation groups and group index in logs of validator
* adds integration test with two validators
* adds comment on how to enable logging in integration test executable itself
* testIntegration: makes list is running nodes injected and available in the body of the test
* validation: adds integration test for historical state
* adds more logging to validator
* integration test: validator only looks 30 days back for historical state
* adds logging of the slotState when removing slots during validation
* review and refactor validator integration tests
* adds validation to the set of integration tests
* Fixes mistyped name of the mock provider module in testMarket
* Fixes a typo in the name of the validation suite in integration tests
* Makes validation unit test a bit easier to follow
* better use of logScopes to reduce duplication
* improves timing and clarifies the test conditions
* uses http as default RPC provider for nodes running in integration tests as a workaround for dropped subscriptions
* simplifies the validation integration tests by waiting for failed request instead of tracking slots
* adds config option allowing selectively to set different provider url
* Brings back the default settings for RPC provider in integration tests
* use http RPC provider for clients in validation integration tests
* fine-tune the tests
* Makes validator integration test more robust - adds extra tracking
* brings tracking of marketplace event back to validator integration test
* refactors integration tests
* deletes tmp file
* adds <<return>> after forcing integration test to fail preliminarily
* re-enables all integration tests and matrix
* stops debug output in CI
* allows to choose a different RPC provider for a given integration test suite
* fixes signature of <<getBlock>> method in mockProvider
* adds missing import which seem to be braking integration tests on windows
* makes sure that clients, SPs, and validators use the same provider url
* makes validator integration tests using http at 127.0.0.1:8545
* testvalidator: stop resubscribing as we are now using http polling as rpc provider
* applying review comments
* groups queryPastStorage overrides together (review comment)
* groups the historical validation tests into a sub suite
* removes the temporary extensions in marketplacesuite and multinodesuite allowing to specify provider url
* simplifies validation integration tests
* Removes debug logs when waiting for request to fail
* Renaming waitForRequestFailed => waitForRequestToFail
* renames blockNumberForBlocksAgo to pastBlockTag and makes it private
* removes redundant debugging logs
* refines logging in validation
* removes dev logging from mockmarket
* improves exception handling in provider helper procs and prepares for extraction to a separate module
* Uses chronos instead of std/times for Duration
* extracts provider and binary search helpers to a separate module
* removes redundant log entry params from validator
* unifies the notation to consistently use method call syntax
* reuses ProviderError from nim-ethers in the provider extension
* clarifies the comment in multinodesuite
* uses == operator to check the predefined tags and raises exception when `BlockTag.pending` is requested.
* when waiting for request to fail, we break on any request state that is not Started
* removes tests that were moved to testProvider from testMarket
* extracts tests that use MockProvider to a separate async suite
* improves performance of the historical state restoration
* removing redundant log messages in validator (groupIndex and groups)
* adds testProvider to testContracts group
* removes unused import in testMarket
* Use http subscriptions instead of websocket for tests
To work around this issue when subscriptions are
inactive for more than 5 minutes:
https://github.com/NomicFoundation/hardhat/issues/2053
Use 100 millisecond polling; default polling interval
of 4 seconds is too close to the 5 second timeout for
`check eventually`.
* use .confirm(1) instead of confirm(0)
confirm(0) doesn't wait at all, confirm(1) waits
for the transaction to be mined
* speed up partial payout integration test
* update nim-ethers to version 0.10.0
includes fixes for http polling and .confirm()
* fix timing of marketplace tests
allow for a bit more time to withdraw funds
* use .confirm(1) in marketplace tests
to ensure that the transaction has been processed
before continuing with the test
* fix timing issue in validation unit test
* fix proof integration test
there were two logic errors in this test:
- a slot is freed anyway at the end of the contract
- when starting the request takes a long time, the
first slot can already be freed because there were
too many missing proofs
* fix intermittent error in contract tests
currentTime() doesn't always correctly reflect
the time of the next transaction
* reduce number of slots in integration test
otherwise the windows runner in the CI won't
be able to start the request before it expires
* fix timing in purchasing test
allow for a bit more time for a request to
be submitted
* fix timing of request submission in test
windows ci is so slow, it can take up to 40 seconds
just to submit a storage request to hardhat
* increase proof period to 90 seconds
* adjust timing of integration tests
reason: with the increased period length of 90 seconds, it
can take longer to wait for a stable challenge at the
beginning of a period.
* increase CI timeout to 2 hours
* Fix slow builds on windows
apparently it takes windows 2-3 seconds to
resolve "localhost" to 127.0.0.1 for every
json-rpc connection that we make 🤦
* chore: bump dependencies, including nim-ethers with chronos v4 support
Bumps the following dependencies:
- nim-ethers to commit 507ac6a4cc71cec9be7693fa393db4a49b52baf9 which contains a pinned nim-eth version. This is to be replaced by a versioned library, so it will be pinned to a particular version. There is a crucial fix in this version of ethers that fixes nonce management which is causing issues in the Codex testnet.
- nim-json-rpc to v0.4.4
- nim-json-serialization to v0.2.8
- nim-serde to v1.2.2
- nim-serialization to v0.2.4
Currently, one of the integration tests is failing.
* fix integration test
- When a state's run was cancelled, it was being caught as an error due to catching all CatchableErrors. This caused a state transition to SaleErrored, however cancellation of run was not actually an error. Handling this correctly fixed the issue.
- Stopping of the clock was moved to after `HostInteractions` (sales) which avoided an assertion around getting time when the clock was not started.
* bump ethers to include nonce fix and filter not found fix
* bump ethers: fixes missing symbol not exported in ethers
* Fix cirdl test imports/exports
* Debugging in ci
* Handle CancelledErrors for state.run in one place only
* Rename `config` to `configuration`
There was a symbol clash preventing compilation and it was easiest to rename `config` to `configuration` in the contracts. Not even remotely ideal, but it was the only way.
* bump ethers to latest
Prevents an issue were `JsonNode.items` symbol could not be found
* More changes to support `config` > `configuration`
* cleanup
* testing to see if this fixes failure in ci
* bumps contracts
- ensures slot is free before allowing reservation
- renames config to configuration to avoid symbol clash
* Add metadata to the manifest
* Remove useless import
* Fix the openapi documentation
* Use optional fields instead of default values
* Remove testRestApi target
* Return failure when the protobuf cannot get the field
* Set download headers and fix cors headers when an error is returned
* Add tests to verify the download headers
* Try to adjust the content length header
* Fix convertion to string
* Remove the content length header
* Remove testRestApi target
* Removing debug messages
* Avoid cancelling states when slot is filled
* improve logging
Improves logging for situations where a Sale should be ignored instead of being considered an error, including when reservation is not allowed and when a slot was filled by another host.
* remove onSlotFilled unit tests from states
* Adds API for fetching manifest only and downloading dataset without stream
* Updates openapi.yaml
* Adds tests for downloading manifest-only and without stream.
* review comments by Giuliano
* updates test clients
* All sort of tweaks
* docs: availability's minPrice doc
* Revert changes to the two node test example
* Change default EC params in REST API
Change default EC params in REST API to 3 nodes and 1 tolerance.
Adjust integration tests to honour these settings.
---------
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* Introduces a start method to prover
* Moves backend creation into start method
* sets up three paths for backend initialization
* Extracts backend initialization to backend-factory
* Implements loading backend from cli files or previously downloaded local files
* Wires up downloading and unzipping
* functional implementation
* Fixes testprover.nim
* Sets up tests for backendfactory
* includes libzip-dev
* pulls in updated contracts
* removes integration cli tests for r1cs, wasm, and zkey file arguments.
* Fixes issue where inner-scope values are lost before returning
* sets local proof verification for dist-test images
* Adds two traces and bumps nim-ethers
* Adds separate path for circuit files
* Create circuit dir if not exists
* fix: make sure requestStorage is mined
* fix: correct place to plug confirm
* test: fixing contracts tests
* Restores gitmodules
* restores nim-datastore reference
* Sets up downloader exe
* sets up tool skeleton
* implements getting of circuit hash
* Implements downloader tool
* sets up test skeleton
* Implements test for cirdl
* includes testTools in testAll
* Cleanup building.md
* cleans up previous downloader implementation
* cleans up testbackendfactory
* moves start of prover into node.nim
* Fills in arguments in example command
* Initializes backend in prover constructor
* Restores tests
* Restores tests for cli instructions
* Review comments by Dmitriy, part 1
* Quotes path in download instruction.
* replaces curl with chronos http session
* Moves cirdl build output to 'build' folder.
* Fixes chronicles log output
* Add cirdl support to the codex Dockerfile
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Add cirdl support to the docker entrypoint
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Add cirdl support to the release workflow
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Disable verify_circuit flag for releases
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Removes backendFactory placeholder type
* wip
* Replaces zip library with status-im/zippy library (which supports zip and tar)
* Updates cirdl to not change circuitdir folder
* Switches from zip to tar.gz
* Review comments by Dmitriy
* updates codex-contracts-eth
* Adds testTools to CI
* Adds check for access to config.circuitdir
* Update fixture circuit zkey
* Update matrix to run tools tests on Windows
* Adds 'deps' dependency for cirdl
* Adjust docker-entrypoint.sh to use CODEX_CIRCUIT_DIR env var
* Review comments by Giuliano
---------
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: Veaceslav Doina <20563034+veaceslavdoina@users.noreply.github.com>
* add changes to use chronos v4 in compat mode
* switch chronos to compat fix branch
* use nimbus-build-system with configurable Nim repo
* add missing imports
* add missing await
* bump compat
* pin nim version in Makefile
* add await instead of asyncSpawn to advertisement queue loop
* bump DHT to v0.5.0
* allow error state of `onBatch` to propagate upwards in test code
* pin Nim compiler commit to avoid fetching stale branch
* make CI build against branch head instead of merge
* fix handling of return values in testslotqueue
* Rest API: add erasure coding constraints when requesting storage
* clean up
* Make error message for "dataset too small" more informative.
* fix API integration test
---------
Co-authored-by: gmega <giuliano.mega@gmail.com>
* Fix verifiable manifest constructor
* Add integration test for verifiable manifest download
Add integration test for testing download of verifiable dataset after creating request for storage
* add missing import
* add testecbug to integration suite
* Remove hardhat instance from integration test
* change description, drop echo
---------
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: gmega <giuliano.mega@gmail.com>
* integration: move REST API tests into their own module
* integration: move upload and download tests into their own module
* integration: move purchasing tests into their own module
* integration: move marketplace tests to the right module
* integration: mine a block *after* starting nodes
To ensure that tests involving multiple nodes do
not start with out-of-sync clocks
* Fix: do not swallow CancellationErrors
* integration: avoid underflow in UInt256
* network: remove unnecessary error handling
No Exceptions can occur, only Defects, because everything
is asyncSpawned.
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* network: do not raise in asyncSpawned proc
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* Aligns response with openapi spec
* Fixes json serialization for local content response
* Review comments by Mark
* Fixes missed rename
* Removes array type for dataList
* json > nim-serde bump
Should wait until serde is integrated into nim-ethers before making these changes as there will be less import exceptions required.
* bump nim-serde
* change func to proc due to chronicles side effects
* import serde into utils/json, use as proxy
import nim-serde into utils/json and use utils/json as a proxy for serde functions, including overloading `%` and `fromJson` for application types.
* update tests to use serde
* bump serde to latest
* remove testjson -- no longer needed
* bump serde in nimble
* updates to reconcile rebase with master
* use a real verifying contract address
* contracts: cleanup
* marketplacesuite: set correct circuit files, interval mining
* Proofs tests updates
Contains changes to get the proving tests working reliably.
* integration: use correct circom artifacts for creating proofs
* integration: cleanup
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
