Commit Graph

11 Commits

Author SHA1 Message Date
Marcin Czenko 92a0eda79a
Validator historical state restoration (#922)
* adds a new overload of queryPastEvents allowing to query past events based on timestamp in the past

* adds state restoration to validator

* refactors a bit to get the tests back to work

* replaces deprecated generic methods from Market with methods for specific event types

* Refactors binary search

* adds market tests for querying past SlotFilled events and binary search

* Takes into account that <<earliest>> block available is not necessarily the genesis block

* Adds more logging and makes testing earliest block boundary more reliable

* adds validation tests for historical state restoration

* adds mockprovider to simplify and improve testing of the edge conditions

* adds slot reservation to the new tests after rebasing

* adds validation groups and group index in logs of validator

* adds integration test with two validators

* adds comment on how to enable logging in integration test executable itself

* testIntegration: makes list is running nodes injected and available in the body of the test

* validation: adds integration test for historical state

* adds more logging to validator

* integration test: validator only looks 30 days back for historical state

* adds logging of the slotState when removing slots during validation

* review and refactor validator integration tests

* adds validation to the set of integration tests

* Fixes mistyped name of the mock provider module in testMarket

* Fixes a typo in the name of the validation suite in integration tests

* Makes validation unit test a bit easier to follow

* better use of logScopes to reduce duplication

* improves timing and clarifies the test conditions

* uses http as default RPC provider for nodes running in integration tests as a workaround for dropped subscriptions

* simplifies the validation integration tests by waiting for failed request instead of tracking slots

* adds config option allowing selectively to set different provider url

* Brings back the default settings for RPC provider in integration tests

* use http RPC provider for clients in validation integration tests

* fine-tune the tests

* Makes validator integration test more robust - adds extra tracking

* brings tracking of marketplace event back to validator integration test

* refactors integration tests

* deletes tmp file

* adds <<return>> after forcing integration test to fail preliminarily

* re-enables all integration tests and matrix

* stops debug output in CI

* allows to choose a different RPC provider for a given integration test suite

* fixes signature of <<getBlock>> method in mockProvider

* adds missing import which seem to be braking integration tests on windows

* makes sure that clients, SPs, and validators use the same provider url

* makes validator integration tests using http at 127.0.0.1:8545

* testvalidator: stop resubscribing as we are now using http polling as rpc provider

* applying review comments

* groups queryPastStorage overrides together (review comment)

* groups the historical validation tests into a sub suite

* removes the temporary extensions in marketplacesuite and multinodesuite allowing to specify provider url

* simplifies validation integration tests

* Removes debug logs when waiting for request to fail

* Renaming waitForRequestFailed => waitForRequestToFail

* renames blockNumberForBlocksAgo to pastBlockTag and makes it private

* removes redundant debugging logs

* refines logging in validation

* removes dev logging from mockmarket

* improves exception handling in provider helper procs and prepares for extraction to a separate module

* Uses chronos instead of std/times for Duration

* extracts provider and binary search helpers to a separate module

* removes redundant log entry params from validator

* unifies the notation to consistently use method call syntax

* reuses ProviderError from nim-ethers in the provider extension

* clarifies the comment in multinodesuite

* uses == operator to check the predefined tags and raises exception when `BlockTag.pending` is requested.

* when waiting for request to fail, we break on any request state that is not Started

* removes tests that were moved to testProvider from testMarket

* extracts tests that use MockProvider to a separate async suite

* improves performance of the historical state restoration

* removing redundant log messages in validator (groupIndex and groups)

* adds testProvider to testContracts group

* removes unused import in testMarket
2024-12-14 05:07:55 +00:00
markspanbroek 29433bad9a
Fix concurrency issues (#993)
* Use http subscriptions instead of websocket for tests

To work around this issue when subscriptions are
inactive for more than 5 minutes:
https://github.com/NomicFoundation/hardhat/issues/2053

Use 100 millisecond polling; default polling interval
of 4 seconds is too close to the 5 second timeout for
`check eventually`.

* use .confirm(1) instead of confirm(0)

confirm(0) doesn't wait at all, confirm(1) waits
for the transaction to be mined

* speed up partial payout integration test

* update nim-ethers to version 0.10.0

includes fixes for http polling and .confirm()

* fix timing of marketplace tests

allow for a bit more time to withdraw funds

* use .confirm(1) in marketplace tests

to ensure that the transaction has been processed
before continuing with the test

* fix timing issue in validation unit test

* fix proof integration test

there were two logic errors in this test:
- a slot is freed anyway at the end of the contract
- when starting the request takes a long time, the
  first slot can already be freed because there were
  too many missing proofs

* fix intermittent error in contract tests

currentTime() doesn't always correctly reflect
the time of the next transaction

* reduce number of slots in integration test

otherwise the windows runner in the CI won't
be able to start the request before it expires

* fix timing in purchasing test

allow for a bit more time for a request to
be submitted

* fix timing of request submission in test

windows ci is so slow, it can take up to 40 seconds
just to submit a storage request to hardhat

* increase proof period to 90 seconds

* adjust timing of integration tests

reason: with the increased period length of 90 seconds, it
can take longer to wait for a stable challenge at the
beginning of a period.

* increase CI timeout to 2 hours

* Fix slow builds on windows

apparently it takes windows 2-3 seconds to
resolve "localhost" to 127.0.0.1 for every
json-rpc connection that we make 🤦
2024-11-25 11:23:04 +00:00
Marcin Czenko 5ace105a66
Validator - support partitioning of the slot id space (#890)
* Adds validatorPartitionSize and validatorPartitionIndex config options

* adds partitioning options to the validation type

* adds partitioning logic to the validator

* ignores partitionIndex when partitionSize is either 0 or 1

* clips the partition index to <<partitionIndex mod partitionSize>>

* handles negative values for the validation partition index

* updates long description of the new validator cli options

* makes default partitionSize to be 0 for better backward compatibility

* Improving formatting on validator CLI

* reactors validation params into a separate type and simplifies validation of validation params

* removes suspected duplication

* fixes typo in validator CLI help

* updates README

* Applies review comments - using optionals and range types to handle validation params

* Adds initializer to the configFactory for validatorMaxSlots

* [Review] update validator CLI description and README

* [Review]: renaming validationParams to validationConfig (config)

* [Review]: move validationconfig.nim to a higher level (next to validation.nim)

* changes backing type of MaxSlots to be int and makes sure slots are validated without limit when maxSlots is set to 0

* adds more end-to-end test for the validator and the groups

* fixes typo in README and conf.nim

* makes `maxSlotsConstraintRespected` and `shouldValidateSlot` private + updates the tests

* fixes public address of the signer account in the marketplace tutorial

* applies review comments - removes two tests
2024-10-02 22:00:40 +00:00
Eric 61ec4275c8
Revert "remove call to `waitFor` from clock.now and make it async (#691)" (#707)
This reverts commit 591be9446a.
2024-02-16 22:12:16 +00:00
markspanbroek 2cf892c467
Smart contracts update: Groth16Proof instead of bytes (#683)
* Smart contracts update: Groth16Proof instead of bytes

* Use dummy verifier for now, until we can create ZK proofs

* Fix tests: submit proof only when slot is filled

* Submit dummy proofs for now

* More detailed log when proof submission failed

* Use dummy verifier for integration tests

For now at least

* Fix mistake in blanket renaming to ethProvider

* Update to latest codex-contracts-eth

* feat: zkey-hash from chain

* Fix zkeyHash

---------

Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
2024-02-07 07:50:35 +01:00
Giuliano Mega 591be9446a
remove call to `waitFor` from clock.now and make it async (#691) 2024-02-05 08:36:28 -03:00
markspanbroek fd3c566b25
update asynctest to 0.5.1 (#671)
* update asynctest to 0.5.0

Co-Authored-By: gmega <giuliano.mega@gmail.com>

* update remaining package imports

* add asynctest package wrapper

---------

Co-authored-by: gmega <giuliano.mega@gmail.com>
2024-01-29 17:03:51 -03:00
markspanbroek d3a22a7b7b
Fix slot queue push (#542)
* [sales] remove availability check before adding to slot queue

* [sales] add missing return statement

* [tests] remove 'eventuallyCheck' helper

* [sales] remove reservations from slot queue

* [tests] rename module `eventually` -> `always`

* [sales] increase slot queue size

Because it will now also hold items for which we haven't
checked availability yet.
2023-09-04 16:42:09 +02:00
Dmitriy Ryajov bd594c9aaf
Create memory-leak detecting test suite (#226)
* adding tracker for streamstore

* adding tracker tests

* Sets up tracker helper functions and closes streams in testnode.nim

* Deploying checksuite for memory leak tracker checking.

* Successfully deploys checksuite and asyncchecksuite.

* Fix leak in testpor.nim

* Fixes leaked storestream in testnetwork.nim

* Fixes integration tests

* Cleanup

* cleanup comment by Mark

---------

Co-authored-by: benbierens <thatbenbierens@gmail.com>
2023-06-22 12:01:21 -06:00
markspanbroek 8cfac5fee8
[validator] fix intermittent test failure (#402)
Test could fail when it was started right at the
boundary of a period (on the system clock)
2023-05-01 17:22:13 +02:00
markspanbroek d56eb6aee1
Validator (#387)
* [contracts] Add SlotFreed event

* [integration] allow test node to be stopped twice

* [cli] add --validator option

* [contracts] remove dead code

* [contracts] instantiate OnChainMarket and OnChainClock only once

* [contracts] add Validation

* [sales] remove duplicate import

* [market] add missing import

* [market] subscribe to all SlotFilled events

* [market] add freeSlot()

* [sales] fix warnings

* [market] subscribe to SlotFreed events

* [contracts] fix warning

* [validator] keep track of filled slots

* [validation] remove slots that have ended

* [proving] absorb Proofs into Market

Both Proofs and Market are abstractions around
the Marketplace contract, having them separately
is more trouble than it's worth at the moment.

* [market] add markProofAsMissing()

* [clock] speed up waiting for clock in tests

* [validator] mark proofs as missing

* [timer] fix error on node shutdown

* [cli] handle --persistence and --validator separately

* [market] allow retrieval of proof timeout value

* [validator] do not subscribe to SlotFreed events

Freed slots are already handled in
removeSlotsThatHaveEnded(), and onSlotsFreed()
interfered with its iterator.

* [validator] Start validation at the start of a new period

To decrease the likelihood that we hit the validation timeout.

* [validator] do not mark proofs as missing after timeout

* [market] check whether proof can be marked as missing

* [validator] simplify validation

Simulate a transaction to mark proof as missing, instead
of trying to keep track of all the conditions that may
lead to a proof being marked as missing.

* [build] use nim-ethers PR #40

Uses "pending" blocktag instead of "latest" blocktag
for better simulation of transactions before sending
them.

https://github.com/status-im/nim-ethers/pull/40

* [integration] integration test for validator

* [validator] monitor a maximum number of slots

Adds cli parameter --validator-max-slots.

* [market] fix missing collateral argument

After rebasing, add the new argument to fillSlot calls.

* [build] update to nim-ethers 0.2.5

* [validator] use Set instead of Table to keep track of slots

* [validator] add logging

* [validator] add test for slot failure

* [market] use "pending" blocktag to use more up to date block time

* [contracts] remove unused import

* [validator] fix: wait until after period ends

The smart contract checks that 'end < block.timestamp',
so we need to wait until the block timestamp is greater
than the period end.
2023-04-19 15:06:00 +02:00