Commit Graph

81 Commits

Author SHA1 Message Date
jangko ecab64adab
reduce imported but not used warning when evmc enabled 2022-10-03 16:41:33 +07:00
jangko 16bc2de1cf
update EF test fixtures and fixes to pass all tests 2022-10-03 16:41:32 +07:00
Jacek Sieka c2ed731fa5
eth: adapt to smaller eth_types (#1210) 2022-09-03 20:15:35 +02:00
jangko a087d65542
reduce test suite time consumption 2022-07-30 08:43:15 +07:00
Zahary Karadjov 137eb97766 Initial implementation of the merge spec
Includes a simple test harness for the merge interop M1 milestone

This aims to enable connecting nimbus-eth2 to nimbus-eth1 within
the testing protocol described here:

https://github.com/status-im/nimbus-eth2/blob/amphora-merge-interop/docs/interop_merge.md

To execute the work-in-progress test, please run:

In terminal 1:
tests/amphora/launch-nimbus.sh

In terminal 2:
tests/amphora/check-merge-test-vectors.sh
2022-01-24 09:44:39 +02:00
Jordan Hrycaj 261c0b51a7
Redesign of BaseVMState descriptor (#923)
* Redesign of BaseVMState descriptor

why:
  BaseVMState provides an environment for executing transactions. The
  current descriptor also provides data that cannot generally be known
  within the execution environment, e.g. the total gasUsed which is
  available not before after all transactions have finished.

  Also, the BaseVMState constructor has been replaced by a constructor
  that does not need pre-initialised input of the account database.

also:
  Previous constructor and some fields are provided with a deprecated
  annotation (producing a lot of noise.)

* Replace legacy directives in production sources

* Replace legacy directives in unit test sources

* fix CI (missing premix update)

* Remove legacy directives

* chase CI problem

* rebased

* Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation

why:
  Constructing a new 'AccountsCache' descriptor can be avoided sometimes
  when the current state root is properly positioned already. Such a
  feature existed already as the update function 'initStateDB()' for the
  'BaseChanDB' where the accounts cache was linked into this desctiptor.

  The function 'initStateDB()' was removed and re-implemented into the
  'BaseVmState' constructor without optimisation. The old version was of
  restricted use as a wrong accounts cache state would unconditionally
  throw an exception rather than conceptually ask for a remedy.

  The optimised 'BaseVmState' re-initialisation has been implemented for
  the 'persistBlocks()' function.

also:
  moved some test helpers to 'test/replay' folder

* Remove unused & undocumented fields from Chain descriptor

why:
  Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
Jamie Lokier 74253c88e3
Arrow Glacier fork
Add the new [Arrow Glacier fork](https://eips.ethereum.org/EIPS/eip-4345).

Only the difficulty calculation is changed, but as a new fork it still affects
a number of places in the code.

To the best of my knowledge the change is only scheduled on Mainnet.

In addition:

- The fork date comments in `chain_config.nim` have been checked against the
  real networks, set consistently in UTC instead of random timezones, and made
  neater.  Maybe we'll keep these when transferring config to a file someday.

- It's added to forkid hash tests (EIP-2124/EIP-2364), of course.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-12-10 13:40:51 +00:00
Jordan Hrycaj 55f7a4425f
Jordan/pow cache management (#888)
* PoW wrapper for verification & mining

why:
  It eases data management of per-Epoch lookup tables. Also some unit
  tests show limits of usefulness on non-specialised machines for
  mining besides developing tests.

details:
  For PoW verification, this patch provides a pretty wrapper hiding the
  details of the ethash/Hashimoto lookup cache management.

  For mining on my development system without special hardware, the
  underlying ethash functions are prohibitively slow. It takes
   * ~20 minutes to prepare the full ethash/Hashimoto lookup dataset
   * a second to run ~25k nonce tests (in the mining loop)

  The mining part might be of some use for generating test data for
  the tx-pool, though.

* Using PowRef as replacement for EpochHashCache + hashimotoLight()

* Fix typo (CI failed)

why:
  was below log level when testing locally

* fix canonical naming
2021-12-10 08:49:57 +00:00
jangko baf508f6ae
move stateDB from VMState to chainDB
previously, every time the VMState was created, it will also create
new stateDB, and this action will nullify the advantages of cached accounts.

the new changes will conserve the accounts cache if the executed blocks
are contiguous. if not the stateDB need to be reinited.

this changes also allow rpcCallEvm and rpcEstimateGas executed properly
using current stateDB instead of creating new one each time they are called.
2021-10-28 18:57:08 +07:00
jangko 521f29c0a0
remove unused calcGasLimit code
we have new calcGasLimit tested in sealing engine.
so we can safely remove the old unused calcGasLimit
2021-08-24 18:30:52 +07:00
Jordan Hrycaj ca07c40a48
Fearture/poa clique tuning (#765)
* Provide API

details:
  API is bundled via clique.nim.

* Set extraValidation as default for PoA chains

why:
  This triggers consensus verification and an update of the list
  of authorised signers. These signers are integral part of the
  PoA block chain.

todo:
  Option argument to control validation for the nimbus binary.

* Fix snapshot state block number

why:
  Using sub-sequence here, so the len() function was wrong.

* Optional start where block verification begins

why:
  Can speed up time building loading initial parts of block chain. For
  PoA, this allows to prove & test that authorised signers can be
  (correctly) calculated starting at any point on the block chain.

todo:
  On Goerli around blocks #193537..#197568, processing time increases
  disproportionally -- needs to be understand

* For Clique test, get old grouping back (7 transactions per log entry)

why:
  Forgot to change back after troubleshooting

* Fix field/function/module-name misunderstanding

why:
  Make compilation work

* Use eth_types.blockHash() rather than utils.hash() in Clique modules

why:
  Prefer lib module

* Dissolve snapshot_misc.nim

details:
  .. into clique_verify.nim (the other source file clique_unused.nim
  is inactive)

* Hide unused AsyncLock in Clique descriptor

details:
  Unused here but was part of the Go reference implementation

* Remove fakeDiff flag from Clique descriptor

details:
  This flag was a kludge in the Go reference implementation used for the
  canonical tests. The tests have been adapted so there is no need for
  the fakeDiff flag and its implementation.

* Not observing minimum distance from epoch sync point

why:
  For compiling PoA state, the go implementation will walk back to the
  epoch header with at least 90000 blocks apart from the current header
  in the absence of other synchronisation points.

  Here just the nearest epoch header is used. The assumption is that all
  the checkpoints before have been vetted already regardless of the
  current branch.

details:
  The behaviour of using the nearest vs the minimum distance epoch is
  controlled by a flag and can be changed at run time.

* Analysing processing time (patch adds some debugging/visualisation support)

why:
  At the first half million blocks of the Goerli replay, blocks on the
  interval #194854..#196224 take exceptionally long to process, but not
  due to PoA processing.

details:
  It turns out that much time is spent in p2p/excecutor.processBlock()
  where the elapsed transaction execution time is significantly greater
  for many of these blocks.

  Between the 1371 blocks #194854..#196224 there are 223 blocks with more
  than 1/2 seconds execution time whereas there are only 4 such blocks
  before and 13 such after this range up to #504192.

* fix debugging symbol in clique_desc (causes CI failing)

* Fixing canonical reference tests

why:
  Two errors were introduced earlier but ovelooked:
   1. "Remove fakeDiff flag .." patch was incomplete
   2. "Not observing minimum distance .." introduced problem w/tests 23/24

details:
  Fixing 2. needed to revert the behaviour by setting the
  applySnapsMinBacklog flag for the Clique descriptor. Also a new
  test was added to lock the new behaviour.

* Remove cruft

why:
  Clique/PoA processing was intended to take place somewhere in
  executor/process_block.processBlock() but was decided later to run
  from chain/persist_block.persistBlock() instead.

* Update API comment

* ditto
2021-07-30 15:06:51 +01:00
Jordan Hrycaj fbff3aea68
Feature/goerli replay clique poa (#743)
* extract unused clique/mining support into separate file

why:
  mining is currently unsupported by nimbus

* Replay first 51840 transactions from Goerli block chain

why:
  Currently Goerli is loaded but the block headers are not verified.
  Replaying allows real data PoA development.

details:
  Simple stupid gzipped dump/undump layer for debugging based on
  the zlib module (no nim-faststream support.)

  This is a replay running against p2p/chain.persistBlocks() where
  the data were captured from.

* prepare stubs for PoA engine

* split executor source into sup-modules

why:
  make room for updates, clique integration should go into
  executor/update_poastate.nim

* Simplify p2p/executor.processBlock() function prototype

why:
  vmState argument always wraps basicChainDB

* split processBlock() into sub-functions

why:
  isolate the part where it will support clique/poa

* provided additional processTransaction() function prototype without _fork_ argument

why:
  with the exception of some tests, the _fork_ argument is always derived
  from the other prototype argument _vmState_

details:
  similar situation with makeReceipt()

* provide new processBlock() version explicitly supporting PoA

details:
  The new processBlock() version supporting PoA is the general one also
  supporting non-PoA networks, it needs an additional _Clique_ descriptor
  function argument for PoA state (if any.)
  The old processBlock() function without the _Clique_ descriptor argument
  retorns an error on PoA networgs (e.g. Goerli.)

* re-implemented Clique descriptor as _ref object_

why:
  gives more flexibility when moving around the descriptor object

details:
  also cleaned up a bit the clique sources

* comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 14:14:45 +01:00
jangko d47be9047f
London: fix test_blockchain_json and test_generalstate_json
new London HF requires update to test code
2021-06-30 20:41:29 +07:00
Jordan Hrycaj a49a812879
Jordan/fix some failing nohive tests (#727)
* continue importing rlp blocks

why:
  a chain of blocks to be imported might have legit blocks
  after rejected blocks

details:
  import loop only stops if the import list is exhausted or if there
  was a decoding error. this adds another four to the count of successful
  no-hive tests.

* verify DAO marked extra data field in block header

why:
  was ignored, scores another two no-hive tests

* verify minimum required difficulty in header validator

why:
  two more nohive tests to succeed

details:
  * subsumed extended header tests under validateKinship() and renamed it
    more appropriately validateHeaderAndKinship()
  * enhanced readability of p2p/chain.nim
  * cleaned up test_blockchain_json.nim

* verify positive gasUsed unless no transactions

why:
  solves another to nohive tests

details:
  straightened test_blockchain_json chech so there is no unconditional
  rejection anymore (based on the input test  scenario)
2021-06-24 16:29:21 +01:00
Jamie Lokier aee0fe39d2
EVM: Remove `vm_types2` everywhere, use common forks list instead
File `vm_types2` is obsolete.  Remove this file and divert all imports to the
common forks list outside the EVM, or in some cases they don't need it anyway.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 15:36:31 +01:00
jangko 91b50235b0 add assertion guard in test_blockchain_json's parseBlock function
we don't want any surprice because of unrecognized key in test fixture
classified as `hasException`.
2021-05-26 15:46:04 +01:00
Jordan Hrycaj 3663b1603f pulled out cache logic into separate file
why:
  handy to re-use, eg. for upcoming clique implementation
2021-05-24 07:57:21 +01:00
Jordan Hrycaj d6a5cecb98 re-wrote validation with exceptionless functions
why:
  exceptions were from test code should be avoided in production code
2021-05-24 07:57:21 +01:00
Jordan Hrycaj 40c7bdfc06 update lookup cache management
details:
  enable fifo behaviour, using cache as argument
2021-05-24 07:57:21 +01:00
Jordan Hrycaj ce8e5511e3 backport from test_blockchain_json, see issue #666 2021-05-24 07:57:21 +01:00
jangko 79a52b11fd
fixes test_blockchain_json: better error message
also remove local EthBlock type and use EthBlock from nim-eth/common
2021-05-17 01:29:03 +07:00
jangko 79044f1e92
eip2718: test_blockchain_json pass test 2021-05-15 18:09:35 +07:00
jangko 2a9c3982d9
remove code duplication from test_blockchain_json
now test_blockchain_json is using block validation code
from p2p/executor.nim instead of using it's own
block validation code.

this will reduce maintenance cost and fixes #592.
2021-05-14 15:37:30 +07:00
jangko db65f92e66
refactor vmConfiguration in test_blockchain_json
instead of using it's own blockNumber to Fork calculation,
we switch to ChainConfig to Fork to ensure smooth transition
when using common block validator.
2021-05-14 15:37:29 +07:00
jangko beb274d98c
remove unnecessary codes from test_blockchain_json
except for genesis block, we are not parsing block header from json
node anymore.

we parse block headers from block RLP, it is the same thing.
2021-05-14 15:37:29 +07:00
jangko f2491e6307
fixes crappy custom genesis and chain config parser
instead of using stdlib/json, now we switch to json_serialization
the result is much tidier code and more robust when parsing
optional fields.

fixes #635
2021-05-13 16:04:08 +07:00
jangko 639674d341
add disableParamFiltering() to both test_generalstate_json and test_blockchain_json
disableParamFiltering() is called in test_all, so its ok when
called via test_all. but when test_generalstate_json or
test_blockchain_json is called independently, and we are using
command line switch, the default param handler from std/unittest
will interfere. hence need to disable it.
2021-04-25 12:20:03 +07:00
Jamie Lokier 8a806da600
Bugfix: Fix blockchain tests: It was quietly skipping 4654 of them
The "new block chain json tests" were being skipped on Linux, but silently so
that CI didn't notice.  These are a significant part of the Ethereum test suite.

See the missing output from `make test`, also visible in CI logs for Linux
targets (prior to this commit):

      [OK] tests/fixtures/eth_tests/TransactionTests/ttGasPrice/TransactionWithGasPriceOverflow.json
      [OK] tests/fixtures/eth_tests/TransactionTests/ttGasPrice/TransactionWithHighGasPrice.json

    [Suite] new block chain json tests
            <-- nothing here

    [Suite] Fork ID tests
      [OK] MainNet
      [OK] RopstenNet

Commit 3d468a7 (`fixes path pointing to eth_tests`) renamed the JSON fixture
source directoryf in the witness-builder tests but not the regular blockchain
tests.  As a result, searching for JSON test files yielded zero results.  To
make this less likely in future, zero results is now an error.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-08 16:40:53 +01:00
Jordan Hrycaj 00ba7a2718
merge vm_forks and vm_opcode_values => vm_type2
why:
  all types, but they cannot be merged int vm_types because of a circular
  dependency.
2021-03-31 17:53:15 +01:00
Jordan Hrycaj 3a3e4d5707
provide vm_forks as import/export wrapper
details:
  moved original vm/interpreter/vm_forks.nim => vm/interpreter/nvm_forks.nim
2021-03-31 16:03:34 +01:00
jangko 03bed02512 split legacy and new test to save time 2021-01-14 23:22:28 +07:00
jangko f6c44ffcc0 fixes EIP2929 CALL opCode 2021-01-14 23:22:28 +07:00
jangko 15cacc749d
clean up EIP2929 for test 2021-01-12 16:17:00 +07:00
jangko f906d177f4
add comments about disabled EIPs 2021-01-11 15:33:30 +07:00
jangko 8f7e45fa08
fixes test codes 2021-01-11 14:54:11 +07:00
jangko 3d468a7a4c
fixes path pointing to eth_tests 2021-01-06 21:45:48 +07:00
jangko 74cccdfe9e
fixes test codes to deal with eth_tests submodule 2021-01-06 17:02:19 +07:00
jangko 165f9fea2e
reduce warnings 2020-07-21 13:15:06 +07:00
jangko 49460b6b1e
move buildWitness from test_blockchain_json to vm_state 2020-06-18 13:16:38 +07:00
jangko 3947e9a853
piggyback generate block witness test on test_blockchain_json 2020-06-06 10:26:36 +07:00
jangko 079579c1cb
fix test_blockchain_json -> add a call to accounts_cache.persist 2020-06-01 12:00:36 +07:00
andri lim af02a3b1b2
reduce unused import warnings 2020-04-15 19:05:57 +07:00
andri lim 5e694c6874
fix test_blockchain_json 2020-04-12 19:02:03 +07:00
andri lim 87bae2bb78
switch to new toFork 2020-04-12 18:02:59 +07:00
andri lim 266e0ddb1e
room for EIP-1283 2020-03-24 17:21:13 +07:00
andri lim 32574fcebe allow user to turn pruning on/off when run test 2020-02-20 09:16:29 +02:00
andri lim ed5710fa17 allow user to select legacy or new test suite for GST and BCT 2020-02-20 09:16:29 +02:00
Ștefan Talpalaru ea38893416
fix test dir name 2020-01-31 22:53:28 +01:00
andri lim df21dd44cb fixes typo 2019-12-10 12:34:48 +02:00
andri lim 9596dbc267 activates blockchain test 2019-12-10 12:34:48 +02:00