Commit Graph

105 Commits

Author SHA1 Message Date
andri lim 990718aa07
Prepare state test and blockchain test for Cancun (#1772) 2023-09-25 06:53:20 +07:00
Jordan Hrycaj 221e6c9e2f
Unified database frontend integration (#1670)
* Nimbus folder environment update

details:
* Integrated `CoreDbRef` for the sources in the `nimbus` sub-folder.
* The `nimbus` program does not compile yet as it needs the updates
  in the parallel `stateless` sub-folder.

* Stateless environment update

details:
* Integrated `CoreDbRef` for the sources in the `stateless` sub-folder.
* The `nimbus` program compiles now.

* Premix environment update

details:
* Integrated `CoreDbRef` for the sources in the `premix` sub-folder.

* Fluffy environment update

details:
* Integrated `CoreDbRef` for the sources in the `fluffy` sub-folder.

* Tools environment update

details:
* Integrated `CoreDbRef` for the sources in the `tools` sub-folder.

* Nodocker environment update

details:
* Integrated `CoreDbRef` for the sources in the
  `hive_integration/nodocker` sub-folder.

* Tests environment update

details:
* Integrated `CoreDbRef` for the sources in the `tests` sub-folder.
* The unit tests compile and run cleanly now.

* Generalise `CoreDbRef` to any `select_backend` supported database

why:
  Generalisation was just missed due to overcoming some compiler oddity
  which was tied to rocksdb for testing.

* Suppress compiler warning for `newChainDB()`

why:
  Warning was added to this function which must be wrapped so that
  any `CatchableError` is re-raised as `Defect`.

* Split off persistent `CoreDbRef` constructor into separate file

why:
  This allows to compile a memory only database version without linking
  the backend library.

* Use memory `CoreDbRef` database by default

detail:
 Persistent DB constructor needs to import `db/core_db/persistent

why:
 Most tests use memory DB anyway. This avoids linking `-lrocksdb` or
 any other backend by default.

* fix `toLegacyBackend()` availability check

why:
  got garbled after memory/persistent split.

* Clarify raw access to MPT for snap sync handler

why:
  Logically, `kvt` is not the raw access for the hexary trie (although
  this holds for the legacy database)
2023-08-04 12:10:09 +01:00
jangko 4d207e49ce
Implement plugable EVM tracer
available tracers:
- Legacy tracer
- new Json tracer
2023-08-03 16:45:39 +07:00
jangko 270dfc33e3
fix test_blockchain_json for new tests 2023-07-06 15:48:35 +07:00
jangko ff1a45e095
fix shanghai withdrawal validation
previously, the withdrawal validation is in process_block only,
but the one in persist block, which is also used in synchronizer
is not validated properly.
2023-06-26 07:46:09 +07:00
andri lim 26a8759c34
implementation of EIP-4844: Shard Blob Transactions (#1440)
* EIP-4844: add pointEvaluation precompiled contract

* EIP-4844: validate transaction and block header

* EIP-4844: implement DataHash Op Code

* EIP-4844: txPool support excessDataGas calculation

* EIP-4844: make sure tx produce correct txHash

* EIP-4844: node should not automatically broadcast blob tx to it's peers

* EIP-4844: add test cases

* EIP-4844: add EIP-4844 support to t8n tool

* EIP-4844: update nim-eth to branch eip-4844

* fix t8n transaction decoding

* add t8n test data

* EIP-4844: fix blobHash opcode

* disable blobHash test when evmc_enable
2023-06-24 20:56:44 +07:00
jangko 8700d8b1e1
reduce compiler warnings 2023-06-12 12:58:53 +07:00
jangko cbd593f514
bump nim-eth and accompanying fixes 2023-05-31 19:10:17 +07:00
jangko 4e58f9d79a
modexp precompile allow arbitrary input length 2023-05-11 07:21:03 +07:00
jangko 633f135c72
align accounts cache with EIP158/161
Some nomenclature used in accounts cache are not what described
in EIP158/161, therefore causing confusion and introduce bugs.
Now it should be fixed.
2023-03-18 18:13:06 +07:00
Adam Spitz 8d9b2522ea
More on withdrawals (#1508)
* Gwei conversion should use u256 because u64 can overflow.

* Make withdrawals follow the EIP-158 state-clearing rules.

(i.e. Empty accounts should be deleted.)

* Allow the zero address in normalizeNumber.

(Necessary for one of the new withdrawals-related tests.)

* Another fix with a withdrawals-related test.
2023-03-17 14:16:24 -04:00
Adam Spitz ec2bd4a9c5
More work on withdrawals (#1503)
* Gwei conversion should use u256 because u64 can overflow.

* Make withdrawals follow the EIP-158 state-clearing rules.

(i.e. Empty accounts should be deleted.)

* Allow the zero address in normalizeNumber.

(Necessary for one of the new withdrawals-related tests.)
2023-03-16 16:34:47 -04:00
Adam Spitz d8a1adacaa
More work on withdrawals (#1482)
* Part of EIP-4895: add withdrawals processing to block processing.

* Refactoring: extracted the engine API handler bodies into procs.

Intending to implement the V2 versions next. (I need the bodies to be
in separate procs so that multiple versions can use them.)

* Working on Engine API changes for Shanghai.

* Updated nim-web3, resolved ambiguity in Hash256 type.

* Updated nim-eth3 to point to master, now that I've merged that.

* I'm confused about what's going on with engine_client.

But let's try resolving this Hash256 ambiguity.

* Still trying to fix this conflict with the Hash256 types.

* Does this work now that nimbus-eth2 has been updated?

* Corrected blockValue in getPayload responses back to UInt256.

c834f67a37

* Working on getting the withdrawals-related tests to pass.

* Fixing more of those Hash256 ambiguities.

(I'm not sure why the nim-web3 library introduced a conflicting type
named Hash256, but right now I just want to get this code to compile again.)

* Bumped a couple of libraries to fix some error messages.

* Needed to get "make fluffy-tools" to pass, too.

* Getting "make nimbus_verified_proxy" to build.
2023-03-09 18:40:55 -05:00
Adam Spitz fad3ed64cf
Time based forking (#1465)
* Refactoring in preparation for time-based forking.

* Timestamp-based hard-fork-transition.

* Workaround SideEffect issue / compiler bug for both failing locations in Portal history code

---------

Co-authored-by: kdeme <kim.demey@gmail.com>
2023-02-16 12:40:07 +01:00
Kim De Mey 73e93f1f11
Reduce Nim 1.6 compiler warnings & hints
* Reduce Nim 1.6 compiler warnings/hints for Fluffy and Nimbus proxy

Mostly raises Defect removals, TaintedString removal and some
unnecessary imports.

Also updating the copyright years alongside.

* Further reduce Nim 1.6 compiler warnings/hints for Nimbus
2023-01-31 13:38:08 +01:00
jangko 6814140c63
unify coinbase state clearing of gst, evmstate, and t8n
it is troublesome if we have to fix it in three places
in case of a bug, it also reduce code duplication.
2023-01-14 17:17:55 +07:00
jangko 192646ca8a
fix test_blockchain_json lastBlockHash comparison 2023-01-14 15:56:08 +07:00
jangko a232c7eb1d
reduce code duplication in multiple test runners 2023-01-13 13:04:55 +07:00
jangko 13e5b1a76b
fix persistHeaderToDb, allow POS block header to be canonical 2022-12-10 09:02:30 +07:00
jangko 7bab58731b
simplify test_blockchain_json a bit 2022-12-09 15:26:53 +07:00
jangko 53e71e8837
better hardForkTransition usage 2022-12-05 15:46:37 +07:00
jangko 94a94c5b65 implement better hardfork management 2022-12-02 13:51:42 +07:00
jangko 66439d69ca
unify chain config parser of t8n, bc test, and consensus simulator 2022-11-25 12:26:29 +07:00
jangko a48cc04ea7
add gray glacier difficulty calculator
fixes #1264
2022-10-19 09:51:27 +07:00
jangko ecab64adab
reduce imported but not used warning when evmc enabled 2022-10-03 16:41:33 +07:00
jangko 16bc2de1cf
update EF test fixtures and fixes to pass all tests 2022-10-03 16:41:32 +07:00
Jacek Sieka c2ed731fa5
eth: adapt to smaller eth_types (#1210) 2022-09-03 20:15:35 +02:00
jangko a087d65542
reduce test suite time consumption 2022-07-30 08:43:15 +07:00
Zahary Karadjov 137eb97766 Initial implementation of the merge spec
Includes a simple test harness for the merge interop M1 milestone

This aims to enable connecting nimbus-eth2 to nimbus-eth1 within
the testing protocol described here:

https://github.com/status-im/nimbus-eth2/blob/amphora-merge-interop/docs/interop_merge.md

To execute the work-in-progress test, please run:

In terminal 1:
tests/amphora/launch-nimbus.sh

In terminal 2:
tests/amphora/check-merge-test-vectors.sh
2022-01-24 09:44:39 +02:00
Jordan Hrycaj 261c0b51a7
Redesign of BaseVMState descriptor (#923)
* Redesign of BaseVMState descriptor

why:
  BaseVMState provides an environment for executing transactions. The
  current descriptor also provides data that cannot generally be known
  within the execution environment, e.g. the total gasUsed which is
  available not before after all transactions have finished.

  Also, the BaseVMState constructor has been replaced by a constructor
  that does not need pre-initialised input of the account database.

also:
  Previous constructor and some fields are provided with a deprecated
  annotation (producing a lot of noise.)

* Replace legacy directives in production sources

* Replace legacy directives in unit test sources

* fix CI (missing premix update)

* Remove legacy directives

* chase CI problem

* rebased

* Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation

why:
  Constructing a new 'AccountsCache' descriptor can be avoided sometimes
  when the current state root is properly positioned already. Such a
  feature existed already as the update function 'initStateDB()' for the
  'BaseChanDB' where the accounts cache was linked into this desctiptor.

  The function 'initStateDB()' was removed and re-implemented into the
  'BaseVmState' constructor without optimisation. The old version was of
  restricted use as a wrong accounts cache state would unconditionally
  throw an exception rather than conceptually ask for a remedy.

  The optimised 'BaseVmState' re-initialisation has been implemented for
  the 'persistBlocks()' function.

also:
  moved some test helpers to 'test/replay' folder

* Remove unused & undocumented fields from Chain descriptor

why:
  Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
Jamie Lokier 74253c88e3
Arrow Glacier fork
Add the new [Arrow Glacier fork](https://eips.ethereum.org/EIPS/eip-4345).

Only the difficulty calculation is changed, but as a new fork it still affects
a number of places in the code.

To the best of my knowledge the change is only scheduled on Mainnet.

In addition:

- The fork date comments in `chain_config.nim` have been checked against the
  real networks, set consistently in UTC instead of random timezones, and made
  neater.  Maybe we'll keep these when transferring config to a file someday.

- It's added to forkid hash tests (EIP-2124/EIP-2364), of course.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-12-10 13:40:51 +00:00
Jordan Hrycaj 55f7a4425f
Jordan/pow cache management (#888)
* PoW wrapper for verification & mining

why:
  It eases data management of per-Epoch lookup tables. Also some unit
  tests show limits of usefulness on non-specialised machines for
  mining besides developing tests.

details:
  For PoW verification, this patch provides a pretty wrapper hiding the
  details of the ethash/Hashimoto lookup cache management.

  For mining on my development system without special hardware, the
  underlying ethash functions are prohibitively slow. It takes
   * ~20 minutes to prepare the full ethash/Hashimoto lookup dataset
   * a second to run ~25k nonce tests (in the mining loop)

  The mining part might be of some use for generating test data for
  the tx-pool, though.

* Using PowRef as replacement for EpochHashCache + hashimotoLight()

* Fix typo (CI failed)

why:
  was below log level when testing locally

* fix canonical naming
2021-12-10 08:49:57 +00:00
jangko baf508f6ae
move stateDB from VMState to chainDB
previously, every time the VMState was created, it will also create
new stateDB, and this action will nullify the advantages of cached accounts.

the new changes will conserve the accounts cache if the executed blocks
are contiguous. if not the stateDB need to be reinited.

this changes also allow rpcCallEvm and rpcEstimateGas executed properly
using current stateDB instead of creating new one each time they are called.
2021-10-28 18:57:08 +07:00
jangko 521f29c0a0
remove unused calcGasLimit code
we have new calcGasLimit tested in sealing engine.
so we can safely remove the old unused calcGasLimit
2021-08-24 18:30:52 +07:00
Jordan Hrycaj ca07c40a48
Fearture/poa clique tuning (#765)
* Provide API

details:
  API is bundled via clique.nim.

* Set extraValidation as default for PoA chains

why:
  This triggers consensus verification and an update of the list
  of authorised signers. These signers are integral part of the
  PoA block chain.

todo:
  Option argument to control validation for the nimbus binary.

* Fix snapshot state block number

why:
  Using sub-sequence here, so the len() function was wrong.

* Optional start where block verification begins

why:
  Can speed up time building loading initial parts of block chain. For
  PoA, this allows to prove & test that authorised signers can be
  (correctly) calculated starting at any point on the block chain.

todo:
  On Goerli around blocks #193537..#197568, processing time increases
  disproportionally -- needs to be understand

* For Clique test, get old grouping back (7 transactions per log entry)

why:
  Forgot to change back after troubleshooting

* Fix field/function/module-name misunderstanding

why:
  Make compilation work

* Use eth_types.blockHash() rather than utils.hash() in Clique modules

why:
  Prefer lib module

* Dissolve snapshot_misc.nim

details:
  .. into clique_verify.nim (the other source file clique_unused.nim
  is inactive)

* Hide unused AsyncLock in Clique descriptor

details:
  Unused here but was part of the Go reference implementation

* Remove fakeDiff flag from Clique descriptor

details:
  This flag was a kludge in the Go reference implementation used for the
  canonical tests. The tests have been adapted so there is no need for
  the fakeDiff flag and its implementation.

* Not observing minimum distance from epoch sync point

why:
  For compiling PoA state, the go implementation will walk back to the
  epoch header with at least 90000 blocks apart from the current header
  in the absence of other synchronisation points.

  Here just the nearest epoch header is used. The assumption is that all
  the checkpoints before have been vetted already regardless of the
  current branch.

details:
  The behaviour of using the nearest vs the minimum distance epoch is
  controlled by a flag and can be changed at run time.

* Analysing processing time (patch adds some debugging/visualisation support)

why:
  At the first half million blocks of the Goerli replay, blocks on the
  interval #194854..#196224 take exceptionally long to process, but not
  due to PoA processing.

details:
  It turns out that much time is spent in p2p/excecutor.processBlock()
  where the elapsed transaction execution time is significantly greater
  for many of these blocks.

  Between the 1371 blocks #194854..#196224 there are 223 blocks with more
  than 1/2 seconds execution time whereas there are only 4 such blocks
  before and 13 such after this range up to #504192.

* fix debugging symbol in clique_desc (causes CI failing)

* Fixing canonical reference tests

why:
  Two errors were introduced earlier but ovelooked:
   1. "Remove fakeDiff flag .." patch was incomplete
   2. "Not observing minimum distance .." introduced problem w/tests 23/24

details:
  Fixing 2. needed to revert the behaviour by setting the
  applySnapsMinBacklog flag for the Clique descriptor. Also a new
  test was added to lock the new behaviour.

* Remove cruft

why:
  Clique/PoA processing was intended to take place somewhere in
  executor/process_block.processBlock() but was decided later to run
  from chain/persist_block.persistBlock() instead.

* Update API comment

* ditto
2021-07-30 15:06:51 +01:00
Jordan Hrycaj fbff3aea68
Feature/goerli replay clique poa (#743)
* extract unused clique/mining support into separate file

why:
  mining is currently unsupported by nimbus

* Replay first 51840 transactions from Goerli block chain

why:
  Currently Goerli is loaded but the block headers are not verified.
  Replaying allows real data PoA development.

details:
  Simple stupid gzipped dump/undump layer for debugging based on
  the zlib module (no nim-faststream support.)

  This is a replay running against p2p/chain.persistBlocks() where
  the data were captured from.

* prepare stubs for PoA engine

* split executor source into sup-modules

why:
  make room for updates, clique integration should go into
  executor/update_poastate.nim

* Simplify p2p/executor.processBlock() function prototype

why:
  vmState argument always wraps basicChainDB

* split processBlock() into sub-functions

why:
  isolate the part where it will support clique/poa

* provided additional processTransaction() function prototype without _fork_ argument

why:
  with the exception of some tests, the _fork_ argument is always derived
  from the other prototype argument _vmState_

details:
  similar situation with makeReceipt()

* provide new processBlock() version explicitly supporting PoA

details:
  The new processBlock() version supporting PoA is the general one also
  supporting non-PoA networks, it needs an additional _Clique_ descriptor
  function argument for PoA state (if any.)
  The old processBlock() function without the _Clique_ descriptor argument
  retorns an error on PoA networgs (e.g. Goerli.)

* re-implemented Clique descriptor as _ref object_

why:
  gives more flexibility when moving around the descriptor object

details:
  also cleaned up a bit the clique sources

* comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 14:14:45 +01:00
jangko d47be9047f
London: fix test_blockchain_json and test_generalstate_json
new London HF requires update to test code
2021-06-30 20:41:29 +07:00
Jordan Hrycaj a49a812879
Jordan/fix some failing nohive tests (#727)
* continue importing rlp blocks

why:
  a chain of blocks to be imported might have legit blocks
  after rejected blocks

details:
  import loop only stops if the import list is exhausted or if there
  was a decoding error. this adds another four to the count of successful
  no-hive tests.

* verify DAO marked extra data field in block header

why:
  was ignored, scores another two no-hive tests

* verify minimum required difficulty in header validator

why:
  two more nohive tests to succeed

details:
  * subsumed extended header tests under validateKinship() and renamed it
    more appropriately validateHeaderAndKinship()
  * enhanced readability of p2p/chain.nim
  * cleaned up test_blockchain_json.nim

* verify positive gasUsed unless no transactions

why:
  solves another to nohive tests

details:
  straightened test_blockchain_json chech so there is no unconditional
  rejection anymore (based on the input test  scenario)
2021-06-24 16:29:21 +01:00
Jamie Lokier aee0fe39d2
EVM: Remove `vm_types2` everywhere, use common forks list instead
File `vm_types2` is obsolete.  Remove this file and divert all imports to the
common forks list outside the EVM, or in some cases they don't need it anyway.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 15:36:31 +01:00
jangko 91b50235b0 add assertion guard in test_blockchain_json's parseBlock function
we don't want any surprice because of unrecognized key in test fixture
classified as `hasException`.
2021-05-26 15:46:04 +01:00
Jordan Hrycaj 3663b1603f pulled out cache logic into separate file
why:
  handy to re-use, eg. for upcoming clique implementation
2021-05-24 07:57:21 +01:00
Jordan Hrycaj d6a5cecb98 re-wrote validation with exceptionless functions
why:
  exceptions were from test code should be avoided in production code
2021-05-24 07:57:21 +01:00
Jordan Hrycaj 40c7bdfc06 update lookup cache management
details:
  enable fifo behaviour, using cache as argument
2021-05-24 07:57:21 +01:00
Jordan Hrycaj ce8e5511e3 backport from test_blockchain_json, see issue #666 2021-05-24 07:57:21 +01:00
jangko 79a52b11fd
fixes test_blockchain_json: better error message
also remove local EthBlock type and use EthBlock from nim-eth/common
2021-05-17 01:29:03 +07:00
jangko 79044f1e92
eip2718: test_blockchain_json pass test 2021-05-15 18:09:35 +07:00
jangko 2a9c3982d9
remove code duplication from test_blockchain_json
now test_blockchain_json is using block validation code
from p2p/executor.nim instead of using it's own
block validation code.

this will reduce maintenance cost and fixes #592.
2021-05-14 15:37:30 +07:00
jangko db65f92e66
refactor vmConfiguration in test_blockchain_json
instead of using it's own blockNumber to Fork calculation,
we switch to ChainConfig to Fork to ensure smooth transition
when using common block validator.
2021-05-14 15:37:29 +07:00
jangko beb274d98c
remove unnecessary codes from test_blockchain_json
except for genesis block, we are not parsing block header from json
node anymore.

we parse block headers from block RLP, it is the same thing.
2021-05-14 15:37:29 +07:00
jangko f2491e6307
fixes crappy custom genesis and chain config parser
instead of using stdlib/json, now we switch to json_serialization
the result is much tidier code and more robust when parsing
optional fields.

fixes #635
2021-05-13 16:04:08 +07:00