Commit Graph

642 Commits

Author SHA1 Message Date
Jordan Hrycaj ee77d704bc
Non-adjacent intervals set management (#1117)
why:
  extracted from snap as separate helper utility

also:
  should go to `stew`, later on
2022-06-07 15:55:42 +01:00
jangko 8b0d700b45
fix EVM stack.[] bug 2022-05-25 12:04:47 +07:00
Jordan Hrycaj 62d31d6f1d
Normalise sync handler prototypes (#1087)
* Use type name eth and snap (rather than snap1)

* Prettified snap/eth handler trace messages

* Regrouped sync sources

details:
  Snap storage related sources are moved to common directory.
  Option --new-sync renamed to --snap-sync

also:
  Normalised logging for secondary/non-protocol handlers.

* Merge protocol wrapper files => protocol.nim

details:
  Merge wrapper sync/protocol_ethxx.nim and sync/protocol_snapxx.nim
  into single file snap/protocol.nim

* Comments cosmetics

* Similar start logic for blockchain_sync.nim and sync/snap.nim

* Renamed p2p/blockchain_sync.nim -> sync/fast.nim
2022-05-13 17:30:10 +01:00
Jordan Hrycaj 3b9f0e3980
Proper Uint256 parser for JSON genesis (#1071)
* Update exception tracinig

* Use lazy JSON parser

why:
  Uint246 type is not directly supported by the JSON serializer

* Json parser update for stringified UInt256 integers

why:
  Now available

* update sub-module branch reference
2022-05-06 09:02:28 +01:00
jangko f2f204293e
first step into styleCheck fixes 2022-04-14 08:39:50 +07:00
Jordan Hrycaj ded38128b5
Jordan/remove unit tests txpool legacy stuff (#1048)
* Update/clarify docu

* Remove legacy stuff for unit tests
2022-04-08 15:05:30 +01:00
Jordan Hrycaj fd6fcc6cc4
Jordan/dissolve txpool jobs queue (#1047)
* Prepare unit tests for running without tx-pool job queue

why:
  Most of the job queue logic can be emulated. This adapts to a few
  pathological test cases.

* Replace tx-pool job queue logic with in-place actions

why:
  This additional execution layer is not needed, anymore which has been
  learned from working with the integration/hive tests.

details:
  Execution of add or deletion jobs are executed in-place. Some actions
  -- as in smartHead() -- have been combined for facilitating the correct
  order actions

* Update production functions, remove txpool legacy stuff
2022-04-08 09:38:47 +01:00
Jordan Hrycaj 8af5c33ef9
Facilitate http code response on websocket JWT authentication failure (#1043)
* Facilitate http code response on websocket JWT authentication failure

* Update JSON-RPC link
2022-04-07 10:37:35 +01:00
Jordan Hrycaj 737236fd6e
Enable JWT authentication for websockets (#1039)
* Enable JWT authentication for websockets

details:
  Currently, this is optional and only enabled when the jwtsecret option
  is set.

  There is a default mechanism to generate a JWT secret if it is not
  explicitly stated. This mechanism is currently unused.

* Make JWT authentication compulsory for websockets

* Fix unit test entry point + cosmetics

* Update JSON-RPC link

* Improvements as suggested by Mamy
2022-04-06 15:11:13 +01:00
Jacek Sieka 1d6a9951d6
move rocksdb support to eth1 (#927)
* move rocksdb support to eth1

only used here / causes unnecessary build deps
2022-04-06 07:28:19 +02:00
jangko 64c2227c7c
json rpc test: fix getStorageAt return type 2022-04-05 16:02:20 +07:00
Jordan Hrycaj 7c603672d3
Comment out unnecessary action in unit test (#1036) 2022-04-05 09:52:24 +01:00
Jordan Hrycaj f11b1a8d8e
Fix typo in tx_head.nim delta calculator
why:
  Causes havoc in most bit fringe cases.

details:
  When setting the head forward, the delta was wrongly registered from
  the static "left" end (which limits the loop) rather than the moving
  "right" end.
2022-04-04 17:07:48 +01:00
Jordan Hrycaj 4ad566a86a
Test bed for chasing issue #1031 2022-04-04 12:42:23 +01:00
jangko 2746f52cc8
txPool: add PoS feeRecipient setter 2022-04-04 09:01:39 +07:00
jangko e04c69df18
txpool: add more test to cover untested features
fix #1029
2022-04-03 15:06:24 +07:00
Jordan Hrycaj 84ff179cd9
Jordan/fix txs sort order (#1025)
* Fix database sort order for local txs

why:
  For convenience, packed txs were stored in the block sorted by
  rank->nonce. Using local accounts, the greedy grabber uses the sort
  order (local,non-local)->rank->nonce which leads to a wrong calculation
  of the txRoot.

* Housekeeping

details:
  Replaced a couple of local eip1559TxNormalization() functions by a
  single public
2022-03-31 17:17:22 +01:00
Jordan Hrycaj 2eb3f414d8
Support for local accounts (#1019)
* Support for local accounts

why:
  Accounts tagged local will be packed with priority over untagged
  accounts

* Added functions for queuing txs and simultaneously setting account locality

why:
  Might be a popular task, in particular for unconditionally adding txs to
  a local (aka prioritised account) via "xp.addLocal(tx,true)"

caveat:
  Untested yet

* fix typo

* backup

* No baseFee for pre-London tx in verifier

why:
  The packer would wrongly discard valid legacy txs.
2022-03-31 09:21:36 +01:00
Jordan Hrycaj a64da329e8
Jordan/ttd debugging helpers (#1013)
* De-noisify some Clique logging

why:
  Too annoying when syncing against Goerly

* Replay devnet# and kiln sessions

why:
  Compiled as local program, the unit test was used for TDD.
2022-03-28 09:35:26 +01:00
Ștefan Talpalaru 51bc1cf87f
dist: precompiled binaries and Docker images (#1015)
* dist: precompiled binaries and Docker images

The builds are reproducible, the binaries are portable and statically link librocksdb.

This took some patching. Upstream PR: https://github.com/facebook/rocksdb/pull/9752

32-bit ARM is missing as a target because two different GCC versions
fail with an ICE when trying to cross-compile RocksDB. Using Clang
instead is too much trouble for a platform that nobody should be using
anyway.

(Clang doesn't come with its own target headers and libraries, can't be
easily convinced to use the ones from GCC, so it needs an fs image from
a 32-bit ARM distro - at which point I stopped caring).

* CI: disable reproducibility test
2022-03-27 13:21:15 +02:00
jangko 32915fde46
move eth_callsigs from test_rpc to rpcclient/eth_api 2022-03-22 08:34:57 +07:00
Jordan Hrycaj 046c97f18b
Activate wire protocol eth/66 (#993)
* Activate wire protocol eth/66

and:
 Disentangle protocol_eth66.nim from import sections

why:
  Importing the protocol_eth66 module is not necessary. There is
  no need to know too many details of the underlying wire protocol. All
  that is needed will be exported by blockchain_sync.nim.

* fixes, and rebase

* Update nimbus/p2p/blockchain_sync.nim

Co-authored-by: Kim De Mey <kim.demey@gmail.com>

* Fixes and rebase

Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2022-03-21 17:12:07 +00:00
jangko a3fe57f2b8
reenable testEngineAPI in test_merge 2022-03-17 12:54:04 +07:00
Jordan Hrycaj ed0e882387
Exported blobs and some scripts to parallel project nimbus-eth1-blobs (#995)
why:
  TDD data and test script that are not needed for CI are externally held.
  This saves space.

also:
  Added support for test-custom_networks.nim to run import Devnet5 dump.
2022-03-16 09:13:17 +00:00
Kim De Mey b00ac490a9
Bump nim-eth and adjust API to support binding to specific address (#994)
- Fixes also a bug which would not start an EthereumNode properly
when no bootstrap nodes are provided
2022-03-16 09:01:35 +01:00
jangko a4b70918ed
temporay disable test_merge 2022-03-11 16:25:23 +07:00
jangko f782327fcf
reimplement engine API rpc kiln spec v2 2022-03-03 11:28:18 +07:00
jangko 667cb6d750
EIP 4399 changes: random -> prevRandao 2022-02-27 14:37:35 +07:00
Jordan Hrycaj 215e9856d3
Jordan/custom network (#962)
* Rearrange/rename test_kintsugu => test_custom_network

why:
  Debug, fix and test more general problems related to running
  nimbus on a custom network.

* Update UInt265/Json parser for --custom-network command line option

why:
  As found out with the Kintsugi configuration, block number and balance
  have the same Nim type which led to misunderstandings. This patch makes
  sure that UInt265 encoded string values "0x11" decodes to 17, and "b"
  and "11" to 11.

* Refactored genesis.toBlock() => genesis.toBlockHeader()

why:
  The function toBlock(g,db) may return different results depending on
  whether the db descriptor argument is nil, or initialised. This is due
  to the db.config data sub-descriptor which may give various outcomes
  for the baseFee field of the genesis header.

  Also, the version where db is non-nil initialised is used internally
  only. So the public rewrite toBlockHeader() that replaces the toBlock()
  function expects a full set of NetworkParams.

* update comments

* Rename toBlockHeader() => toGenesisHeader()

why:
  Polymorphic prototype used for BaseChainDB or NetworkParams argument.
  With a BaseChainDB descriptor argument, the name shall imply that the
  header is generated from the config fields rather than fetched from
  the database.

* Added command line option --static-peers-file

why:
  Handy feature to keep peer nodes in a file, similar to the
  --bootstrap-file option.
2022-02-11 16:28:39 +00:00
jangko 231487197f
fix test_helper: able to parse overflow tx value 2022-02-10 15:48:38 +07:00
jangko fcad04d9ff
bump ethereum tests
- including test for EIP-3607
- tx value overflow
- tx nonce overflow during CREATE
2022-02-10 15:48:26 +07:00
jangko 28cdfcaf6b
fix EIP-4399 'random' opcode
- fix previous implementation of EIP-4399
- now `random` opcode can be used with evmc_enabled
2022-02-08 20:23:40 +07:00
jangko 5e5afaeacc
EIP-4399 'RANDOM' opcode tests 2022-02-05 16:17:41 +07:00
Jordan Hrycaj 77c9b8c2f0
Kludge needed for setting up custom network (#944)
* Kludge needed for setting up custom network

why:
  Some non-features in the persistent hexary trie DB produce an assert
  error when initiating the Kinsugi network.

details:
  This fix should be temporary, only.

* Fix OS detection

why:
  directive detectOs() bails out on Windows if checking for Ubuntu
2022-02-01 12:04:20 +00:00
Jordan Hrycaj bced43c339
test environment for studying crash of hexary trie (#938)
* test environment for studying crash of hexary trie

why:
  the persistent test case will crash unless in genesis.toBlock():
  + pruneTrie is set false, or
  + the directive "tdb.put(emptyRlpHash.data,emptyRlp)" is added right
    before the "for k, v in account.storage:" loop

* different tests for OS variants
2022-01-26 11:56:53 +00:00
Zahary Karadjov 3cbb920406
Migrate to Engine API spec version v1.0.0-alpha.5; More progress towards working M1 2022-01-25 12:52:50 +02:00
jangko 01980041d5
setup nimbus tx pool for app wide usage
currently the txpool is used at:
- sealer engine
- graphql ethapi
- json rpc ethapi
- ws rpc ethapi
2022-01-25 11:38:49 +07:00
Zahary Karadjov 137eb97766 Initial implementation of the merge spec
Includes a simple test harness for the merge interop M1 milestone

This aims to enable connecting nimbus-eth2 to nimbus-eth1 within
the testing protocol described here:

https://github.com/status-im/nimbus-eth2/blob/amphora-merge-interop/docs/interop_merge.md

To execute the work-in-progress test, please run:

In terminal 1:
tests/amphora/launch-nimbus.sh

In terminal 2:
tests/amphora/check-merge-test-vectors.sh
2022-01-24 09:44:39 +02:00
Jordan Hrycaj 103656dbb5 TxPool implementation
details:
  For documentation, see comments in the file tx_pool.nim.

  For prettified manual pages run 'make docs' in the nimbus directory and
  point your web browser to the newly created 'docs' directory.
2022-01-22 08:26:57 +02:00
Jordan Hrycaj 9545767c53 update ec_recover
why:
  Previous version was based on lru_cache which is ugly. This module is
  based on the stew/keyed_queue library module.

other:
  There are still some other modules rely on lru_cache which should be
  removed.
2022-01-22 08:26:57 +02:00
Jordan Hrycaj 261c0b51a7
Redesign of BaseVMState descriptor (#923)
* Redesign of BaseVMState descriptor

why:
  BaseVMState provides an environment for executing transactions. The
  current descriptor also provides data that cannot generally be known
  within the execution environment, e.g. the total gasUsed which is
  available not before after all transactions have finished.

  Also, the BaseVMState constructor has been replaced by a constructor
  that does not need pre-initialised input of the account database.

also:
  Previous constructor and some fields are provided with a deprecated
  annotation (producing a lot of noise.)

* Replace legacy directives in production sources

* Replace legacy directives in unit test sources

* fix CI (missing premix update)

* Remove legacy directives

* chase CI problem

* rebased

* Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation

why:
  Constructing a new 'AccountsCache' descriptor can be avoided sometimes
  when the current state root is properly positioned already. Such a
  feature existed already as the update function 'initStateDB()' for the
  'BaseChanDB' where the accounts cache was linked into this desctiptor.

  The function 'initStateDB()' was removed and re-implemented into the
  'BaseVmState' constructor without optimisation. The old version was of
  restricted use as a wrong accounts cache state would unconditionally
  throw an exception rather than conceptually ask for a remedy.

  The optimised 'BaseVmState' re-initialisation has been implemented for
  the 'persistBlocks()' function.

also:
  moved some test helpers to 'test/replay' folder

* Remove unused & undocumented fields from Chain descriptor

why:
  Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
Jordan Hrycaj cea2a83b0a
Update/clarify tx validation (#917)
details:
  1. The check for cumulativeGasUsed + tx.gasLimit <= header.gasLimit
     makes neither sense nor is it part of the Eip1559 specs. Nevertheless
     a check tx.gasLimit <= header.gasLimit is added to satisfy some
     unit test (see comments in validateTransaction() body.)
  2. As a replacement check for the one removed in 1, a check for
     cumulativeGasUsed + gasBurned <= header.gasLimit has been added
     (see comments in processTransactionImpl() body.)
  3. Prototypes for processTransaction() variants have been cleaned up and
     commented.

why:
  Detail 1. in particular produces an error for tightly packed blocks when
  the last tx in the list has a generous gasLimit.
2022-01-10 09:04:06 +00:00
Jordan Hrycaj 1f774c01a2
Jordan/accounts cache scenario (#904)
* crash test scenario

details:
  Example code for inspecting nested block chain and accounts cache
  database transaction framework. There seems to be a pathological
  case where the system crashes after a rollback (as appeared in the
  tx-pool packer code.)

* simplified crash scenario

* Workable solution (as suggested by Andri)

details:
  Avoiding db.rollback() (db.commit() is OK) while vmState.stateDB is
  alive.

* Rename text_txcrash => test_accounts_cache

why:
  Unit tests covers part of accounts_cache handling

* comment update
2021-12-13 11:58:05 +00:00
Jamie Lokier 74253c88e3
Arrow Glacier fork
Add the new [Arrow Glacier fork](https://eips.ethereum.org/EIPS/eip-4345).

Only the difficulty calculation is changed, but as a new fork it still affects
a number of places in the code.

To the best of my knowledge the change is only scheduled on Mainnet.

In addition:

- The fork date comments in `chain_config.nim` have been checked against the
  real networks, set consistently in UTC instead of random timezones, and made
  neater.  Maybe we'll keep these when transferring config to a file someday.

- It's added to forkid hash tests (EIP-2124/EIP-2364), of course.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-12-10 13:40:51 +00:00
Jordan Hrycaj 55f7a4425f
Jordan/pow cache management (#888)
* PoW wrapper for verification & mining

why:
  It eases data management of per-Epoch lookup tables. Also some unit
  tests show limits of usefulness on non-specialised machines for
  mining besides developing tests.

details:
  For PoW verification, this patch provides a pretty wrapper hiding the
  details of the ethash/Hashimoto lookup cache management.

  For mining on my development system without special hardware, the
  underlying ethash functions are prohibitively slow. It takes
   * ~20 minutes to prepare the full ethash/Hashimoto lookup dataset
   * a second to run ~25k nonce tests (in the mining loop)

  The mining part might be of some use for generating test data for
  the tx-pool, though.

* Using PowRef as replacement for EpochHashCache + hashimotoLight()

* Fix typo (CI failed)

why:
  was below log level when testing locally

* fix canonical naming
2021-12-10 08:49:57 +00:00
jangko 960539df81
unify evm call for both json-rpc and graphql
also fixes rpcEstimateGas for both of json-rpc and graphql
2021-10-31 12:24:40 +07:00
jangko baf508f6ae
move stateDB from VMState to chainDB
previously, every time the VMState was created, it will also create
new stateDB, and this action will nullify the advantages of cached accounts.

the new changes will conserve the accounts cache if the executed blocks
are contiguous. if not the stateDB need to be reinited.

this changes also allow rpcCallEvm and rpcEstimateGas executed properly
using current stateDB instead of creating new one each time they are called.
2021-10-28 18:57:08 +07:00
jangko d93a8bc4a1
make macro_assembler to use the same testEvmCall
it also allow the macro_assembler to write more test
of CALL and CREATE family which previously not possible
2021-10-14 15:10:12 +07:00
jangko eb2251ec37
simplify evm call of test_precompiles
first step towards evm call variation reduction
2021-10-14 15:10:11 +07:00
jangko d8dedb0469
bump ethereum/tests fixtures 2021-09-29 10:55:31 +07:00