Commit Graph

855 Commits

Author SHA1 Message Date
jangko a5d4759bfd
enhance net-key command line option to accept random, hex, and path
- previously it only accept hex
- fix #587
2022-07-30 08:46:11 +07:00
jangko a087d65542
reduce test suite time consumption 2022-07-30 08:43:15 +07:00
Jordan Hrycaj 73b628491d
Clique snapshots reorg (#1169)
* Add persistent snapshot size logging

why:
  Suspecting too much space used

snapshot statistic:
  [..]
  blockNumber=2214912 nSnaps=2236 snapsTotal=1.14m
  blockNumber=2215936 nSnaps=2237 snapsTotal=1.14m
  [..]
  Persisting blocks fromBlock=2216449 toBlock=2216640
  36458496	datadir-nimbus-goerlish/data/nimbus/

* Replace legacy `lru_cache` by `keyed_queue`

why:
  `keyed_queue` generalises `lru_cache`

snapshot statistic:
  [..]
  blockNumber=2234368 nSnaps=2259 snapsTotal=1.15m
  blockNumber=2235392 nSnaps=2260 snapsTotal=1.15m
  [..]
  Persisting blocks fromBlock=2235649 toBlock=2235840
  37627288	datadir-nimbus-goerlish/data/nimbus/

* Increase persistent snapshot storage interval by 300%

snapshot statistic:
      [..]
      blockNumber=2232320 nSnaps=620 snapsTotal=0.30m
      blockNumber=2236416 nSnaps=621 snapsTotal=0.30m
      [..]
      Persisting blocks fromBlock=2237185 toBlock=2237376
      37627288	datadir-nimbus-goerlish/data/nimbus/

* Cull legacy debugging environment for clique

why:
  Chronicles provides a better choice (when properly set up)
2022-07-21 19:16:28 +01:00
jangko e6938af437
apply jwt auth to rpcHttpServer and update jwt auth of rpcWebsocketServer
fixes #967
2022-07-18 16:56:44 +07:00
jangko 6cfaaf5b45
bump json-rpc, secp256k1, blscurve, and accompanying fixes 2022-07-12 20:01:02 +07:00
jangko 2b4baff8ec
move block validation from execution payload generator to engine api 2022-07-04 20:36:30 +07:00
jangko 709d8ef255
fix markCanonicalChain bug 2022-07-04 19:32:14 +07:00
Jordan Hrycaj 7293a54e58
Added sepolia specs (#1148)
* Added sepolia specs

* temporarily avoid latest `master` branch from `nim-eth1`

why:
  Currently does not cleanly compile after the `bearssl` split api update.
2022-07-01 21:16:26 +01:00
Jordan Hrycaj 134fe26997
Store proved snap accounts (#1145)
* Relocated `IntervalSets` to nim-stew repo

* Accumulate accounts on temporary kv-DB

why:
  Explore the data as returned from snap/1. Will be converted to a
  `eth/db` next.

details:
  Verify and accumulate per/state-root accounts downloaded via snap.

also:
  Some unit tests

* Replace `Table` by `TrieDatabaseRef` for accounts accumulator

* update ticker statistics

details:
  mean/variance based counter update

* allow persistent db for proved accounts

* rebase, and globally activate unit test

* fix statistics
2022-07-01 12:42:17 +01:00
KonradStaniec fdb048ed21
Add getLogs implementation (#1143)
* Add eth_getLogs json-rpc endpoint in Fluffy and Nimbus
2022-06-29 17:44:08 +02:00
jangko 4a50b00c37
rpc: change getStorageAt param type to HexDataStr
if using HexQuantityStr, it will fail when given hex string with leading zeros
2022-06-27 17:47:59 +07:00
jangko 00a43234d7
fix enable rpc logic in makeConfig 2022-06-17 07:54:13 +07:00
Jordan Hrycaj c123e1eb93
Updated account scheduler (#1124)
* Using `IntervalSet` type data for `LeafRange`

* Updated log ticker

* Update to `eth67`

details:
  Disabled by default, use `ENABLE_LEGACY_ETH66=0` to enable
  No support for `Get/NodeData` dialogue via eth, anymore

* Dissolved fetch/common.nim

details;
  the log/ticker part becomes ticker.nim
  the interval range management is merged into fetch.nim

* Updated account scheduler

why:
  The previous scheduler fetched each account once (for different state
  roots.) The updated scheduler re-calibrates after a change of the state
  root and potentially (until told otherwise) fetches all possible
  accounts.

* Fix `high(P)` fringe cases in `IntervalSet` handling

why:
  The `high(P)` value for a point type `P` cannot be represented with
  half open intervals `[a,b)` for a,b points of `P`. So this single value
  needs extra treatment which was slightly wrong.

* Updated docu/comments

also:
  rebased

* Update scheduler

details:
  Change the `pivot` management when creating new accounts lists. It is
  strictly increasing (and wrapping around) depending on last updated
  accounts list.
2022-06-16 09:58:50 +01:00
Jordan Hrycaj ee77d704bc
Non-adjacent intervals set management (#1117)
why:
  extracted from snap as separate helper utility

also:
  should go to `stew`, later on
2022-06-07 15:55:42 +01:00
jangko 8b0d700b45
fix EVM stack.[] bug 2022-05-25 12:04:47 +07:00
Jordan Hrycaj 62d31d6f1d
Normalise sync handler prototypes (#1087)
* Use type name eth and snap (rather than snap1)

* Prettified snap/eth handler trace messages

* Regrouped sync sources

details:
  Snap storage related sources are moved to common directory.
  Option --new-sync renamed to --snap-sync

also:
  Normalised logging for secondary/non-protocol handlers.

* Merge protocol wrapper files => protocol.nim

details:
  Merge wrapper sync/protocol_ethxx.nim and sync/protocol_snapxx.nim
  into single file snap/protocol.nim

* Comments cosmetics

* Similar start logic for blockchain_sync.nim and sync/snap.nim

* Renamed p2p/blockchain_sync.nim -> sync/fast.nim
2022-05-13 17:30:10 +01:00
Jordan Hrycaj 3b9f0e3980
Proper Uint256 parser for JSON genesis (#1071)
* Update exception tracinig

* Use lazy JSON parser

why:
  Uint246 type is not directly supported by the JSON serializer

* Json parser update for stringified UInt256 integers

why:
  Now available

* update sub-module branch reference
2022-05-06 09:02:28 +01:00
jangko f2f204293e
first step into styleCheck fixes 2022-04-14 08:39:50 +07:00
Jordan Hrycaj ded38128b5
Jordan/remove unit tests txpool legacy stuff (#1048)
* Update/clarify docu

* Remove legacy stuff for unit tests
2022-04-08 15:05:30 +01:00
Jordan Hrycaj fd6fcc6cc4
Jordan/dissolve txpool jobs queue (#1047)
* Prepare unit tests for running without tx-pool job queue

why:
  Most of the job queue logic can be emulated. This adapts to a few
  pathological test cases.

* Replace tx-pool job queue logic with in-place actions

why:
  This additional execution layer is not needed, anymore which has been
  learned from working with the integration/hive tests.

details:
  Execution of add or deletion jobs are executed in-place. Some actions
  -- as in smartHead() -- have been combined for facilitating the correct
  order actions

* Update production functions, remove txpool legacy stuff
2022-04-08 09:38:47 +01:00
Jordan Hrycaj 8af5c33ef9
Facilitate http code response on websocket JWT authentication failure (#1043)
* Facilitate http code response on websocket JWT authentication failure

* Update JSON-RPC link
2022-04-07 10:37:35 +01:00
Jordan Hrycaj 737236fd6e
Enable JWT authentication for websockets (#1039)
* Enable JWT authentication for websockets

details:
  Currently, this is optional and only enabled when the jwtsecret option
  is set.

  There is a default mechanism to generate a JWT secret if it is not
  explicitly stated. This mechanism is currently unused.

* Make JWT authentication compulsory for websockets

* Fix unit test entry point + cosmetics

* Update JSON-RPC link

* Improvements as suggested by Mamy
2022-04-06 15:11:13 +01:00
Jacek Sieka 1d6a9951d6
move rocksdb support to eth1 (#927)
* move rocksdb support to eth1

only used here / causes unnecessary build deps
2022-04-06 07:28:19 +02:00
jangko 64c2227c7c
json rpc test: fix getStorageAt return type 2022-04-05 16:02:20 +07:00
Jordan Hrycaj 7c603672d3
Comment out unnecessary action in unit test (#1036) 2022-04-05 09:52:24 +01:00
Jordan Hrycaj f11b1a8d8e
Fix typo in tx_head.nim delta calculator
why:
  Causes havoc in most bit fringe cases.

details:
  When setting the head forward, the delta was wrongly registered from
  the static "left" end (which limits the loop) rather than the moving
  "right" end.
2022-04-04 17:07:48 +01:00
Jordan Hrycaj 4ad566a86a
Test bed for chasing issue #1031 2022-04-04 12:42:23 +01:00
jangko 2746f52cc8
txPool: add PoS feeRecipient setter 2022-04-04 09:01:39 +07:00
jangko e04c69df18
txpool: add more test to cover untested features
fix #1029
2022-04-03 15:06:24 +07:00
Jordan Hrycaj 84ff179cd9
Jordan/fix txs sort order (#1025)
* Fix database sort order for local txs

why:
  For convenience, packed txs were stored in the block sorted by
  rank->nonce. Using local accounts, the greedy grabber uses the sort
  order (local,non-local)->rank->nonce which leads to a wrong calculation
  of the txRoot.

* Housekeeping

details:
  Replaced a couple of local eip1559TxNormalization() functions by a
  single public
2022-03-31 17:17:22 +01:00
Jordan Hrycaj 2eb3f414d8
Support for local accounts (#1019)
* Support for local accounts

why:
  Accounts tagged local will be packed with priority over untagged
  accounts

* Added functions for queuing txs and simultaneously setting account locality

why:
  Might be a popular task, in particular for unconditionally adding txs to
  a local (aka prioritised account) via "xp.addLocal(tx,true)"

caveat:
  Untested yet

* fix typo

* backup

* No baseFee for pre-London tx in verifier

why:
  The packer would wrongly discard valid legacy txs.
2022-03-31 09:21:36 +01:00
Jordan Hrycaj a64da329e8
Jordan/ttd debugging helpers (#1013)
* De-noisify some Clique logging

why:
  Too annoying when syncing against Goerly

* Replay devnet# and kiln sessions

why:
  Compiled as local program, the unit test was used for TDD.
2022-03-28 09:35:26 +01:00
Ștefan Talpalaru 51bc1cf87f
dist: precompiled binaries and Docker images (#1015)
* dist: precompiled binaries and Docker images

The builds are reproducible, the binaries are portable and statically link librocksdb.

This took some patching. Upstream PR: https://github.com/facebook/rocksdb/pull/9752

32-bit ARM is missing as a target because two different GCC versions
fail with an ICE when trying to cross-compile RocksDB. Using Clang
instead is too much trouble for a platform that nobody should be using
anyway.

(Clang doesn't come with its own target headers and libraries, can't be
easily convinced to use the ones from GCC, so it needs an fs image from
a 32-bit ARM distro - at which point I stopped caring).

* CI: disable reproducibility test
2022-03-27 13:21:15 +02:00
jangko 32915fde46
move eth_callsigs from test_rpc to rpcclient/eth_api 2022-03-22 08:34:57 +07:00
Jordan Hrycaj 046c97f18b
Activate wire protocol eth/66 (#993)
* Activate wire protocol eth/66

and:
 Disentangle protocol_eth66.nim from import sections

why:
  Importing the protocol_eth66 module is not necessary. There is
  no need to know too many details of the underlying wire protocol. All
  that is needed will be exported by blockchain_sync.nim.

* fixes, and rebase

* Update nimbus/p2p/blockchain_sync.nim

Co-authored-by: Kim De Mey <kim.demey@gmail.com>

* Fixes and rebase

Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2022-03-21 17:12:07 +00:00
jangko a3fe57f2b8
reenable testEngineAPI in test_merge 2022-03-17 12:54:04 +07:00
Jordan Hrycaj ed0e882387
Exported blobs and some scripts to parallel project nimbus-eth1-blobs (#995)
why:
  TDD data and test script that are not needed for CI are externally held.
  This saves space.

also:
  Added support for test-custom_networks.nim to run import Devnet5 dump.
2022-03-16 09:13:17 +00:00
Kim De Mey b00ac490a9
Bump nim-eth and adjust API to support binding to specific address (#994)
- Fixes also a bug which would not start an EthereumNode properly
when no bootstrap nodes are provided
2022-03-16 09:01:35 +01:00
jangko a4b70918ed
temporay disable test_merge 2022-03-11 16:25:23 +07:00
jangko f782327fcf
reimplement engine API rpc kiln spec v2 2022-03-03 11:28:18 +07:00
jangko 667cb6d750
EIP 4399 changes: random -> prevRandao 2022-02-27 14:37:35 +07:00
Jordan Hrycaj 215e9856d3
Jordan/custom network (#962)
* Rearrange/rename test_kintsugu => test_custom_network

why:
  Debug, fix and test more general problems related to running
  nimbus on a custom network.

* Update UInt265/Json parser for --custom-network command line option

why:
  As found out with the Kintsugi configuration, block number and balance
  have the same Nim type which led to misunderstandings. This patch makes
  sure that UInt265 encoded string values "0x11" decodes to 17, and "b"
  and "11" to 11.

* Refactored genesis.toBlock() => genesis.toBlockHeader()

why:
  The function toBlock(g,db) may return different results depending on
  whether the db descriptor argument is nil, or initialised. This is due
  to the db.config data sub-descriptor which may give various outcomes
  for the baseFee field of the genesis header.

  Also, the version where db is non-nil initialised is used internally
  only. So the public rewrite toBlockHeader() that replaces the toBlock()
  function expects a full set of NetworkParams.

* update comments

* Rename toBlockHeader() => toGenesisHeader()

why:
  Polymorphic prototype used for BaseChainDB or NetworkParams argument.
  With a BaseChainDB descriptor argument, the name shall imply that the
  header is generated from the config fields rather than fetched from
  the database.

* Added command line option --static-peers-file

why:
  Handy feature to keep peer nodes in a file, similar to the
  --bootstrap-file option.
2022-02-11 16:28:39 +00:00
jangko 231487197f
fix test_helper: able to parse overflow tx value 2022-02-10 15:48:38 +07:00
jangko fcad04d9ff
bump ethereum tests
- including test for EIP-3607
- tx value overflow
- tx nonce overflow during CREATE
2022-02-10 15:48:26 +07:00
jangko 28cdfcaf6b
fix EIP-4399 'random' opcode
- fix previous implementation of EIP-4399
- now `random` opcode can be used with evmc_enabled
2022-02-08 20:23:40 +07:00
jangko 5e5afaeacc
EIP-4399 'RANDOM' opcode tests 2022-02-05 16:17:41 +07:00
Jordan Hrycaj 77c9b8c2f0
Kludge needed for setting up custom network (#944)
* Kludge needed for setting up custom network

why:
  Some non-features in the persistent hexary trie DB produce an assert
  error when initiating the Kinsugi network.

details:
  This fix should be temporary, only.

* Fix OS detection

why:
  directive detectOs() bails out on Windows if checking for Ubuntu
2022-02-01 12:04:20 +00:00
Jordan Hrycaj bced43c339
test environment for studying crash of hexary trie (#938)
* test environment for studying crash of hexary trie

why:
  the persistent test case will crash unless in genesis.toBlock():
  + pruneTrie is set false, or
  + the directive "tdb.put(emptyRlpHash.data,emptyRlp)" is added right
    before the "for k, v in account.storage:" loop

* different tests for OS variants
2022-01-26 11:56:53 +00:00
Zahary Karadjov 3cbb920406
Migrate to Engine API spec version v1.0.0-alpha.5; More progress towards working M1 2022-01-25 12:52:50 +02:00
jangko 01980041d5
setup nimbus tx pool for app wide usage
currently the txpool is used at:
- sealer engine
- graphql ethapi
- json rpc ethapi
- ws rpc ethapi
2022-01-25 11:38:49 +07:00
Zahary Karadjov 137eb97766 Initial implementation of the merge spec
Includes a simple test harness for the merge interop M1 milestone

This aims to enable connecting nimbus-eth2 to nimbus-eth1 within
the testing protocol described here:

https://github.com/status-im/nimbus-eth2/blob/amphora-merge-interop/docs/interop_merge.md

To execute the work-in-progress test, please run:

In terminal 1:
tests/amphora/launch-nimbus.sh

In terminal 2:
tests/amphora/check-merge-test-vectors.sh
2022-01-24 09:44:39 +02:00
Jordan Hrycaj 103656dbb5 TxPool implementation
details:
  For documentation, see comments in the file tx_pool.nim.

  For prettified manual pages run 'make docs' in the nimbus directory and
  point your web browser to the newly created 'docs' directory.
2022-01-22 08:26:57 +02:00
Jordan Hrycaj 9545767c53 update ec_recover
why:
  Previous version was based on lru_cache which is ugly. This module is
  based on the stew/keyed_queue library module.

other:
  There are still some other modules rely on lru_cache which should be
  removed.
2022-01-22 08:26:57 +02:00
Jordan Hrycaj 261c0b51a7
Redesign of BaseVMState descriptor (#923)
* Redesign of BaseVMState descriptor

why:
  BaseVMState provides an environment for executing transactions. The
  current descriptor also provides data that cannot generally be known
  within the execution environment, e.g. the total gasUsed which is
  available not before after all transactions have finished.

  Also, the BaseVMState constructor has been replaced by a constructor
  that does not need pre-initialised input of the account database.

also:
  Previous constructor and some fields are provided with a deprecated
  annotation (producing a lot of noise.)

* Replace legacy directives in production sources

* Replace legacy directives in unit test sources

* fix CI (missing premix update)

* Remove legacy directives

* chase CI problem

* rebased

* Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation

why:
  Constructing a new 'AccountsCache' descriptor can be avoided sometimes
  when the current state root is properly positioned already. Such a
  feature existed already as the update function 'initStateDB()' for the
  'BaseChanDB' where the accounts cache was linked into this desctiptor.

  The function 'initStateDB()' was removed and re-implemented into the
  'BaseVmState' constructor without optimisation. The old version was of
  restricted use as a wrong accounts cache state would unconditionally
  throw an exception rather than conceptually ask for a remedy.

  The optimised 'BaseVmState' re-initialisation has been implemented for
  the 'persistBlocks()' function.

also:
  moved some test helpers to 'test/replay' folder

* Remove unused & undocumented fields from Chain descriptor

why:
  Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
Jordan Hrycaj cea2a83b0a
Update/clarify tx validation (#917)
details:
  1. The check for cumulativeGasUsed + tx.gasLimit <= header.gasLimit
     makes neither sense nor is it part of the Eip1559 specs. Nevertheless
     a check tx.gasLimit <= header.gasLimit is added to satisfy some
     unit test (see comments in validateTransaction() body.)
  2. As a replacement check for the one removed in 1, a check for
     cumulativeGasUsed + gasBurned <= header.gasLimit has been added
     (see comments in processTransactionImpl() body.)
  3. Prototypes for processTransaction() variants have been cleaned up and
     commented.

why:
  Detail 1. in particular produces an error for tightly packed blocks when
  the last tx in the list has a generous gasLimit.
2022-01-10 09:04:06 +00:00
Jordan Hrycaj 1f774c01a2
Jordan/accounts cache scenario (#904)
* crash test scenario

details:
  Example code for inspecting nested block chain and accounts cache
  database transaction framework. There seems to be a pathological
  case where the system crashes after a rollback (as appeared in the
  tx-pool packer code.)

* simplified crash scenario

* Workable solution (as suggested by Andri)

details:
  Avoiding db.rollback() (db.commit() is OK) while vmState.stateDB is
  alive.

* Rename text_txcrash => test_accounts_cache

why:
  Unit tests covers part of accounts_cache handling

* comment update
2021-12-13 11:58:05 +00:00
Jamie Lokier 74253c88e3
Arrow Glacier fork
Add the new [Arrow Glacier fork](https://eips.ethereum.org/EIPS/eip-4345).

Only the difficulty calculation is changed, but as a new fork it still affects
a number of places in the code.

To the best of my knowledge the change is only scheduled on Mainnet.

In addition:

- The fork date comments in `chain_config.nim` have been checked against the
  real networks, set consistently in UTC instead of random timezones, and made
  neater.  Maybe we'll keep these when transferring config to a file someday.

- It's added to forkid hash tests (EIP-2124/EIP-2364), of course.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-12-10 13:40:51 +00:00
Jordan Hrycaj 55f7a4425f
Jordan/pow cache management (#888)
* PoW wrapper for verification & mining

why:
  It eases data management of per-Epoch lookup tables. Also some unit
  tests show limits of usefulness on non-specialised machines for
  mining besides developing tests.

details:
  For PoW verification, this patch provides a pretty wrapper hiding the
  details of the ethash/Hashimoto lookup cache management.

  For mining on my development system without special hardware, the
  underlying ethash functions are prohibitively slow. It takes
   * ~20 minutes to prepare the full ethash/Hashimoto lookup dataset
   * a second to run ~25k nonce tests (in the mining loop)

  The mining part might be of some use for generating test data for
  the tx-pool, though.

* Using PowRef as replacement for EpochHashCache + hashimotoLight()

* Fix typo (CI failed)

why:
  was below log level when testing locally

* fix canonical naming
2021-12-10 08:49:57 +00:00
jangko 960539df81
unify evm call for both json-rpc and graphql
also fixes rpcEstimateGas for both of json-rpc and graphql
2021-10-31 12:24:40 +07:00
jangko baf508f6ae
move stateDB from VMState to chainDB
previously, every time the VMState was created, it will also create
new stateDB, and this action will nullify the advantages of cached accounts.

the new changes will conserve the accounts cache if the executed blocks
are contiguous. if not the stateDB need to be reinited.

this changes also allow rpcCallEvm and rpcEstimateGas executed properly
using current stateDB instead of creating new one each time they are called.
2021-10-28 18:57:08 +07:00
jangko d93a8bc4a1
make macro_assembler to use the same testEvmCall
it also allow the macro_assembler to write more test
of CALL and CREATE family which previously not possible
2021-10-14 15:10:12 +07:00
jangko eb2251ec37
simplify evm call of test_precompiles
first step towards evm call variation reduction
2021-10-14 15:10:11 +07:00
jangko d8dedb0469
bump ethereum/tests fixtures 2021-09-29 10:55:31 +07:00
jangko 908dc21478
evm: fixes EIP2929 opcodes
op balanceEIP2929, extCodeHashEIP2929, extCodeSizeEIP2929, and
extCodeCopyEIP2929 are fixed due to their wrong gasConsume
position
2021-09-22 11:58:06 +07:00
jangko f3d194c05e
grpahql: add EIP-2718 and EIP-1559 features to graphql API
EIP-2718:
- chainID: Long! of Query
- chainID: Long of Transaction

EIP-1559:
- baseFeePerGas: BigInt of Block
- effectiveGasPrice: BigInt of Transaction
- maxFeePerGas: BigInt of Transaction
- maxPriorityFeePerGas: BigInt of Transaction
2021-09-21 13:35:49 +07:00
jangko a3badea928
config: fix new config based on input from jamie and zahary 2021-09-18 17:34:51 +07:00
jangko 69f2a0f95a
config: replace stdlib parseOpt with nim-confutils
fixes #581
2021-09-18 17:34:46 +07:00
jangko 48d497580a
config: remove last instance of getConfiguration usage from nimbus code
this is a preparation for migration to confutils based config
although there is still some getConfiguration usage in tests code
it will be removed after new config arrived
2021-09-08 21:25:14 +07:00
jangko c9cfebfa97
config: rearrange getConfiguration usage
avoid using getConfiguration inside object construction and
replace it with passing suitable param
2021-09-08 08:07:10 +07:00
jangko 9108301eef
config: remove global rng from NimbusConfiguration
move the rng to EthContext
2021-09-07 22:02:29 +07:00
jangko 34972c6cea
config: remove accounts management from NimbusConfiguration
a new AccountsManager and EthContext is created for managing
keystore and accounts

this is a preparation for new config using ConfUtils
2021-09-07 22:02:29 +07:00
jangko 14d2edcb26
chain config preset: add london block number
MainNet     12_965_000
RopstenNet  10_499_401
RinkebyNet   8_897_988
GoerliNet    5_062_605
2021-09-02 12:24:04 +07:00
jangko 521f29c0a0
remove unused calcGasLimit code
we have new calcGasLimit tested in sealing engine.
so we can safely remove the old unused calcGasLimit
2021-08-24 18:30:52 +07:00
jangko c99153df22
fixes clique signerFn return type
and also add test related to this signerFn
2021-08-19 19:00:30 +07:00
jangko 18b26a0089
implement calcEIP1559GasLimit
CalcGasLimit1559 calculates the next block gas limit under 1559 rules.
this function is needed in upcoming sealing engine implementation
2021-08-18 20:23:38 +07:00
jangko 5e87624315
config: copy chainId to networkid if networkid not set in cli
although they are technically different, but in reality,
many networks are using the same id for ChainId dan NetworkId.
in this commit, we set networkid from config file's chainId.
2021-08-06 07:31:02 +07:00
jangko 4459fc3519
add more networkid cli test 2021-08-06 07:31:01 +07:00
jangko 1da4346295
config: fixes bug networkid parser
previously it mistakenly parse into the `result`
now it correctly parse networkId into res.
2021-08-06 07:31:01 +07:00
jangko c69e57e5d4
config: fix config file parser
- allow clique period and epoch to be configured via config file
- this also activate poaEngine mode
- remove clique period configuration from cli to reduce confusion
- fix #786
2021-08-06 07:30:53 +07:00
Jordan Hrycaj 4713bd4cf4
#768 Moved/re-implemented ecRecover() from Clique sources to utils/ec_recover
why:
  The same functionality was differently implemented in one or the
  other form.

details:
  Caching and non-caching variants available
2021-08-05 12:27:10 +07:00
Jamie Lokier 9aea669363
Tests: Add loopMul.json and loopExp.json to skipped, slow tests
Add these to the list of specially disabled tests during `make test`,
as they are exceptionally slow.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-08-03 15:03:05 +01:00
Jordan Hrycaj ec9354d2d0
Jordan/poa voting header (#782)
* Provide PoA voting header generator

why:
  Handy for hive/smoke test

details:
  Header generator is a re-implementation of the generator previously
  used for the canonical reference tests.

* try fixing ci out-of-mem condition

why:
  for some reason, the ci began behaving like a real win7/i386 machine
  where gcc is limited to 64k optimiser space

* fix comments, typos ..
2021-08-03 08:15:32 +01:00
Jordan Hrycaj ca07c40a48
Fearture/poa clique tuning (#765)
* Provide API

details:
  API is bundled via clique.nim.

* Set extraValidation as default for PoA chains

why:
  This triggers consensus verification and an update of the list
  of authorised signers. These signers are integral part of the
  PoA block chain.

todo:
  Option argument to control validation for the nimbus binary.

* Fix snapshot state block number

why:
  Using sub-sequence here, so the len() function was wrong.

* Optional start where block verification begins

why:
  Can speed up time building loading initial parts of block chain. For
  PoA, this allows to prove & test that authorised signers can be
  (correctly) calculated starting at any point on the block chain.

todo:
  On Goerli around blocks #193537..#197568, processing time increases
  disproportionally -- needs to be understand

* For Clique test, get old grouping back (7 transactions per log entry)

why:
  Forgot to change back after troubleshooting

* Fix field/function/module-name misunderstanding

why:
  Make compilation work

* Use eth_types.blockHash() rather than utils.hash() in Clique modules

why:
  Prefer lib module

* Dissolve snapshot_misc.nim

details:
  .. into clique_verify.nim (the other source file clique_unused.nim
  is inactive)

* Hide unused AsyncLock in Clique descriptor

details:
  Unused here but was part of the Go reference implementation

* Remove fakeDiff flag from Clique descriptor

details:
  This flag was a kludge in the Go reference implementation used for the
  canonical tests. The tests have been adapted so there is no need for
  the fakeDiff flag and its implementation.

* Not observing minimum distance from epoch sync point

why:
  For compiling PoA state, the go implementation will walk back to the
  epoch header with at least 90000 blocks apart from the current header
  in the absence of other synchronisation points.

  Here just the nearest epoch header is used. The assumption is that all
  the checkpoints before have been vetted already regardless of the
  current branch.

details:
  The behaviour of using the nearest vs the minimum distance epoch is
  controlled by a flag and can be changed at run time.

* Analysing processing time (patch adds some debugging/visualisation support)

why:
  At the first half million blocks of the Goerli replay, blocks on the
  interval #194854..#196224 take exceptionally long to process, but not
  due to PoA processing.

details:
  It turns out that much time is spent in p2p/excecutor.processBlock()
  where the elapsed transaction execution time is significantly greater
  for many of these blocks.

  Between the 1371 blocks #194854..#196224 there are 223 blocks with more
  than 1/2 seconds execution time whereas there are only 4 such blocks
  before and 13 such after this range up to #504192.

* fix debugging symbol in clique_desc (causes CI failing)

* Fixing canonical reference tests

why:
  Two errors were introduced earlier but ovelooked:
   1. "Remove fakeDiff flag .." patch was incomplete
   2. "Not observing minimum distance .." introduced problem w/tests 23/24

details:
  Fixing 2. needed to revert the behaviour by setting the
  applySnapsMinBacklog flag for the Clique descriptor. Also a new
  test was added to lock the new behaviour.

* Remove cruft

why:
  Clique/PoA processing was intended to take place somewhere in
  executor/process_block.processBlock() but was decided later to run
  from chain/persist_block.persistBlock() instead.

* Update API comment

* ditto
2021-07-30 15:06:51 +01:00
Jamie Lokier 22241f3cff
GraphQL: Update `protocolVersion` expected result to hard-coded 65
Use a hard-coded number instead of the same expression as the client, so that
bugs introduced via that expression are detected.  Using the same expression as
the client can hide issues when the value is wrong in both places.

When the expected value genuinely changes, it'll be obvious.
Just change this number.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-27 14:12:48 +01:00
Jamie Lokier b3cf42ed28
RPC: Update `eth_protocolVersion` expected result to hard-coded 65
Use a hard-coded number instead of the same expression as the client, so that
bugs introduced via that expression are detected.  Using the same expression as
the client can hide issues when the value is wrong in both places.

When the expected value genuinely changes, it'll be obvious.
Just change this number.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-27 14:12:46 +01:00
Jamie Lokier 00647b373c
Sync: Switch other modules over to the `eth/65` module
Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-27 14:12:41 +01:00
Jordan Hrycaj a0d0e35a70
Renamed source file clique_utils => clique_helpers (#762)
* Renamed source file clique_utils => clique_helpers

why:
  New name is more in line with other modules where local libraries
  are named similarly.

* re-implemented PoA verification module as clique_verify.nim

details:
  The verification code was ported from the go sources and provisionally
  stored in the clique_misc.nim source file.

todo:
  Bring it to life.

* re-design Snapshot descriptor as: ref object

why:
  Avoids some copying descriptor objects

details:
  The snapshot management in clique_snapshot.nim has been cleaned up.

todo:
  There is a lot of unnecessary copying & sub-list manipulation of
  seq[BlockHeader] lists which needs to be simplified by managing
  index intervals.

* optimised sequence handling for Clique/PoA

why:
  To much ado about nothing

details:
  * Working with shallow sequences inside PoA processing avoids
    unnecessary copying.
  * Using degenerate lists in the cliqueVerify() batch where only the
    parent (and no other ancestor) is needed.

todo:
  Expose only functions that are needed, shallow sequences should be
  handles with care.

* fix var-parameter function argument

* Activate PoA engine -- currently proof of concept

details:
  PoA engine is activated with newChain(extraValidation = true) applied
  to a PoA network.

status and todo:
  The extraValidation flag on the Chain object can be set at a later
  state which allows to pre-load parts of the block chain without
  verification. Setting it later will only go back the block chain to
  the latest epoch checkpoint. This is inherent to the Clique protocol,
  needs testing though.

  PoA engine works in fine weather mode on Goerli replay. With the
  canonical eip-225 tests, there are quite a few fringe conditions
  that fail. These can easily fudged over to make things work but need
  some more work to understand and correct properly.

* Make the last offending verification header available

why:
  Makes some fringe case tests work.

details:
  Within a failed transaction comprising several blocks, this
  feature help to identify the offending block if there was a
  PoA verification error.

* Make PoA header verifier store the final snapshot

why:
  The last snapshot needed by the verifier is the one of the parent but
  the list of authorised signer is derived from the current snapshot. So
  updating to the latest snapshot provides the latest signers list.

details:
  Also, PoA processing has been implemented as transaction in
  persistBlocks() with Clique state rollback.

  Clique tests succeed now.

* Avoiding double yields in iterator => replaced by template

why:
  Tanks to Andri who observed it (see #762)

* Calibrate logging interval and fix logging event detection

why:
  Logging interval as copied from Go implementation was too large and
  needed re-calibration. Elapsed time calculation was bonkers, negative
  the wrong way round.
2021-07-21 14:31:52 +01:00
Jordan Hrycaj cfe955c962
Feature/implement poa processing (#748)
* re-shuffled Clique functions

why:
  Due to the port from the go-sources, the interface logic is not optimal
  for nimbus. The main visible function is currently snapshot() and most
  of the _procurement_ of this function result has been moved to a
  sub-directory.

* run eip-225 Clique test against p2p/chain.persistBlocks()

why:
  Previously, loading the test block chains was fugdged with the purpose
  only to fill the database. As it is now clear how nimbus works on
  Goerli, the same can be achieved with a more realistic scenario.

details:
  Eventually these tests will be pre-cursor to the reply tests for the
  Goerli chain supporting TDD approach with more simple cases.

* fix exception annotations for executor module

why:
  needed for exception tracking

details:
  main annoyance are vmState methods (in state.nim) which potentially
  throw a base level Exception (a proc would only throws CatchableError)

* split p2p/chain into sub-modules and fix exception annotations

why:
  make space for implementing PoA stuff

* provide over-loadable Clique PRNG

why:
  There is a PRNG provided for generating reproducible number sequences.
  The functions which employ the PRNG to generate time slots were ported
  ported from the go-implementation. They are currently unused.

* implement trusted signer assembly in p2p/chain.persistBlocks()

details:
  * PoA processing moved there at the end of a transaction. Currently,
   there is no action (eg. transaction rollback) if this fails.
  * The unit tests with staged blocks work ok. In particular, there should
    be tests with to-be-rejected blocks.
  * TODO: 1.Optimise throughput/cache handling; 2.Verify headers

* fix statement cast in pool.nim

* added table features to LRU cache

why:
  Clique uses the LRU cache using a mixture of volatile online items
  from the LRU cache and database checkpoints for hard synchronisation.
  For performance, Clique needs more table like features.

details:
  First, last, and query key added, as well as efficient random delete
  added. Also key-item pair iterator added for debugging.

* re-factored LRU snapshot caching

why:
  Caching was sub-optimal (aka. bonkers) in that it skipped over memory
  caches in many cases and so mostly rebuild the snapshot from the
  last on-disk checkpoint.

details;
  The LRU snapshot toValue() handler has been moved into the module
  clique_snapshot. This is for the fact that toValue() is not supposed
  to see the whole LRU cache database. So there must be a higher layer
  working with the the whole LRU cache and the on-disk checkpoint
  database.

also:
  some clean up

todo:
  The code still assumes that the block headers are valid in itself. This
  is particular important when an epoch header (aka re-sync header) is
  processed as it must contain the PoA result of all previous headers.

  So blocks need to be verified when they come in before used for PoA
  processing.

* fix some snapshot cache fringe cases

why:
  Must not index empty sequences in clique_snapshot module
2021-07-14 16:13:27 +01:00
KonradStaniec a59a2c61b6
[FEATURE] Add support for handling experimental api call (#746)
* [FEATURE] Add support for handling experimental api call
2021-07-07 11:04:18 +02:00
Jordan Hrycaj fbff3aea68
Feature/goerli replay clique poa (#743)
* extract unused clique/mining support into separate file

why:
  mining is currently unsupported by nimbus

* Replay first 51840 transactions from Goerli block chain

why:
  Currently Goerli is loaded but the block headers are not verified.
  Replaying allows real data PoA development.

details:
  Simple stupid gzipped dump/undump layer for debugging based on
  the zlib module (no nim-faststream support.)

  This is a replay running against p2p/chain.persistBlocks() where
  the data were captured from.

* prepare stubs for PoA engine

* split executor source into sup-modules

why:
  make room for updates, clique integration should go into
  executor/update_poastate.nim

* Simplify p2p/executor.processBlock() function prototype

why:
  vmState argument always wraps basicChainDB

* split processBlock() into sub-functions

why:
  isolate the part where it will support clique/poa

* provided additional processTransaction() function prototype without _fork_ argument

why:
  with the exception of some tests, the _fork_ argument is always derived
  from the other prototype argument _vmState_

details:
  similar situation with makeReceipt()

* provide new processBlock() version explicitly supporting PoA

details:
  The new processBlock() version supporting PoA is the general one also
  supporting non-PoA networks, it needs an additional _Clique_ descriptor
  function argument for PoA state (if any.)
  The old processBlock() function without the _Clique_ descriptor argument
  retorns an error on PoA networgs (e.g. Goerli.)

* re-implemented Clique descriptor as _ref object_

why:
  gives more flexibility when moving around the descriptor object

details:
  also cleaned up a bit the clique sources

* comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 14:14:45 +01:00
jangko 6d266178d7
London HF: bump ethereum/tests submodule
This commit concludes a series of London related PRs
fixes #716
2021-06-30 20:44:35 +07:00
jangko c858148a43
fix test_graphql because of London HF changes 2021-06-30 20:44:35 +07:00
jangko d47be9047f
London: fix test_blockchain_json and test_generalstate_json
new London HF requires update to test code
2021-06-30 20:41:29 +07:00
jangko 4a188788bd
preparation for EIP-1559 implementation
- unify signTx in test_helper and signTransaction in rpc_utils
  and put it into transaction.nim
- clean up mess by previous EIP-2930
2021-06-29 07:33:48 +07:00
Jordan Hrycaj a49a812879
Jordan/fix some failing nohive tests (#727)
* continue importing rlp blocks

why:
  a chain of blocks to be imported might have legit blocks
  after rejected blocks

details:
  import loop only stops if the import list is exhausted or if there
  was a decoding error. this adds another four to the count of successful
  no-hive tests.

* verify DAO marked extra data field in block header

why:
  was ignored, scores another two no-hive tests

* verify minimum required difficulty in header validator

why:
  two more nohive tests to succeed

details:
  * subsumed extended header tests under validateKinship() and renamed it
    more appropriately validateHeaderAndKinship()
  * enhanced readability of p2p/chain.nim
  * cleaned up test_blockchain_json.nim

* verify positive gasUsed unless no transactions

why:
  solves another to nohive tests

details:
  straightened test_blockchain_json chech so there is no unconditional
  rejection anymore (based on the input test  scenario)
2021-06-24 16:29:21 +01:00
jangko 2269d16c4c
reintegrate test_rpc back into all_tests
for a long time, test_rpc will crash all_tests
if they are combined together. but now it works fine
2021-06-22 08:32:32 +07:00
jangko 3a826b8406
replace all unittest usage with unittest2
- test_graphql
- test_rpc
- test_hexstrings

fixes #691
2021-06-21 20:30:01 +07:00
jangko d0782cdb0d
fixes some of graphql resolver
following recent fixes in upstream hive,
we also update our graphql resolvers
2021-06-17 18:18:28 +07:00
Jordan Hrycaj 82e6cd991d maintenance update
why:
  some handy features were intended to support the unit test from
  the clique/clique_test.go source (the other one is from
  clique/snapshot_test.go.)
  as this test cannot realistically be implemented without the full
  api (includes mining support), it is left as that
2021-06-17 08:03:57 +01:00
Jordan Hrycaj 90b012ad3f clarify epoch sync handling (effectively a comment update only)
why:
  autorisation list verification is performed in the main module along
  with other header verifications
2021-06-17 08:03:57 +01:00
Jordan Hrycaj dd7ca174f0 all snapshot unit tests succeed
details:
  for extra verbosity compile as: nim c -r -d:debug [..] test_clique.nim
2021-06-17 08:03:57 +01:00
Jordan Hrycaj 61e460c125 Most snapshot unit tests work
details:
  three test cases still fail which are skipped
  test suite is linked to all_tests list
2021-06-17 08:03:57 +01:00
Jordan Hrycaj 87edd80557 Update snapshot smoke test
details:
  can initialise & load all tests

todo:
  double check tests that are supposed to return error
  follow up succesful voting results
2021-06-17 08:03:57 +01:00
Jordan Hrycaj 1de2cc1a77 Basic tests for Clique PoA/Consensus engine
details:
  test scenario from eip-225 reference implementation,
  set up unittes2 test framework
  smoke test for first sample ok (not functional yet)
2021-06-17 08:03:57 +01:00
Jamie Lokier 168e69080b
Tests: Disable precompile fixtures when ENABLE_EVMC=1
The precompile fixture tests (`test_precompiles.nim`) are now incompatible with
ENABLE_EVMC=1, because `noTransfer` does not work with the new EVMC path.

This will start working again after the host/EVM context rework (needed for
other reasons anyway), but for now disable these tests so that other work can
continue.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 18:29:40 +01:00
Jamie Lokier 7c90d8de70
EVM: Remove `vm_forks` everywhere, use common forks list instead
The common forks list was already used, redirected via `vm_forks` for
historical compatibility.  Remove the old `vm_forks` now and divert all imports
to the common forks list outside the EVM.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 15:36:31 +01:00
Jamie Lokier aee0fe39d2
EVM: Remove `vm_types2` everywhere, use common forks list instead
File `vm_types2` is obsolete.  Remove this file and divert all imports to the
common forks list outside the EVM, or in some cases they don't need it anyway.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 15:36:31 +01:00
Jamie Lokier 84269ddccf
Forks: Use capitalized names again for presentation (logging etc.)
Fork names were originally capitalized, and were made lower case by @narimiran
in commit 36a7519 to satisfy `parseEnum` in some tests.  Restore the
capitalization and make the tests work with it.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 15:36:31 +01:00
Jamie Lokier 775231eef1
EVM: Apply EIP-6 in the code (affects both vm and vm2)
The rationale in EIP-6[1] for changing names to `selfDestruct` applies to code
as much as it does to specs.  Also, Ethereum uses the new names consistently,
so it's useful for our code to match the terms used in later EIP specs and
testsuite entries.

This change is straightforward, and is a prerequisite for patches to come that
do things with the `selfDestruct` fields.

[1] https://eips.ethereum.org/EIPS/eip-6
Hudson Jameson, "EIP-6: Renaming SUICIDE opcode," Ethereum Improvement
Proposals, no. 6, November 2015.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 15:36:30 +01:00
Jamie Lokier 3ba22809f0
Whisper: Remove all Whisper tests and test support
Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-01 18:12:47 +01:00
Jamie Lokier 5e718bcbe2
EVM: Remove most unused imports of `vm_*` files
Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-01 16:54:38 +01:00
Jamie Lokier f4de9d3c80
Tests: Use `ZERO_ADDRESS` in precompile tests
Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-05-27 12:16:27 +01:00
Jamie Lokier 78233a4edc
Tests: Explain rules of "to" for call/create, use `ZERO_ADDRESS`
The conditions mentioned in the old TODO comment have been checked.  All
fixtures have either 40 hex digits or empty string for "to".  There is a test
with all-zeros, and it means send to that account, not contract creation.
Empty string means contract creation.

This patch does not change the relaxed parsing where fewer than 40 digits is
accepted.  We should probably be stricter about this.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-05-27 12:15:02 +01:00
jangko 91b50235b0 add assertion guard in test_blockchain_json's parseBlock function
we don't want any surprice because of unrecognized key in test fixture
classified as `hasException`.
2021-05-26 15:46:04 +01:00
Jordan Hrycaj e5947f4db6 Deep copy semantics for LRU cache
why:
  follows standard nim semantics

details:
  changed Table to TableRef in previous patch which was the
  wrong choice (see andri's comment.)
2021-05-26 11:12:52 +01:00
Jordan Hrycaj 7b72109afa Use sorted RLP serialisation for LRU cache
why:
  previously, table data were stored with the table iterator. while
  loading a table with permuted entries will always reconstruct equivalent
  tables (in the sense of `==`), serialisation data are not comparable.

  this patch produces always the same serialised data for equivalent
  tables.
2021-05-26 07:58:12 +01:00
Jordan Hrycaj b83b47e541 LRU cache tests makeover
why:
  source-local unit tests would hardly be triggered by github CI as rightly
  criticised with the last patch.

details:
  source-local unit tests have been moved to tests folder.

  this version also contains rlp serialisation code so rlp encode/decode
  will apply tranparently. this is not needed in p2p/validate but will be
   useful with the clique protocol.
2021-05-26 07:58:12 +01:00
Jordan Hrycaj 3663b1603f pulled out cache logic into separate file
why:
  handy to re-use, eg. for upcoming clique implementation
2021-05-24 07:57:21 +01:00
Jordan Hrycaj d6a5cecb98 re-wrote validation with exceptionless functions
why:
  exceptions were from test code should be avoided in production code
2021-05-24 07:57:21 +01:00
Jordan Hrycaj 40c7bdfc06 update lookup cache management
details:
  enable fifo behaviour, using cache as argument
2021-05-24 07:57:21 +01:00
Jordan Hrycaj ce8e5511e3 backport from test_blockchain_json, see issue #666 2021-05-24 07:57:21 +01:00
jangko a0d10f5728
drop PublicNetwork enum usage and replace it with NetworkId
we cannot limit the `--networkid` switch to values available in
`PublicNetwork` enum. it should able to accept very wide range of
custom NetworkId.
2021-05-20 14:04:16 +07:00
jangko 0ecf9fe1af
add more query fields and resolvers to graphql api
after EIP2718/EIP2930, we have additional fields:

type AccessTuple {
  address: Address!
  storageKeys : [Bytes32!]
}

type Transaction {
  r: BigInt!
  s: BigInt!
  v: BigInt!
  # Envelope transaction support
  type: Int
  accessList: [AccessTuple!]
}

close #606
2021-05-18 07:32:03 +07:00
jangko 43bc010c9c
remove test_vm_json
reason:
- they already covered by test_blockchain_json
  and test_generalstate_json.
- new ethereum/tests already remove them,
  nothing to tests anymore
2021-05-17 11:14:34 +07:00
jangko 77272544c0
fixes test_helper's jsonTestImpl
when a skipped test case not in the skip list
but skipped using `testStatusIMPL`, the generated
markdown file list it as failed.

now that bug is fixed.
2021-05-17 11:14:33 +07:00
jangko 224b1e833b
bump ethereum/tests submodule
reason: allow EIP-2930 test cases to participate in test
2021-05-17 01:30:44 +07:00
jangko 79a52b11fd
fixes test_blockchain_json: better error message
also remove local EthBlock type and use EthBlock from nim-eth/common
2021-05-17 01:29:03 +07:00
jangko 76543da456
disable EIP-2537: Precompile for BLS12-381 curve operations
reason: not included in berlin hard fork

but we keep the code around, for future inclusion
2021-05-17 01:29:03 +07:00
jangko 3ccc4642f2
disable EIP-2315: Simple Subroutines for the EVM
reason: not included in berlin hard fork
2021-05-17 01:29:03 +07:00
jangko 6fc3df637c
reenable EIP-2565: modExp gas cost
now it's officially included in berlin hard fork
2021-05-17 01:28:31 +07:00
jangko a2712c5c7a
eip2718: fixes test_rpc.nim 2021-05-15 18:09:36 +07:00
jangko 79044f1e92
eip2718: test_blockchain_json pass test 2021-05-15 18:09:35 +07:00
jangko 2a9c3982d9
remove code duplication from test_blockchain_json
now test_blockchain_json is using block validation code
from p2p/executor.nim instead of using it's own
block validation code.

this will reduce maintenance cost and fixes #592.
2021-05-14 15:37:30 +07:00
jangko db65f92e66
refactor vmConfiguration in test_blockchain_json
instead of using it's own blockNumber to Fork calculation,
we switch to ChainConfig to Fork to ensure smooth transition
when using common block validator.
2021-05-14 15:37:29 +07:00
jangko beb274d98c
remove unnecessary codes from test_blockchain_json
except for genesis block, we are not parsing block header from json
node anymore.

we parse block headers from block RLP, it is the same thing.
2021-05-14 15:37:29 +07:00
jangko f2491e6307
fixes crappy custom genesis and chain config parser
instead of using stdlib/json, now we switch to json_serialization
the result is much tidier code and more robust when parsing
optional fields.

fixes #635
2021-05-13 16:04:08 +07:00
jangko 97f4226171
update berlin fork number in config.nim
also update test_forkid because of berlin changes
2021-05-12 17:24:27 +07:00
jangko 5ee918f4ef
fixes test_graphql crash due to recent changes related to `chainId`
now test_graphql takes another route to initialize the empty db
that is safer instead of bypassing commonly used route.
2021-05-12 09:45:09 +07:00
jangko 2d3d450075
fixes `validateFixedLenHex` in graphql/ethapi.nim
now it can detect too long hex besides padding too short hex
2021-05-12 08:12:26 +07:00
jangko d0546becfb
add query complexity calculator to graphql/ethapi
this will allow us to pass two more hive tests
2021-05-10 22:22:04 +07:00
jangko ad1edd9af6
fixes graphql test cases
after recent bugfix in graphql resolvers, many of
test cases need to updated too
2021-05-05 11:20:12 +07:00
Jamie Lokier 1574136a25
Precompiles: Change precompile tests to use fixtureCallEvm
Move the EVM setup and call in precompile tests to `fixtureCallEvm` in
`call_evm`.  Extra return values needed for testing are returned specially, and
the convention for reporting gas used is changed to match `asmCallEvm`.

Although the precompile tests used `execPrecompiles` before, `executeOpcodes`
does perfectly well as a substitute, allowing `fixtureCallEvm` to be shared.

_Significantly, this patch also makes `Computation` more or less an internal
type of the EVM now._

Nothing outside the EVM (except `call_evm`) needs access any more to
`Computation`, `execComputation`, `executeOpcodes` or `execPrecompiles`.
Many imports can be trimmed, some files removed, and EVMC is much closer.

(As a bonus, the functions in `call_evm` reveal what capabilities parts of the
program have needed over time, makes certain bugs and inconsistencies clearer,
and suggests how to refactor into a more useful shared entry point.)

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-05-05 02:31:46 +01:00
Jamie Lokier 236a65d598
Fixtures: Make fixture "vm json tests" use new function fixtureCallEvm
Move the EVM setup and call in fixtures "vm json tests" to new function
`fixtureCallEvm` in `call_evm`.  Extra return values needed for testing are
returned specially.

This entry point is different from all other `..CallEvm` type functions,
because it uses `executeOpcodes` instead of `execComputation`, so it doesn't
update the account balance or nonce on entry and exit from the EVM.

The new code is a bit redundant and simplistic intentionally, as the purpose is
to move functionality to `call_evm` with high confidence nothing really
changed.  The calls will be jointly refactored afterwards to merge differences.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-05-04 15:21:15 +01:00
Jamie Lokier 9e99bb6cd9
Fixtures: Prepare fixtureSetupComputation to support fixtureCallEvm
In the `text_vm_json` ("fixtures") test code, there is another variant of
`rpcSetupComputation` and `txSetupComputation` with slightly different
paremeters.  The similarity is obvious.

It is a special setup for testing, though, as it requires slightly different
parameters.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-05-04 15:21:14 +01:00
jangko 39ce2390ae
fixes `getRecipient`: using `sender` param instead of calculating sender itself
usually, there is always a sender around `getRecipient` call.
no need to recalculate sender. and more important, in some of
JSON-RPC/GraphQL call, the sender is come from `rpcCallData`,
not from `tx.getSender`. or in ohter situation when the tx is
an unsigned tx, without `r,s,v` fields to calculate sender.
2021-05-04 15:31:47 +07:00
jangko c46b7186ca
update graphql test case because of bugfixes in nim-graphql
recent introspection bugfixes in nim-graphql alter the result
of introspection system.
2021-05-04 14:26:10 +07:00
jangko 10142ad711
add missing `disableParamFiltering` in test_graphql.nim
if there is no `disableParamFiltering()`, and the test is run via
all_tests.nim, no test unit will be executed
2021-05-04 13:03:08 +07:00
Jamie Lokier d2586c3a73
Assembler: Make macro_assembler tests use new function asmCallEvm
Move the EVM setup and call in `macro_assembler` (`runVM`) entirely to new
function `asmCallEvm` in `call_evm`.  Extra return values needed for
testing are returned specially from `asmCallEvm`.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-05-04 01:36:00 +01:00
Jamie Lokier 834449d943
Assembler: Second asmSetupComputation, calls the first
The second `asmSetupComputation looks up state by block number and preceding
block number, modifies the first transaction with code for testing, and uses
some parts of that transaction to setup an an EVM test.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-05-04 01:03:55 +01:00
Jamie Lokier 5728491d60
Assembler: First asmSetupComputation to support asmCallEvm
In the `macro_assembler` test code, `initComputation` is another variant of
`rpcSetupComputation` and `txSetupComputation` with slightly different
paremeters.  The similarity is obvious.

It is a special setup for testing, though, as it requires a contract-creation
transaction for parameters, but sets up a `CALL` execution not `CREATE`.

Gather this into `call_evm`: `initComputation` -> `asmSetupComputation`.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-05-04 01:03:55 +01:00
Jordan Hrycaj 59d7ba1f1e print compiler warning about the VM used
why:
  handy to have confirmation about which of the three different VMs
  is activated
2021-04-28 15:24:14 +03:00
jangko 639674d341
add disableParamFiltering() to both test_generalstate_json and test_blockchain_json
disableParamFiltering() is called in test_all, so its ok when
called via test_all. but when test_generalstate_json or
test_blockchain_json is called independently, and we are using
command line switch, the default param handler from std/unittest
will interfere. hence need to disable it.
2021-04-25 12:20:03 +07:00
jangko a95cec3aa8
add graphql test suite 2021-04-24 11:01:56 +07:00
Jamie Lokier 8a806da600
Bugfix: Fix blockchain tests: It was quietly skipping 4654 of them
The "new block chain json tests" were being skipped on Linux, but silently so
that CI didn't notice.  These are a significant part of the Ethereum test suite.

See the missing output from `make test`, also visible in CI logs for Linux
targets (prior to this commit):

      [OK] tests/fixtures/eth_tests/TransactionTests/ttGasPrice/TransactionWithGasPriceOverflow.json
      [OK] tests/fixtures/eth_tests/TransactionTests/ttGasPrice/TransactionWithHighGasPrice.json

    [Suite] new block chain json tests
            <-- nothing here

    [Suite] Fork ID tests
      [OK] MainNet
      [OK] RopstenNet

Commit 3d468a7 (`fixes path pointing to eth_tests`) renamed the JSON fixture
source directoryf in the witness-builder tests but not the regular blockchain
tests.  As a result, searching for JSON test files yielded zero results.  To
make this less likely in future, zero results is now an error.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-08 16:40:53 +01:00
Jordan Hrycaj c8582583ef
merge vm_memory, vm_interpreter, and vm_utils_numeric => vm_internals
why:
  currently used for tests only
2021-03-31 18:15:27 +01:00
Jordan Hrycaj 00ba7a2718
merge vm_forks and vm_opcode_values => vm_type2
why:
  all types, but they cannot be merged int vm_types because of a circular
  dependency.
2021-03-31 17:53:15 +01:00
Jordan Hrycaj 7c28d5d362
provide vm_utils_numeric as import/export wrapper
details:
  moved original vm/interpreter/utils/utils_numeric.nim => vm/interpreter/utils/utils_numeric.nim
2021-03-31 16:49:07 +01:00
Jordan Hrycaj 99568c9b46
provide vm_opcode_values as import/export wrapper
details:
  moved original vm/interpreter/opcode_values.nim => vm/interpreter/nvm_opcode_values.nim
2021-03-31 16:49:03 +01:00
Jordan Hrycaj cf63b9b03f
provide vm_memory as import/export wrapper
details:
  moved original vm/memory.nim => vm/nvm_memory.nim
2021-03-31 16:48:44 +01:00
Jordan Hrycaj 7b5d00307c
provide vm_precompiles as import/export wrapper
details:
  moved original vm/precompiles.nim => vm/nvm_precompiles.nim
2021-03-31 16:47:15 +01:00
Jordan Hrycaj 5ce7ca6b32
provide vm_interpreter as import/export wrapper
details:
  moved original vm/interpreter.nim => vm/nvm_interpreter.nim
2021-03-31 16:47:08 +01:00
Jordan Hrycaj eee24de450
provide vm_message as import/export wrapper
details:
  moved original vm/message.nim => vm/nvm_message.nim
2021-03-31 16:47:02 +01:00
Jordan Hrycaj cf3a356d76
provide vm_computation as import/export wrapper
details:
  moved original vm/computation.nim => vm/nvm_computation.nim
2021-03-31 16:38:10 +01:00
Jordan Hrycaj 3a3e4d5707
provide vm_forks as import/export wrapper
details:
  moved original vm/interpreter/vm_forks.nim => vm/interpreter/nvm_forks.nim
2021-03-31 16:03:34 +01:00
jangko 8486c1834c fixes #558, use distinct uint for ChainId and NetworkId to prevent confusion 2021-02-14 11:01:27 +07:00
jangko 5248be1766 bump eth_tests to pre YOLO v3 2021-02-03 14:31:16 +07:00
jangko b6ad47f3a4 fixes evmc bug and add github action job to test evmc 2021-01-20 11:50:07 +07:00
jangko 03bed02512 split legacy and new test to save time 2021-01-14 23:22:28 +07:00
jangko ad284e3d25 fixes EIP2929 SLOAD 2021-01-14 23:22:28 +07:00
jangko f6c44ffcc0 fixes EIP2929 CALL opCode 2021-01-14 23:22:28 +07:00
jangko 9709525916
fix byzantium bug related to revertPrecompileTouched 2021-01-13 08:08:56 +07:00
jangko 15cacc749d
clean up EIP2929 for test 2021-01-12 16:17:00 +07:00
jangko f906d177f4
add comments about disabled EIPs 2021-01-11 15:33:30 +07:00
jangko f2b483d6ad
access list implementation 2021-01-11 14:54:55 +07:00
jangko 08c8b12821
bump eth_tests 2021-01-11 14:54:33 +07:00
jangko 8f7e45fa08
fixes test codes 2021-01-11 14:54:11 +07:00
jangko 16dd053a72
bump eth_tests 2021-01-06 21:46:16 +07:00
jangko 3d468a7a4c
fixes path pointing to eth_tests 2021-01-06 21:45:48 +07:00
jangko 74cccdfe9e
fixes test codes to deal with eth_tests submodule 2021-01-06 17:02:19 +07:00
jangko c0db998947
add eth_tests submodule 2021-01-06 17:01:46 +07:00
jangko 18bfd34d84
remove some of json fixtures 2021-01-06 17:01:03 +07:00
Ștefan Talpalaru 5e17bd5e16
test_precompiles: fix enum case
This fixes compilation with Nim-1.4.2.
2020-12-09 04:04:27 +01:00
andri lim de12fb3c94
Merge pull request #546 from status-im/eip2537
Eip2537 implementation
2020-12-02 20:39:24 +07:00
jangko 0799b4534c
EIP2537 part 2 2020-11-28 23:13:10 +07:00
jangko c1b7ae5b02
EIP2537 part 1 2020-11-27 21:42:17 +07:00
narimiran 73c4837d0d change case of `berlin`
In `vm_forks.nim`, `FkBerlin`'s value is (lowercase) `"berlin"`.

Another example of using lowercase in similar code:
`constantinople` in `test_op_bit.nim`
2020-11-27 09:50:28 +01:00
andri lim 360b74327e
Merge pull request #544 from status-im/precompiles_test_rework
Precompiles test rework
2020-11-26 10:34:26 +07:00
jangko 92fec713db
add EIP2537 test vectors 2020-11-25 20:55:53 +07:00
jangko 971e00e580
precompile contracts test rework 2020-11-25 20:42:15 +07:00
jangko a263e6b1a6
implement EIP2315 tests 2020-11-25 18:23:02 +07:00
jangko 5bb6418bcb
implement EIP2565 2020-11-24 16:19:02 +07:00
jangko 97f73fd03d
implement EIP 2046 2020-11-19 14:23:07 +07:00
jangko f987e86562
implement more eth rpc 2020-07-30 14:21:11 +07:00
jangko 9c38266ba7
implement eth_estimateGas 2020-07-29 12:42:32 +07:00
jangko c9802edfce
setup block and state env for more complex eth rpc tests 2020-07-28 23:48:45 +07:00
jangko d089a61539
add compiletime evm bytecode generator to macro_assembler 2020-07-24 19:54:27 +07:00
jangko bb89a296dd
implement eth_signTransaction, eth_sendTransaction, eth_sendRawTransaction, eth_call 2020-07-24 19:44:36 +07:00
jangko 7819dae7ce
implement eth_signTransaction 2020-07-23 22:30:42 +07:00
jangko f82dff64fa
implement more eth rpc and keystore management 2020-07-23 14:54:32 +07:00
jangko 336efdb0c3
implement web3, net, and some eth namespace rpc 2020-07-22 23:57:55 +07:00
jangko 165f9fea2e
reduce warnings 2020-07-21 13:15:06 +07:00
jangko 91eb5db346
save some CI time by combining tools build 2020-07-21 11:28:03 +07:00
jangko 845671bf0a
fix compilation error following breaking changes in nim-eth 2020-07-20 13:50:05 +07:00
jangko e37cacd8f1
implement forkid calculation and tests for each supported network 2020-07-04 13:23:09 +07:00
jangko 12ddfee675
fix compilation error related to lib-secp256k1 changes 2020-06-24 17:07:33 +07:00
jangko 49460b6b1e
move buildWitness from test_blockchain_json to vm_state 2020-06-18 13:16:38 +07:00
jangko 3947e9a853
piggyback generate block witness test on test_blockchain_json 2020-06-06 10:26:36 +07:00
jangko 8efcfd32d8
fix test_rpc: switch state_db to accounts_cache 2020-06-01 20:37:00 +07:00
jangko 8ee0ab4e89
fix macro_assembler -> add call to acounts_cache.persist 2020-06-01 13:58:14 +07:00
jangko 844071033a
fix at various places related to missing accounts_cache.persist call 2020-06-01 13:45:32 +07:00
jangko 079579c1cb
fix test_blockchain_json -> add a call to accounts_cache.persist 2020-06-01 12:00:36 +07:00
jangko 46e9c11d3f
fix test_generalstate_json -> add a call to accounts_cache.persist 2020-06-01 11:49:56 +07:00
jangko 9cce8d695b
fix test_helpers -> add a call to accounts_cache.persist 2020-06-01 11:28:27 +07:00
jangko 71514a0a66
replace state_db with accounts_cache 2020-05-30 10:14:59 +07:00
jangko 7c0d24a648
add instruction comment for test_fuzz 2020-05-20 12:41:05 +07:00
jangko 9d101948ab
use 'const' instead of 'let' for module names in all_tests 2020-05-20 11:43:46 +07:00
jangko 048c8a41f1
all_tests now print the name of failed module instead of only index number 2020-05-20 11:35:27 +07:00
andri lim 7a0215608e
Merge pull request #496 from status-im/stateless_client_experiment
[WIP] Stateless client experiment: The Block Witness
2020-05-08 19:58:50 +07:00
andri lim 758ebabd87
working test_witness_json prototype 2020-05-08 12:16:24 +07:00
andri lim 9c999ffa9c
enable block witness test 2020-05-05 20:46:17 +07:00
andri lim c9e49bf68a
fix accounts cache init API 2020-04-29 12:00:44 +07:00
andri lim 8747fe1ecd add optional outputName to jsonTest macro 2020-04-21 12:05:46 +03:00
Jacek Sieka 4ade5797ee
rlp: don't use ranges / experimental features (#495) 2020-04-20 20:12:44 +02:00
andri lim af02a3b1b2
reduce unused import warnings 2020-04-15 19:05:57 +07:00
andri lim 5e694c6874
fix test_blockchain_json 2020-04-12 19:02:03 +07:00
andri lim d2b0ca62b2
fix GlacierMuir to MuirGlacier typo 2020-04-12 18:13:22 +07:00
andri lim 5ca7370407
add ropsten difficulty test 2020-04-12 18:09:18 +07:00
andri lim 87bae2bb78
switch to new toFork 2020-04-12 18:02:59 +07:00
andri lim d69ede6060 add goerli testnet genesis data[skip ci] 2020-04-09 19:13:17 +03:00
Jacek Sieka 1d472cf090
Eth keys (#482)
* bump nim-eth, fix deprecated calls
2020-04-05 15:12:48 +02:00
andri lim 266e0ddb1e
room for EIP-1283 2020-03-24 17:21:13 +07:00
Zed fff29f7a6a Replace markdown report code with testutils 2020-03-10 14:03:04 +02:00
andri lim 0a35cce48a add accounts cache read only operations tests 2020-02-27 14:38:42 +02:00
andri lim 08168ef121 Update BCT json fixtures 2020-02-21 13:01:35 +02:00
andri lim e12fdec7c7 update GST json fixtures 2020-02-21 13:01:21 +02:00
andri lim 6b6584c4d0 Glacier Muir update 2020-02-20 09:21:35 +02:00
andri lim a6cde0928e [skip ci] remove test_generalstate_failing.nim 2020-02-20 09:16:29 +02:00
andri lim 32574fcebe allow user to turn pruning on/off when run test 2020-02-20 09:16:29 +02:00
andri lim ed5710fa17 allow user to select legacy or new test suite for GST and BCT 2020-02-20 09:16:29 +02:00
andri lim 2fbabd25a4 implement aleth/geth/parity compatibility mode -- 100% pass test 2020-02-20 09:08:44 +02:00
andri lim 4a786d8cf7 update tx tests json fixtures 2020-02-20 09:02:20 +02:00
andri lim 15c9fa54ec fixes modexp gasFee bug 2020-02-18 20:11:36 +02:00
andri lim ad1b27d2a4 fixes failed tx test 2020-02-12 17:55:14 +02:00
andri lim 85c36647d1 make tests green 2020-02-12 17:53:26 +02:00
andri lim 1e447ee2db
make tests green 2020-02-05 23:08:26 +07:00
Ștefan Talpalaru ea38893416
fix test dir name 2020-01-31 22:53:28 +01:00
andri lim 55494f06e5
move 'validateTransaction' from GST into 'processTransaction' 2020-01-24 19:52:55 +07:00
andri lim 349d033d05 add whenIsMainModule to test_tracer_json 2020-01-23 18:07:44 +02:00
andri lim d656f7f701 rename 'txContext' to 'setupTxContext' 2020-01-23 18:07:44 +02:00
andri lim 933b2dad78 remove code from EVM Message and load code in 'Computation' 2020-01-23 18:07:44 +02:00