Commit Graph

189 Commits

Author SHA1 Message Date
Kim De Mey 8523d8b40e
Bump submodules and fix related issues (#1448)
* Updated to the latest nim-eth, nim-rocksdb, nim-web3

* Bump nimbus-eth2 module and fix related issues

Temporarily disabling Portal beacon light client network as it is
a lot of copy pasted code that did not yet take into account
forks. This will require a bigger rework and was not yet tested
in an actual network anyhow.

* More nimbus fixes after module bumps

---------

Co-authored-by: Adam Spitz <adamspitz@status.im>
Co-authored-by: jangko <jangko128@gmail.com>
2023-01-27 15:57:48 +01:00
jangko 13e5b1a76b
fix persistHeaderToDb, allow POS block header to be canonical 2022-12-10 09:02:30 +07:00
jangko 94a94c5b65 implement better hardfork management 2022-12-02 13:51:42 +07:00
jangko 66439d69ca
unify chain config parser of t8n, bc test, and consensus simulator 2022-11-25 12:26:29 +07:00
jangko 6b28b64395
update hive statistics 2022-11-18 11:58:04 +07:00
jangko 43f4b99a1b
disable NewBlockHashes and NewBlock of eth wire handler after POS transition
fix #1133
2022-11-14 16:17:34 +07:00
Kim De Mey 74a83c1229
Bump nim-eth and remove all now unneeded p2p related imports (#1273) 2022-10-20 10:34:59 +02:00
jangko 16bc2de1cf
update EF test fixtures and fixes to pass all tests 2022-10-03 16:41:32 +07:00
Jacek Sieka c2ed731fa5
eth: adapt to smaller eth_types (#1210) 2022-09-03 20:15:35 +02:00
jangko fe54f93ab6
bump nim eth
better method not implemented message of AbstractChainDB
2022-08-22 20:11:36 +07:00
jangko 11908c78ec
engine api test: handle JsonRpcError for each client rpc call 2022-08-05 09:51:16 +07:00
jangko 21140191da
attempt to fix flaky simulator test 2022-08-02 11:48:59 +07:00
jangko 2c0a8fa26d
update to supported macOS in Github CI 2022-07-30 08:42:44 +07:00
jangko eb972fd598
attemp to fix simulator ci 2022-07-24 15:47:06 +07:00
jangko a48f69a89a
hive: engine simulator add jwt auth test cases 2022-07-18 16:56:44 +07:00
jangko 2b4baff8ec
move block validation from execution payload generator to engine api 2022-07-04 20:36:30 +07:00
jangko ce098bf88f
fix engine api test 2022-07-04 19:32:14 +07:00
Kim De Mey fb12793482
Bump nim-eth and nim-bearssl and accompanying fixes (#1150) 2022-07-04 09:38:02 +02:00
jangko 7f0bc71b65
add invalidMissingAncestorReOrg test case 2022-06-27 20:18:54 +07:00
jangko 4a50b00c37
rpc: change getStorageAt param type to HexDataStr
if using HexQuantityStr, it will fail when given hex string with leading zeros
2022-06-27 17:47:59 +07:00
jangko 71ac6b7de5
engine-api-test: fix transactionReorg test case 2022-06-27 13:01:32 +07:00
jangko b80eca0718
json-rpc: able to query finalized block and safe block header
engine-api:
- store safe block hash and finalized block hash

engine-api test:
- fix test case related to safe block hash and finalized block hash
2022-06-27 11:15:54 +07:00
jangko 8dce76c49f
add engine api test fixtures: PoW chain rlp 2022-06-22 08:10:10 +07:00
jangko d07ef2ee56
fix engine api and angine api test 2022-06-22 08:10:01 +07:00
jangko 69a1000d77
more engine api test 2022-06-15 15:56:45 +07:00
jangko ee13e5fdae
completing 'PrevRandao Opcode Transactions' test case in engine api test 2022-06-15 07:55:14 +07:00
jangko 5bd134e2f0
more engine api tests 2022-06-01 20:32:07 +07:00
jangko 0ad7ffcdbe
hive: proper rpc test 2022-05-31 15:42:01 +07:00
jangko 9843c9428f
fix related to engine api alpha.9 2022-05-29 11:23:03 +07:00
Jordan Hrycaj 62d31d6f1d
Normalise sync handler prototypes (#1087)
* Use type name eth and snap (rather than snap1)

* Prettified snap/eth handler trace messages

* Regrouped sync sources

details:
  Snap storage related sources are moved to common directory.
  Option --new-sync renamed to --snap-sync

also:
  Normalised logging for secondary/non-protocol handlers.

* Merge protocol wrapper files => protocol.nim

details:
  Merge wrapper sync/protocol_ethxx.nim and sync/protocol_snapxx.nim
  into single file snap/protocol.nim

* Comments cosmetics

* Similar start logic for blockchain_sync.nim and sync/snap.nim

* Renamed p2p/blockchain_sync.nim -> sync/fast.nim
2022-05-13 17:30:10 +01:00
Jordan Hrycaj 5467abed8f
update docu (#1074)
why:
  extract_consensus_data.nim has been integrated into consensus_sim.nim
2022-05-06 16:49:48 +01:00
jangko dd339a1fb0
add simulators ci 2022-04-21 12:27:03 +07:00
jangko 6fcd63cb58
modify hive simulators to run in CI 2022-04-21 12:01:18 +07:00
jangko f2f204293e
first step into styleCheck fixes 2022-04-14 08:39:50 +07:00
jangko 1534719f42
fix hive script
new hive features introduces breaking changes we need to address:
- expose engine api port 8551 in docker script
- export enode.sh file to hive-bin folder
- add terminalTotalDifficulty to mapper.jq
2022-04-14 08:24:58 +07:00
jangko 4d126f2461
hive: add ethereum/engine simulator
some test with multiple client still need polishing.
merge test using specially crafted blocks need to be added.
2022-04-13 08:05:58 +07:00
Jordan Hrycaj 046c97f18b
Activate wire protocol eth/66 (#993)
* Activate wire protocol eth/66

and:
 Disentangle protocol_eth66.nim from import sections

why:
  Importing the protocol_eth66 module is not necessary. There is
  no need to know too many details of the underlying wire protocol. All
  that is needed will be exported by blockchain_sync.nim.

* fixes, and rebase

* Update nimbus/p2p/blockchain_sync.nim

Co-authored-by: Kim De Mey <kim.demey@gmail.com>

* Fixes and rebase

Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2022-03-21 17:12:07 +00:00
jangko 25e7debf5e
fix broken graphql_sim 2022-03-08 21:43:20 +07:00
jangko 01980041d5
setup nimbus tx pool for app wide usage
currently the txpool is used at:
- sealer engine
- graphql ethapi
- json rpc ethapi
- ws rpc ethapi
2022-01-25 11:38:49 +07:00
Jamie Lokier df29b98079
Hive: Add Arrow Glacier fork to Hive integration + related fixes
Adds changes to Nimbus Hive support for the new
[Arrow Glacier fork](https://eips.ethereum.org/EIPS/eip-4345).

While here:

- Fix typo in `nimbus.sh`:
  `HIVE_FORK_MUIRGLACIER` => `HIVE_FORK_MUIR_GLACIER` (just a comment).

- Add `muirGlacierBlock` to the JSON generated in `extract_consensus_data.nim`.
  This makes it symmetric with the JSON parsed in `mapper.jq`.

- Removed "At5" network names which are not used by any of the test suite.
  These are `ByzantiumToConstantinopleAt5`, `ConstantinopleFixToIstanbulAt5`
  and `IstanbulToBerlinAt5`.

  The motivation for removing these instead of systematically including all
  possibilities was that I realised `LondonToArrowGlacierAt5` does not appear
  anywhere in the current test suite, even though `ArrowGlacier` does.  As each
  section in the code is rather large already, I thought it cleaner to not add
  this one, and keep only the ones the test suite actually uses.

  This also now better matches the code in `test_blockchain_json.nim`.

- Sorted `HomesteadToDaoAt5` before `HomesteadToEIP150At5` because the DAO
  fork happened earlier than EIP-150 in real life.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-12-10 13:40:54 +00:00
jangko cdfcdfc85b
hive: nodocker rpc simulator skeleton 2021-11-30 16:03:41 +07:00
jangko 37fe6fa32f
fixes hive/consensus_sim command line param
recent changes of confutils based config requires
some command line adjustment.
2021-11-09 17:55:13 +07:00
jangko be91bb349b
hive: fixes readme.md and nimbus.sh
the introduction of new config requires the new command line options
be applied to the commands in nimbus.sh
2021-09-18 17:37:37 +07:00
jangko a3badea928
config: fix new config based on input from jamie and zahary 2021-09-18 17:34:51 +07:00
jangko 69f2a0f95a
config: replace stdlib parseOpt with nim-confutils
fixes #581
2021-09-18 17:34:46 +07:00
jangko 48d497580a
config: remove last instance of getConfiguration usage from nimbus code
this is a preparation for migration to confutils based config
although there is still some getConfiguration usage in tests code
it will be removed after new config arrived
2021-09-08 21:25:14 +07:00
jangko c9cfebfa97
config: rearrange getConfiguration usage
avoid using getConfiguration inside object construction and
replace it with passing suitable param
2021-09-08 08:07:10 +07:00
jangko dda8914fdd
hive: update current state of the tests
-    smoke/clique:            0 pass,     1 fail,     1 total
+    smoke/clique:            1 pass,     0 fail,     1 total
2021-08-30 21:43:34 +07:00
jangko dcbbfd08ae
hive script: move HIVE_CLIQUE_PERIOD processing from nimbus.sh to mapper.jq 2021-08-10 06:51:05 +07:00
jangko cc89d40a41
hive: use new command line options in nimbus.sh
they are:
- ws
- wsapi
- wsbind
- clique-period
- import-key
- engine-signer
2021-08-02 19:41:39 +07:00
Jamie Lokier 00647b373c
Sync: Switch other modules over to the `eth/65` module
Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-27 14:12:41 +01:00
jangko bdbe2a8325
hive: update Current state of the tests
-    ethereum/consensus:  28249 pass,     0 fail, 28249 total (Berlin, without London)
+    ethereum/consensus:  47951 pass,     0 fail, 47951 total London
2021-07-02 16:42:09 +07:00
jangko da1346bdaa
hive: fixes missing baseFeePerGas field in mapper.jq
this is the last bugfix needed to run hive/ethereum/consensus
simulator and pass all London test cases.
2021-07-02 16:36:19 +07:00
jangko 6574d5187d
hive: fix missing comma in hive_integration/mapper.jq
from:
  "berlinBlock": env.HIVE_FORK_BERLIN|to_int
  "londonBlock": env.HIVE_FORK_LONDON|to_int

to:
  "berlinBlock": env.HIVE_FORK_BERLIN|to_int,
  "londonBlock": env.HIVE_FORK_LONDON|to_int
2021-07-01 08:14:11 +07:00
jangko 0245c8e2c7
hive: update extract_consensus_data tool
add London HF changes
2021-06-30 20:43:47 +07:00
jangko 5159ad7aac
preparation for London hard fork
This preparation is needed for subsequent
EIPs included in London.

- Add London to Fork enum
- Block number to fork
- Parsing London fork in chain config
- Prepare gas costs table for London
- Prepare EVM opcode dispatcher for London
- Block rewards for London
- Prepare hive script for London
2021-06-29 07:34:45 +07:00
jangko 36247ed6f4
hive: update current state of the tests
- ethereum/consensus:  28186 pass,    59 fail, 28245 total
+ ethereum/consensus:  28249 pass,     0 fail, 28249 total (Berlin, without London)
2021-06-26 12:13:17 +07:00
Jordan Hrycaj 2d6bf34175
Re-adjust canonical head to parent of block to be inserted (#726)
* Re-adjust canonical head to parent of block to be inserted

why:
  of the failing tests that remain to be solved, 30 of those will succeed
  if the canonical database chain head is cleverly adjusted -- yes, it
  looks like a hack, indeed.

details:
  at the moment, this hack works for the non-hive tests only and is
  triggered by a boolean argument passed on to the chain.persistBlocks()
  method.

* Use parent instead of canonical head for block to be inserted

why:
  side chains need to be inserted typically somewhere before the
  canonical head.

details:
  the previous _hack_ was unnecessary and removed, it was inspired by
  some verification in persistBlocks() which explicitly referenced the
  canonical head (which now might or might not refer to the newly inserted
  header.)

* remove unnecessary code + comment
2021-06-22 17:52:31 +01:00
Jordan Hrycaj cad1b5a678
verify age of uncle's parent (#719)
why:
  parent must be older => check needed for bcFrontierToHomestead test
  cases UncleFromFrontierInHomestead and UnclePopulation
2021-06-18 08:37:59 +01:00
jangko 1836d725e9
update hive stat for ethereum/graphql suite
from: ethereum/graphql:       36 pass,    10 fail,    46 total
to  : ethereum/graphql:       40 pass,     6 fail,    46 total
2021-06-17 18:18:28 +07:00
jangko d0782cdb0d
fixes some of graphql resolver
following recent fixes in upstream hive,
we also update our graphql resolvers
2021-06-17 18:18:28 +07:00
jangko cbed4a368b
hive: update current state of the tests
from: ethereum/sync:           0 pass,     1 fail,     1 total
to  : ethereum/sync:           2 pass,     2 fail,     4 total
2021-05-20 18:47:11 +07:00
jangko 19f343b979
hive: use HIVE_NETWORK_ID to initialize networkId if available
devp2p/eth simulator is using this HIVE_NETWORK_ID to configure client
2021-05-20 14:04:17 +07:00
jangko a0d10f5728
drop PublicNetwork enum usage and replace it with NetworkId
we cannot limit the `--networkid` switch to values available in
`PublicNetwork` enum. it should able to accept very wide range of
custom NetworkId.
2021-05-20 14:04:16 +07:00
jangko d12e7d22bf
update hive stat
additional simulators can run:
- `devp2p/discv4`
- `devp2p/eth`

from:
devp2p/discv4:           0 pass,    14 fail,    14 total
devp2p/eth:              0 pass,     1 fail,     1 total

to:
devp2p/discv4:           3 pass,    11 fail,    14 total
devp2p/eth:              1 pass,     8 fail,     9 total
2021-05-19 17:51:13 +07:00
jangko 2fd2665026
update hive client script
add enode.sh
use HIVE_BOOTNODES flag in nimbus.sh
2021-05-19 17:51:12 +07:00
Jordan Hrycaj 91e24e3581
collected wisdom after working with hive/docker (#669) 2021-05-18 16:38:35 +01:00
jangko 18553156b1
hive: update hive flags usage checklist in nimbus.sh
additional flags we use:
- [x] HIVE_NETWORK_ID              network ID number to use for the eth protocol
- [x] HIVE_LOGLEVEL                client loglevel (0-5)
- [x] HIVE_GRAPHQL_ENABLED         enables graphql on port 8545
2021-05-18 09:31:11 +07:00
jangko cc1034aecf
hive: fixes mapper.jq related to DAO fork
hive always set 'HIVE_FORK_DAO_VOTE' to '1'.

set 'daoForkSupport' to 'true' if only 'HIVE_FORK_DAO_BLOCK'
is set by hive, not depends only on 'HIVE_FORK_DAO_VOTE'.

This is because 'nimbus-eth1' internal is different compared to 'geth'.
2021-05-18 09:23:12 +07:00
jangko b02095a45a
hive: update current state of the tests
additional simulators can be run:
- `ethereum/rpc`
- `smoke/network`
- `smoke/genesis`

number of passess and fails:
from: ethereum/consensus:  27353 pass,   892 fail, 28245 total
to  : ethereum/consensus:  28186 pass,    59 fail, 28245 total

new:
- smoke/genesis:           3 pass,     0 fail,     3 total
- smoke/network:           1 pass,     0 fail,     1 total
- smoke/clique:            0 pass,     1 fail,     1 total
2021-05-18 09:01:11 +07:00
jangko af1537fcab
fixes and add hive simulators written in nim
now we have hive simulators written in nim:
- ethereum/consensus
- ethereum/graphql

Using these simulators, we can debug test cases without have to run
hive or docker.
2021-05-17 20:48:22 +07:00
Jordan Hrycaj d1348824f8
update conversion to start option argument inside hive container (#658)
why:
  not overloading genesis.json by converted version, using new file
  with edited contents
2021-05-14 08:56:21 +01:00
jangko fe92942fa2
update hive/ethereum/consensus stat
from: ethereum/consensus:  16556 pass, 11643 fail, 28199 total
to  : ethereum/consensus:  27353 pass,   892 fail, 28245 total
2021-05-14 08:18:54 +07:00
jangko 911bc0b1c3
update mapper.jq in hive_integration/nimbus
this changes is required due to recent #654.
custom genesis and chain config parser are fixed
and the genesis fields are grouped into "genesis"
field, similar with chain config fields,
they are grouped in "config" field.
2021-05-13 18:09:28 +07:00
jangko 6966324f3b
add our own hive/graphql simulator into hive_integration
now we can run hive/graphql test without using hive or docker.
it's useful for debugging session. but lacking nice result report
like real hive.
2021-05-12 21:21:53 +07:00
jangko 8ec72e86bc
update hive/rpc stat
from: ethereum/rpc:            0 pass,     1 fail,     1 total
to  : ethereum/rpc:            3 pass,    35 fail,    38 total
2021-05-12 12:55:29 +07:00
jangko 86939d9248
fixes `Forks can't be assigned out of order` error in hive
this is a quick fix for #635, allow us to run hive test again
2021-05-12 07:54:37 +07:00
jangko a401f43155
update hive/graphql stat in hive_integration/README.md
from: ethereum/graphql:       34 pass,    12 fail,    46 total
to  : ethereum/graphql:       36 pass,    10 fail,    46 total
2021-05-11 08:50:53 +07:00
jangko d2c0d2f534
add '--nat:0.0.0.0' flag to hive_integration/nimbus/nimbus.sh
trying Jacek suggestion in #591, I added nat setting to
nimbus-eth1 hive shell script. visible difference after adding
this flag is the nat library not complaining about
"the gateway does not support nat-pmp" anymore.

but the slow startup time described in #591 is yet to be measured
again, although I already see improvement when executing
ethereum/consensus category in hive.
2021-05-07 19:48:10 +07:00
jangko 6b639c6f0a
update hive_integration/README.md section 'Current state of the tests'
before: ethereum/graphql:       31 pass,    15 fail,    46 total
after : ethereum/graphql:       34 pass,    12 fail,    46 total
2021-05-07 19:45:59 +07:00
jangko 9e055bd15f
update hive_integration/README.md
from: ethereum/graphql:        1 pass,    46 fail,    46 total
to:   ethereum/graphql:       31 pass,    15 fail,    46 total
2021-05-05 18:43:26 +07:00
jangko 6049073c69
fixes nimbus.sh script for hive integration related to graphql
- add `chain.rlp` import code
- add --graphql and --graphqlbind to enable graphql service
2021-04-30 12:56:19 +07:00
Jamie Lokier 44744596fd
Hive: Attach current nimbus-eth1 test results (summary)
Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-26 18:30:05 +01:00
Jamie Lokier 480e2d4009
Hive: Improvements to documentation
- Explain some of the prerequisites in more detail (Go version).

- Practical issues around Go version and Docker configuration to make it work.

- Make the first example be one that works with nimbus-eth1.
  (Instead of giving an obscure error message that looks like the user's Docker
  setup is broken).

- Rename to `README.md` per convention.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-26 18:27:29 +01:00
jangko bbf5790b5e
fixes docker address and port binding in hive entrypoint script 2021-04-08 08:52:49 +07:00
andri lim 9823b3d132
fixes nimbus hive client code 2021-04-08 08:52:41 +07:00
jangko 2befeceeac
add instructions how to run hive and nimbus in hive 2021-04-08 08:52:40 +07:00
jangko e237e21372
add hive client files 2021-04-08 08:52:40 +07:00
jangko a7344b13d1
add debugging tool related to hive: extract_consensus_data 2021-04-08 08:52:30 +07:00