Commit Graph

118 Commits

Author SHA1 Message Date
Jordan Hrycaj 6fb48517ba
Add snap protocol service stub (#1438)
* Cosmetics, update logger `topics`

* Clean up sync/start methods in nimbus

why:
* The `protocols` list selects served (as opposed to sync) protocols only.
* The `SyncMode.Default` object is allocated with the other possible sync
  mode objects.

* Add snap service stub to `nimbus`

* Provide full set of snap response handler stubs

* Bicarb for the latest CI hiccup

why:
  Might be a change in the CI engine for MacOS.
2023-01-20 15:01:29 +00:00
Jordan Hrycaj fda7971aaf
Reorganise eth handlers (#1436)
* Reorganise eth handlers

why:
  Make space for `snap` handlers in a similar fashion.

* fix typo
2023-01-18 15:00:14 +00:00
jangko 56f169b23e
rename Fast Sync to Legacy Sync
fix #1332
2022-12-05 09:42:25 +07:00
jangko 94a94c5b65 implement better hardfork management 2022-12-02 13:51:42 +07:00
Jordan Hrycaj 7688148565
Snap sync can start on saved checkpoint (#1327)
* Stop negotiating pivot if peer repeatedly replies w/usesless answers

why:
  There is some fringe condition where a peer replies with legit but
  useless empty headers repetely. This goes on until somebody stops.
  We stop now.

* Rename `missingNodes` => `sickSubTries`

why:
  These (probably missing) nodes represent in reality fully or partially
  missing sub-tries. The top nodes may even exist, e.g. as a shallow
  sub-trie.

also:
  Keep track of account healing on/of by bool variable `accountsHealing`
  controlled in `pivot_helper.execSnapSyncAction()`

* Add `nimbus` option argument `snapCtx` for starting snap recovery (if any)

also:
+ Trigger the recovery (or similar) process from inside the global peer
  worker initialisation `worker.setup()` and not by the `snap.start()`
  function.
+ Have `runPool()` returned a `bool` code to indicate early stop to
  scheduler.

* Can import partial snap sync checkpoint at start

details:
 + Modified what is stored with the checkpoint in `snapdb_pivot.nim`
 + Will be loaded within `runDaemon()` if activated

* Forgot to import total coverage range

why:
  Only the top (or latest) pivot needs coverage but the total coverage
  is the list of all ranges for all pivots -- simply forgotten.
2022-11-25 14:56:42 +00:00
jangko fffe071f86
eth wire protocol: implement NewBlock and NewBlockHashes handler
and it also do invasive changes to fast sync because
they are tightly related.

fix #673
2022-11-18 01:22:31 +07:00
Jordan Hrycaj 9aa925cf36
Update sync scheduler (#1297)
* Add `stop()` methods to shutdown to shutdown procedure

why:
  Nasty behaviour when hitting Ctrl-C, otherwise

* Add background service to sync scheduler

why:
  The background service will be used for sync data import and recovery
  after restart.

   It is controlled by the sync scheduler for an easy turn/on off API.

also:
  Simplified snap ticker time calc.

* Fix typo
2022-11-14 14:13:00 +00:00
jangko 43f4b99a1b
disable NewBlockHashes and NewBlock of eth wire handler after POS transition
fix #1133
2022-11-14 16:17:34 +07:00
jangko 9dd256cae7
eth wire handlers: implement transactions exchange 2022-11-11 09:24:32 +07:00
jangko 3fa1b012e6
initial wire protocol transformation
rework on the eth wire protocol handlers.
curently still missing 4 handlers implementation.
but the framework is ready for eexpansion.
2022-10-15 19:48:21 +07:00
Jordan Hrycaj eca5882238
Isolating sync action modules (#1249)
* Miscellaneous updates TBC

* Disentangled pivot2 module from snap

why:
  Wrote as template on top of sync so it can be shared by fast and snap
  sync.

* Renamed and relocated pivot sources

* Integrated `best_pivot` module into full and snap sync

why:
  Full sync used an older version of `best_pivot`

* isolating download module from full sync

why;
  might be shared with snap sync at a later stage
2022-09-30 09:22:14 +01:00
jangko c6f35142a8
simple peer manager to handle static peers reconnection
fix #618
2022-08-26 22:07:56 +07:00
Jordan Hrycaj 7d7e26d45f
Experimental bulk loader tests (#1187)
why:
  Rocksdb bulk loading might provide a slight advantage when loading
  larger data sets into the system
2022-08-12 16:42:07 +01:00
Jordan Hrycaj 5f0e89a41e
Snap accounts bulk import preparer (#1183)
* Provided common scheduler API, applied to `full` sync

* Use hexary trie as storage for proofs_db records

also:
 + Store metadata with account for keeping track of account state
 + add iterator over accounts

* Common scheduler API applied to `snap` sync

* Prepare for accounts bulk import

details:
+ Added some ad-hoc checks for proving accounts data received from the
  snap/1 (will be replaced by proper database version when ready)
+ Added code that dumps some of the received snap/1 data into a file
  (turned of by default, see `worker_desc.nim`)
2022-08-04 09:04:30 +01:00
jangko a5d4759bfd
enhance net-key command line option to accept random, hex, and path
- previously it only accept hex
- fix #587
2022-07-30 08:46:11 +07:00
Jordan Hrycaj 5d98f68c09
Sync update to work with sepolia reorgs (#1168)
* Error return in `persistBlocks()` on initial `VmState` roblem

why:
  previously threw an exception

* Updated sync mode option

why:
 using enum rather than bool => space for more

* Added sync mode `full`, re-factued legacy sync

also:
  rebased

* Fix typo (crashes `pesistBlocks()` otherwise)

also:
  rebase to master

* Reduce log ticker noise by suppressing duplicate messages

* Clarify staged queue overflow handling

why:
  backtrack/re-org mode in `stageItem()` should be detected by both,
  the global indicator or the work item where it might have moved into.

also:
  rebased
2022-07-21 13:14:41 +01:00
jangko 4721fc7a54
add CORS support for HTTP local services(json-rpc, engine-api, graphql)
There is also a stub for websocket CORS handler. We still need to add
non immediate response management to websock.
2022-07-19 16:12:36 +07:00
jangko e6938af437
apply jwt auth to rpcHttpServer and update jwt auth of rpcWebsocketServer
fixes #967
2022-07-18 16:56:44 +07:00
jangko 2b4baff8ec
move block validation from execution payload generator to engine api 2022-07-04 20:36:30 +07:00
jangko a37f8b17e2
fix rpc and websocket server config if they share the same port with engine api
also fixes json-rpc-engine-api server and websocket-engine-api server shutdown code,
checking if they actually created or not at startup.

fix #1119
2022-06-13 19:32:19 +07:00
Jordan Hrycaj ba940a5ce7
Snap sync simplify object inheritance (#1098)
* Reorg SnapPeerBase descriptor, notably start/stop flags

details:
  Instead of using three boolean flags startedFetch, stopped, and
  stopThisState a single enum type is used with values SyncRunningOk,
  SyncStopRequest, and SyncStopped.

* Restricting snap to eth66 and later

why:
  Id-tracked request/response wire protocol can handle overlapped
  responses when requests are sent in row.

* Align function names with source code file names

why:
  Easier to reconcile when following the implemented logic.

* Update trace logging (want file locations)

why:
  The macros previously used hid the relevant file location (when
  `chroniclesLineNumbers` turned on.) It rather printed the file
  location of the template that was wrapping `trace`.

* Use KeyedQueue table instead of sequence

why:
  Quick access, easy configuration as LRU or FIFO with max entries
  (currently LRU.)

* Dissolve `SnapPeerEx` object extension into `SnapPeer`

why;
  It is logically cleaner and more obvious not to inherit from
  `SnapPeerBase` but to specify opaque field object references of the
  merged `SnapPeer` object. These can then be locally inherited.

* Dissolve `SnapSyncEx` object extension into `SnapSync`

why;
  It is logically cleaner and more obvious not to inherit from
  `SnapSyncEx` but to specify opaque field object references of
  the `SnapPeer` object. These can then be locally inherited.

  Also, in the re-factored code here the interface descriptor
  `SnapSyncCtx` inherited `SnapSyncEx` which was sub-optimal (OO
  inheritance makes it easier to work with call back functions.)
2022-05-23 17:53:19 +01:00
Jordan Hrycaj 62d31d6f1d
Normalise sync handler prototypes (#1087)
* Use type name eth and snap (rather than snap1)

* Prettified snap/eth handler trace messages

* Regrouped sync sources

details:
  Snap storage related sources are moved to common directory.
  Option --new-sync renamed to --snap-sync

also:
  Normalised logging for secondary/non-protocol handlers.

* Merge protocol wrapper files => protocol.nim

details:
  Merge wrapper sync/protocol_ethxx.nim and sync/protocol_snapxx.nim
  into single file snap/protocol.nim

* Comments cosmetics

* Similar start logic for blockchain_sync.nim and sync/snap.nim

* Renamed p2p/blockchain_sync.nim -> sync/fast.nim
2022-05-13 17:30:10 +01:00
Jordan Hrycaj 569d426ea8
Restore not waiting for fastBlockchainSync() to succeed on start up (#1077)
why:
  Accidentally wrapped into waitFor() directive with reviving jl/sync
  branch.

also:
  Decorate eth/66 and snap/1 protocol trace messages with protocol
  type and version
2022-05-10 09:02:34 +01:00
Jordan Hrycaj 58e0543920
Squashed snap-sync-preview patch (#1076)
* Squashed snap-sync-preview patch

why:
  Providing end results makes it easier to have an overview.

  Collected patch set comments are available as nimbus/sync/ChangeLog.md
  in chronological order, oldest first.

* Removed some cruft and obsolete imports, normalised logging
2022-05-09 15:04:48 +01:00
jangko f2f204293e
first step into styleCheck fixes 2022-04-14 08:39:50 +07:00
jangko 25302f4fd0
non blocking startup
- remove `waitFor` and store async result in Future[T] for future execution
  instead of blocking other services from starting and running.
  This also fixes very long delay Ctrl-C issue. Fixes #585

- always create sealing engine instance even though there is no
  signer. other services such engine api need it.
2022-04-14 07:20:42 +07:00
Jordan Hrycaj 8af5c33ef9
Facilitate http code response on websocket JWT authentication failure (#1043)
* Facilitate http code response on websocket JWT authentication failure

* Update JSON-RPC link
2022-04-07 10:37:35 +01:00
Jordan Hrycaj 737236fd6e
Enable JWT authentication for websockets (#1039)
* Enable JWT authentication for websockets

details:
  Currently, this is optional and only enabled when the jwtsecret option
  is set.

  There is a default mechanism to generate a JWT secret if it is not
  explicitly stated. This mechanism is currently unused.

* Make JWT authentication compulsory for websockets

* Fix unit test entry point + cosmetics

* Update JSON-RPC link

* Improvements as suggested by Mamy
2022-04-06 15:11:13 +01:00
Ștefan Talpalaru 51bc1cf87f
dist: precompiled binaries and Docker images (#1015)
* dist: precompiled binaries and Docker images

The builds are reproducible, the binaries are portable and statically link librocksdb.

This took some patching. Upstream PR: https://github.com/facebook/rocksdb/pull/9752

32-bit ARM is missing as a target because two different GCC versions
fail with an ICE when trying to cross-compile RocksDB. Using Clang
instead is too much trouble for a platform that nobody should be using
anyway.

(Clang doesn't come with its own target headers and libraries, can't be
easily convinced to use the ones from GCC, so it needs an fs image from
a 32-bit ARM distro - at which point I stopped caring).

* CI: disable reproducibility test
2022-03-27 13:21:15 +02:00
jangko 1008e87ec2
disable discovery if maxPeers set to zero 2022-03-25 14:01:18 +07:00
Jordan Hrycaj 046c97f18b
Activate wire protocol eth/66 (#993)
* Activate wire protocol eth/66

and:
 Disentangle protocol_eth66.nim from import sections

why:
  Importing the protocol_eth66 module is not necessary. There is
  no need to know too many details of the underlying wire protocol. All
  that is needed will be exported by blockchain_sync.nim.

* fixes, and rebase

* Update nimbus/p2p/blockchain_sync.nim

Co-authored-by: Kim De Mey <kim.demey@gmail.com>

* Fixes and rebase

Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2022-03-21 17:12:07 +00:00
Kim De Mey b00ac490a9
Bump nim-eth and adjust API to support binding to specific address (#994)
- Fixes also a bug which would not start an EthereumNode properly
when no bootstrap nodes are provided
2022-03-16 09:01:35 +01:00
jangko d74e9cbf49
add warning if engine-api enabled without enabling sealing engine 2022-03-08 14:51:27 +07:00
jangko f782327fcf
reimplement engine API rpc kiln spec v2 2022-03-03 11:28:18 +07:00
Zahary Karadjov 3cbb920406
Migrate to Engine API spec version v1.0.0-alpha.5; More progress towards working M1 2022-01-25 12:52:50 +02:00
jangko 01980041d5
setup nimbus tx pool for app wide usage
currently the txpool is used at:
- sealer engine
- graphql ethapi
- json rpc ethapi
- ws rpc ethapi
2022-01-25 11:38:49 +07:00
Zahary Karadjov 137eb97766 Initial implementation of the merge spec
Includes a simple test harness for the merge interop M1 milestone

This aims to enable connecting nimbus-eth2 to nimbus-eth1 within
the testing protocol described here:

https://github.com/status-im/nimbus-eth2/blob/amphora-merge-interop/docs/interop_merge.md

To execute the work-in-progress test, please run:

In terminal 1:
tests/amphora/launch-nimbus.sh

In terminal 2:
tests/amphora/check-merge-test-vectors.sh
2022-01-24 09:44:39 +02:00
Jamie Lokier 6a7803a9e4
EVMC: Improve `--evm`, remove it in non-EVMC builds, change imports
- Remove the `--evm` option on non-EVMC builds.
  `when` around an option doesn't work with confutils; it fails to compile.
  Workaround that by setting the `ignore` pragma on EVMC-specific options.
  (Thanks @jangko for that new pragma).  I prefer this to a solution which
  moves the whole option's pragma elsewhere, especially if we add more options.

- Improve the help text, so that it shows the standard library extension on
  each target platform (or none if on another platform).

- Undo b3f21bf4 "add missing evmc_enabled conditional compilation in
  evmc_dynamic_loader".  Move the conditional to `nimbus.nim`, and take more
  care there to only use the loader function in EVMC builds.

  It's ok to just not include this EVMC-only module (like some other EVMC
  modules), rather than making the module itself a bit broken: Without this
  change, it references a function that's not imported or linked to, and it
  only links because there is no call sequence reaching that function.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-12-14 15:51:09 +00:00
Jamie Lokier cf49b8d4fc
EVMC: Option `--evm`, load third-party EVM as a shared library
This patch adds:

- Load and use a third-party EVM in a shared library, instead of Nimbus EVM.
- New option `--evm` to specify which library to load.
- The library and this loader conforms to the [EVMC]
  (https://evmc.ethereum.org/) 9.x specification.

Any third-party EVM which is compatible with EVMC version 9.x and supports EVM1
contract code will be accepted.  The operating system's shared library format
applies.  These are `.so*` files on Linux, `.dll` files on Windows and `.dylib`
files on Mac.

The alternative EVM can be selected in two ways:

- Nimbus command line option `--evm:<path>`.
- Environment variable `NIMBUS_EVM=<path>`.

The reason for an environment variable is this allows all the test programs to
run with a third-party EVM as well.  Some don't parse command line options.

There are some limitations to be aware of:

- The third-party EVM must use EVMC version 9.x, no other major version.
  EVMC 9.x supports EIP-1559 / London fork and older transactions.

- Nested `*CALL` and `CREATE*` operations don't use the third-party EVM yet.
  These call the built-in Nimbus EVM.  This mixing of different EVMs between
  levels is explicitly allowed in specs, so there is no problem doing it.

- The third-party EVM doesn't need to support precompiles, because those are
  nested calls, which use the built-in Nimbus EVM.

- Third-party EVMs execute contracts correctly, but fail the final `rootHash`
  match.  The reason is that some account state changes, which are correct, are
  currently inside the Nimbus EVM and need to be moved to EVMC host logic.
  *This is a known work in progress*.  The EVM execution itself is fine.

Test results using "evmone" third-party EVM:

- [evmone](https://github.com/ethereum/evmone) has been tested.  Only on
  Linux but it "should" work on Windows and Mac equally well.

- [Version 0.8.1](https://github.com/ethereum/evmone/releases/tag/v0.8.1) was
  used because it is compatible with EVMC 9.x, which is required for the
  EIP-1559 / London fork, which Nimbus supports.  Version 0.8.0 could be used
  but it looks like an important bug was fixed in 0.8.1.

- evmone runs fine and the trace output looks good.  The calls and arguments
  are the same as the built-in Nimbus EVM for tests that have been checked
  manually, except evmone skips some calls that can be safely skipped.

- The final `rootHash` is incorrect, due to the *work in progress* mentioned
  above which is not part of the evmone execution.  Due to this, it's possible
  to try evmone and verify expected behaviours, which also validates our own
  EVMC implementation, but it can't be used as a full substitute yet.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-12-10 16:32:18 +00:00
jangko f051c2530e
fixes related to nim-json-rpc bump 2021-11-30 14:13:20 +07:00
jangko a3badea928
config: fix new config based on input from jamie and zahary 2021-09-18 17:34:51 +07:00
jangko 69f2a0f95a
config: replace stdlib parseOpt with nim-confutils
fixes #581
2021-09-18 17:34:46 +07:00
jangko 48d497580a
config: remove last instance of getConfiguration usage from nimbus code
this is a preparation for migration to confutils based config
although there is still some getConfiguration usage in tests code
it will be removed after new config arrived
2021-09-08 21:25:14 +07:00
jangko c9cfebfa97
config: rearrange getConfiguration usage
avoid using getConfiguration inside object construction and
replace it with passing suitable param
2021-09-08 08:07:10 +07:00
jangko 9108301eef
config: remove global rng from NimbusConfiguration
move the rng to EthContext
2021-09-07 22:02:29 +07:00
jangko 7dbc44f88c
implement simple PoA sealing engine
the goal of this module is to pass hive/smoke/clique test
and also support for hive/ethereum/rpc test

fixes #801
2021-08-24 14:49:13 +07:00
bmoo b09ad5cacb
code cleanup removed unused imports 2021-08-18 10:35:36 +07:00
jangko 77092641b5
add websocket rpc server 2021-08-06 07:32:19 +07:00
Jordan Hrycaj ca07c40a48
Fearture/poa clique tuning (#765)
* Provide API

details:
  API is bundled via clique.nim.

* Set extraValidation as default for PoA chains

why:
  This triggers consensus verification and an update of the list
  of authorised signers. These signers are integral part of the
  PoA block chain.

todo:
  Option argument to control validation for the nimbus binary.

* Fix snapshot state block number

why:
  Using sub-sequence here, so the len() function was wrong.

* Optional start where block verification begins

why:
  Can speed up time building loading initial parts of block chain. For
  PoA, this allows to prove & test that authorised signers can be
  (correctly) calculated starting at any point on the block chain.

todo:
  On Goerli around blocks #193537..#197568, processing time increases
  disproportionally -- needs to be understand

* For Clique test, get old grouping back (7 transactions per log entry)

why:
  Forgot to change back after troubleshooting

* Fix field/function/module-name misunderstanding

why:
  Make compilation work

* Use eth_types.blockHash() rather than utils.hash() in Clique modules

why:
  Prefer lib module

* Dissolve snapshot_misc.nim

details:
  .. into clique_verify.nim (the other source file clique_unused.nim
  is inactive)

* Hide unused AsyncLock in Clique descriptor

details:
  Unused here but was part of the Go reference implementation

* Remove fakeDiff flag from Clique descriptor

details:
  This flag was a kludge in the Go reference implementation used for the
  canonical tests. The tests have been adapted so there is no need for
  the fakeDiff flag and its implementation.

* Not observing minimum distance from epoch sync point

why:
  For compiling PoA state, the go implementation will walk back to the
  epoch header with at least 90000 blocks apart from the current header
  in the absence of other synchronisation points.

  Here just the nearest epoch header is used. The assumption is that all
  the checkpoints before have been vetted already regardless of the
  current branch.

details:
  The behaviour of using the nearest vs the minimum distance epoch is
  controlled by a flag and can be changed at run time.

* Analysing processing time (patch adds some debugging/visualisation support)

why:
  At the first half million blocks of the Goerli replay, blocks on the
  interval #194854..#196224 take exceptionally long to process, but not
  due to PoA processing.

details:
  It turns out that much time is spent in p2p/excecutor.processBlock()
  where the elapsed transaction execution time is significantly greater
  for many of these blocks.

  Between the 1371 blocks #194854..#196224 there are 223 blocks with more
  than 1/2 seconds execution time whereas there are only 4 such blocks
  before and 13 such after this range up to #504192.

* fix debugging symbol in clique_desc (causes CI failing)

* Fixing canonical reference tests

why:
  Two errors were introduced earlier but ovelooked:
   1. "Remove fakeDiff flag .." patch was incomplete
   2. "Not observing minimum distance .." introduced problem w/tests 23/24

details:
  Fixing 2. needed to revert the behaviour by setting the
  applySnapsMinBacklog flag for the Clique descriptor. Also a new
  test was added to lock the new behaviour.

* Remove cruft

why:
  Clique/PoA processing was intended to take place somewhere in
  executor/process_block.processBlock() but was decided later to run
  from chain/persist_block.persistBlock() instead.

* Update API comment

* ditto
2021-07-30 15:06:51 +01:00
Jamie Lokier c435409292
Sync: Move `blockchain_sync` code and use it with `eth/65`
Move `blockchain_sync.nim` from `nim-eth` to `nimbus-eth1`.

This lets `blockchain_sync` use the `eth/65` protocol to synchronise with more
modern peers than before.

Practically, the effect is the sync process runs more quickly and reliably than
before.  It finds usable peers, and they are up to date.

Note, this is mostly old code, and it mostly performs "classic sync", the
original Ethereum method.  Here's a summary of this code:

- It decides on a blockchain canonical head by sampling a few peers.
- Starting from block 0 (genesis), it downloads each block header and
  block, mostly in order.
- After it downloads each block, it executes the EVM transactions in that block
  and updates state trie from that, before going to the next block.
- This way the database state is updated by EVM executions in block order,
  and new state is persisted to the trie database after each block.

Even though it mentions Geth "fast sync" (comments near end of file), and has
some elements, it isn't really.  The most obvious missing part is this code
_doesn't download a state trie_, it calculates all state from block 0.
Geth "fast sync" has several parts:

1. Find an agreed common chain among several peers to treat as probably secure,
   and a sufficiently long suffix to provide "statistical economic consensus"
   when it is validated.
2. Perform a subset of PoW calculations, skipping forward over a segment to
   verify some of the PoWs according to a pattern in the relevant paper.
3. Download the state trie from the block at the start of that last segment.
4. Execute only the blocks/transactions in that last segment, using the
   downloaded state trie, to fill out the later states and properly validate the
   blocks in the last segment.

Some other issues with `blockchain_sync` code:

- If it ever reaches the head of the chain, it doesn't follow new blocks with
  increasing block numbers, at least not rapidly.
- If the chain undergoes a reorg, this code won't fetch a block number it has
  already fetched, so it can't accept the reorg.  It will end up conflicted
  with peers. This hasn't mattered because the development focus has been on
  the bulk of the catching up process, not the real-time head and reorgs.
- So it probably doesn't work correctly when it gets close to the head due to
  many small reorgs, though it might for subtle reasons.
- Some of the network message handling isn't sufficiently robust, and it
  discards some replies that have valid data according to specification.
- On rare occasions the initial query mapping block hash to number can
  fail (because the peer's state changes).
- It makes some assumptions about the state of peers based on their responses
  which may not be valid (I'm not convinced they are).  The method for working
  out "trusted" peers that agree a common chain prefix is clever.  It compares
  peers by asking each peer if it has the header matching another peer's
  canonical head block by hash.  But it's not clear that merely knowing about a
  block constitutes agreement about the canonical chain.  (If it did, query by
  block number would give the same answer more authoritatively.)

Nonetheless, being able to run this sync process on `eth/65` is useful.

<# interactive rebase in progress; onto 66532e8a

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-27 14:12:53 +01:00