Commit Graph

116 Commits

Author SHA1 Message Date
jangko e36d2f432a
fix default --key-store location if --data-dir is custom 2022-12-22 11:17:04 +07:00
jangko 94a94c5b65 implement better hardfork management 2022-12-02 13:51:42 +07:00
Jordan Hrycaj 7688148565
Snap sync can start on saved checkpoint (#1327)
* Stop negotiating pivot if peer repeatedly replies w/usesless answers

why:
  There is some fringe condition where a peer replies with legit but
  useless empty headers repetely. This goes on until somebody stops.
  We stop now.

* Rename `missingNodes` => `sickSubTries`

why:
  These (probably missing) nodes represent in reality fully or partially
  missing sub-tries. The top nodes may even exist, e.g. as a shallow
  sub-trie.

also:
  Keep track of account healing on/of by bool variable `accountsHealing`
  controlled in `pivot_helper.execSnapSyncAction()`

* Add `nimbus` option argument `snapCtx` for starting snap recovery (if any)

also:
+ Trigger the recovery (or similar) process from inside the global peer
  worker initialisation `worker.setup()` and not by the `snap.start()`
  function.
+ Have `runPool()` returned a `bool` code to indicate early stop to
  scheduler.

* Can import partial snap sync checkpoint at start

details:
 + Modified what is stored with the checkpoint in `snapdb_pivot.nim`
 + Will be loaded within `runDaemon()` if activated

* Forgot to import total coverage range

why:
  Only the top (or latest) pivot needs coverage but the total coverage
  is the list of all ranges for all pivots -- simply forgotten.
2022-11-25 14:56:42 +00:00
Kim De Mey 74a83c1229
Bump nim-eth and remove all now unneeded p2p related imports (#1273) 2022-10-20 10:34:59 +02:00
jangko c6f35142a8
simple peer manager to handle static peers reconnection
fix #618
2022-08-26 22:07:56 +07:00
jangko a5d4759bfd
enhance net-key command line option to accept random, hex, and path
- previously it only accept hex
- fix #587
2022-07-30 08:46:11 +07:00
Jordan Hrycaj 5d98f68c09
Sync update to work with sepolia reorgs (#1168)
* Error return in `persistBlocks()` on initial `VmState` roblem

why:
  previously threw an exception

* Updated sync mode option

why:
 using enum rather than bool => space for more

* Added sync mode `full`, re-factued legacy sync

also:
  rebased

* Fix typo (crashes `pesistBlocks()` otherwise)

also:
  rebase to master

* Reduce log ticker noise by suppressing duplicate messages

* Clarify staged queue overflow handling

why:
  backtrack/re-org mode in `stageItem()` should be detected by both,
  the global indicator or the work item where it might have moved into.

also:
  rebased
2022-07-21 13:14:41 +01:00
jangko 4721fc7a54
add CORS support for HTTP local services(json-rpc, engine-api, graphql)
There is also a stub for websocket CORS handler. We still need to add
non immediate response management to websock.
2022-07-19 16:12:36 +07:00
Jordan Hrycaj feda401e85
remove unused entry (#1151)
why:
  temporary value, `nim-eth` master did not compile cleanly
2022-07-04 12:47:15 +01:00
Jordan Hrycaj 7293a54e58
Added sepolia specs (#1148)
* Added sepolia specs

* temporarily avoid latest `master` branch from `nim-eth1`

why:
  Currently does not cleanly compile after the `bearssl` split api update.
2022-07-01 21:16:26 +01:00
jangko 00a43234d7
fix enable rpc logic in makeConfig 2022-06-17 07:54:13 +07:00
jangko a37f8b17e2
fix rpc and websocket server config if they share the same port with engine api
also fixes json-rpc-engine-api server and websocket-engine-api server shutdown code,
checking if they actually created or not at startup.

fix #1119
2022-06-13 19:32:19 +07:00
Jordan Hrycaj 62d31d6f1d
Normalise sync handler prototypes (#1087)
* Use type name eth and snap (rather than snap1)

* Prettified snap/eth handler trace messages

* Regrouped sync sources

details:
  Snap storage related sources are moved to common directory.
  Option --new-sync renamed to --snap-sync

also:
  Normalised logging for secondary/non-protocol handlers.

* Merge protocol wrapper files => protocol.nim

details:
  Merge wrapper sync/protocol_ethxx.nim and sync/protocol_snapxx.nim
  into single file snap/protocol.nim

* Comments cosmetics

* Similar start logic for blockchain_sync.nim and sync/snap.nim

* Renamed p2p/blockchain_sync.nim -> sync/fast.nim
2022-05-13 17:30:10 +01:00
Jordan Hrycaj 58e0543920
Squashed snap-sync-preview patch (#1076)
* Squashed snap-sync-preview patch

why:
  Providing end results makes it easier to have an overview.

  Collected patch set comments are available as nimbus/sync/ChangeLog.md
  in chronological order, oldest first.

* Removed some cruft and obsolete imports, normalised logging
2022-05-09 15:04:48 +01:00
jangko f2f204293e
first step into styleCheck fixes 2022-04-14 08:39:50 +07:00
Jordan Hrycaj 737236fd6e
Enable JWT authentication for websockets (#1039)
* Enable JWT authentication for websockets

details:
  Currently, this is optional and only enabled when the jwtsecret option
  is set.

  There is a default mechanism to generate a JWT secret if it is not
  explicitly stated. This mechanism is currently unused.

* Make JWT authentication compulsory for websockets

* Fix unit test entry point + cosmetics

* Update JSON-RPC link

* Improvements as suggested by Mamy
2022-04-06 15:11:13 +01:00
Ștefan Talpalaru 51bc1cf87f
dist: precompiled binaries and Docker images (#1015)
* dist: precompiled binaries and Docker images

The builds are reproducible, the binaries are portable and statically link librocksdb.

This took some patching. Upstream PR: https://github.com/facebook/rocksdb/pull/9752

32-bit ARM is missing as a target because two different GCC versions
fail with an ICE when trying to cross-compile RocksDB. Using Clang
instead is too much trouble for a platform that nobody should be using
anyway.

(Clang doesn't come with its own target headers and libraries, can't be
easily convinced to use the ones from GCC, so it needs an fs image from
a 32-bit ARM distro - at which point I stopped caring).

* CI: disable reproducibility test
2022-03-27 13:21:15 +02:00
jangko 9230e6a024
fixes engine api help message in config.nim
somehow a const string still need `$` to display the
content instead of the identifier itself.
2022-03-08 13:29:34 +07:00
jangko d1dd5d5cad
ttd from command-line takes precedence over ttd from config-file 2022-03-08 11:27:14 +07:00
Jordan Hrycaj 215e9856d3
Jordan/custom network (#962)
* Rearrange/rename test_kintsugu => test_custom_network

why:
  Debug, fix and test more general problems related to running
  nimbus on a custom network.

* Update UInt265/Json parser for --custom-network command line option

why:
  As found out with the Kintsugi configuration, block number and balance
  have the same Nim type which led to misunderstandings. This patch makes
  sure that UInt265 encoded string values "0x11" decodes to 17, and "b"
  and "11" to 11.

* Refactored genesis.toBlock() => genesis.toBlockHeader()

why:
  The function toBlock(g,db) may return different results depending on
  whether the db descriptor argument is nil, or initialised. This is due
  to the db.config data sub-descriptor which may give various outcomes
  for the baseFee field of the genesis header.

  Also, the version where db is non-nil initialised is used internally
  only. So the public rewrite toBlockHeader() that replaces the toBlock()
  function expects a full set of NetworkParams.

* update comments

* Rename toBlockHeader() => toGenesisHeader()

why:
  Polymorphic prototype used for BaseChainDB or NetworkParams argument.
  With a BaseChainDB descriptor argument, the name shall imply that the
  header is generated from the config fields rather than fetched from
  the database.

* Added command line option --static-peers-file

why:
  Handy feature to keep peer nodes in a file, similar to the
  --bootstrap-file option.
2022-02-11 16:28:39 +00:00
Zahary Karadjov 3cbb920406
Migrate to Engine API spec version v1.0.0-alpha.5; More progress towards working M1 2022-01-25 12:52:50 +02:00
Zahary Karadjov 137eb97766 Initial implementation of the merge spec
Includes a simple test harness for the merge interop M1 milestone

This aims to enable connecting nimbus-eth2 to nimbus-eth1 within
the testing protocol described here:

https://github.com/status-im/nimbus-eth2/blob/amphora-merge-interop/docs/interop_merge.md

To execute the work-in-progress test, please run:

In terminal 1:
tests/amphora/launch-nimbus.sh

In terminal 2:
tests/amphora/check-merge-test-vectors.sh
2022-01-24 09:44:39 +02:00
Jamie Lokier 6a7803a9e4
EVMC: Improve `--evm`, remove it in non-EVMC builds, change imports
- Remove the `--evm` option on non-EVMC builds.
  `when` around an option doesn't work with confutils; it fails to compile.
  Workaround that by setting the `ignore` pragma on EVMC-specific options.
  (Thanks @jangko for that new pragma).  I prefer this to a solution which
  moves the whole option's pragma elsewhere, especially if we add more options.

- Improve the help text, so that it shows the standard library extension on
  each target platform (or none if on another platform).

- Undo b3f21bf4 "add missing evmc_enabled conditional compilation in
  evmc_dynamic_loader".  Move the conditional to `nimbus.nim`, and take more
  care there to only use the loader function in EVMC builds.

  It's ok to just not include this EVMC-only module (like some other EVMC
  modules), rather than making the module itself a bit broken: Without this
  change, it references a function that's not imported or linked to, and it
  only links because there is no call sequence reaching that function.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-12-14 15:51:09 +00:00
jangko d5082df5d8
fix config using new features from nim-confutils
- `network` and `discovery` got additional longDesc,
  the help text now become more descriptive.
- `networkId` and `networkParams` now is ignored by confutils
  and become ordinary fields of NimbusConf.
2021-12-12 16:45:58 +07:00
Jamie Lokier cf49b8d4fc
EVMC: Option `--evm`, load third-party EVM as a shared library
This patch adds:

- Load and use a third-party EVM in a shared library, instead of Nimbus EVM.
- New option `--evm` to specify which library to load.
- The library and this loader conforms to the [EVMC]
  (https://evmc.ethereum.org/) 9.x specification.

Any third-party EVM which is compatible with EVMC version 9.x and supports EVM1
contract code will be accepted.  The operating system's shared library format
applies.  These are `.so*` files on Linux, `.dll` files on Windows and `.dylib`
files on Mac.

The alternative EVM can be selected in two ways:

- Nimbus command line option `--evm:<path>`.
- Environment variable `NIMBUS_EVM=<path>`.

The reason for an environment variable is this allows all the test programs to
run with a third-party EVM as well.  Some don't parse command line options.

There are some limitations to be aware of:

- The third-party EVM must use EVMC version 9.x, no other major version.
  EVMC 9.x supports EIP-1559 / London fork and older transactions.

- Nested `*CALL` and `CREATE*` operations don't use the third-party EVM yet.
  These call the built-in Nimbus EVM.  This mixing of different EVMs between
  levels is explicitly allowed in specs, so there is no problem doing it.

- The third-party EVM doesn't need to support precompiles, because those are
  nested calls, which use the built-in Nimbus EVM.

- Third-party EVMs execute contracts correctly, but fail the final `rootHash`
  match.  The reason is that some account state changes, which are correct, are
  currently inside the Nimbus EVM and need to be moved to EVMC host logic.
  *This is a known work in progress*.  The EVM execution itself is fine.

Test results using "evmone" third-party EVM:

- [evmone](https://github.com/ethereum/evmone) has been tested.  Only on
  Linux but it "should" work on Windows and Mac equally well.

- [Version 0.8.1](https://github.com/ethereum/evmone/releases/tag/v0.8.1) was
  used because it is compatible with EVMC 9.x, which is required for the
  EIP-1559 / London fork, which Nimbus supports.  Version 0.8.0 could be used
  but it looks like an important bug was fixed in 0.8.1.

- evmone runs fine and the trace output looks good.  The calls and arguments
  are the same as the built-in Nimbus EVM for tests that have been checked
  manually, except evmone skips some calls that can be safely skipped.

- The final `rootHash` is incorrect, due to the *work in progress* mentioned
  above which is not part of the evmone execution.  Due to this, it's possible
  to try evmone and verify expected behaviours, which also validates our own
  EVMC implementation, but it can't be used as a full substitute yet.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-12-10 16:32:18 +00:00
jangko a3badea928
config: fix new config based on input from jamie and zahary 2021-09-18 17:34:51 +07:00
jangko 69f2a0f95a
config: replace stdlib parseOpt with nim-confutils
fixes #581
2021-09-18 17:34:46 +07:00
jangko 48d497580a
config: remove last instance of getConfiguration usage from nimbus code
this is a preparation for migration to confutils based config
although there is still some getConfiguration usage in tests code
it will be removed after new config arrived
2021-09-08 21:25:14 +07:00
jangko c9cfebfa97
config: rearrange getConfiguration usage
avoid using getConfiguration inside object construction and
replace it with passing suitable param
2021-09-08 08:07:10 +07:00
jangko 9108301eef
config: remove global rng from NimbusConfiguration
move the rng to EthContext
2021-09-07 22:02:29 +07:00
jangko 34972c6cea
config: remove accounts management from NimbusConfiguration
a new AccountsManager and EthContext is created for managing
keystore and accounts

this is a preparation for new config using ConfUtils
2021-09-07 22:02:29 +07:00
jangko 14d2edcb26
chain config preset: add london block number
MainNet     12_965_000
RopstenNet  10_499_401
RinkebyNet   8_897_988
GoerliNet    5_062_605
2021-09-02 12:24:04 +07:00
jangko 77092641b5
add websocket rpc server 2021-08-06 07:32:19 +07:00
jangko 5e87624315
config: copy chainId to networkid if networkid not set in cli
although they are technically different, but in reality,
many networks are using the same id for ChainId dan NetworkId.
in this commit, we set networkid from config file's chainId.
2021-08-06 07:31:02 +07:00
jangko 1da4346295
config: fixes bug networkid parser
previously it mistakenly parse into the `result`
now it correctly parse networkId into res.
2021-08-06 07:31:01 +07:00
jangko c69e57e5d4
config: fix config file parser
- allow clique period and epoch to be configured via config file
- this also activate poaEngine mode
- remove clique period configuration from cli to reduce confusion
- fix #786
2021-08-06 07:30:53 +07:00
jangko c142119487
add new command line options: ws, wsbind, and wsapi
new command line options:
  --ws                    Enable the Websocket JSON-RPC server
  --wsbind:<value>        Set address:port pair(s) (comma-separated) Websocket JSON-RPC server will bind to (default: localhost:8546)
  --wsapi:<value>         Enable specific set of Websocket RPC API from list (comma-separated) (available: eth, debug)

fixes #770
2021-07-31 19:27:13 +07:00
jangko 6015a4e029
add new command line options: clique-period, engine-signer, and import-key
new command line options:
  --clique-period:<value> Enables clique support. value is block time in seconds
  --engine-signer:<value> Enables mining. value is EthAddress in hex
  --import-key:<path>     Import unencrypted 32 bytes hex private key file

fixes #771
2021-07-31 19:16:30 +07:00
Jordan Hrycaj ca07c40a48
Fearture/poa clique tuning (#765)
* Provide API

details:
  API is bundled via clique.nim.

* Set extraValidation as default for PoA chains

why:
  This triggers consensus verification and an update of the list
  of authorised signers. These signers are integral part of the
  PoA block chain.

todo:
  Option argument to control validation for the nimbus binary.

* Fix snapshot state block number

why:
  Using sub-sequence here, so the len() function was wrong.

* Optional start where block verification begins

why:
  Can speed up time building loading initial parts of block chain. For
  PoA, this allows to prove & test that authorised signers can be
  (correctly) calculated starting at any point on the block chain.

todo:
  On Goerli around blocks #193537..#197568, processing time increases
  disproportionally -- needs to be understand

* For Clique test, get old grouping back (7 transactions per log entry)

why:
  Forgot to change back after troubleshooting

* Fix field/function/module-name misunderstanding

why:
  Make compilation work

* Use eth_types.blockHash() rather than utils.hash() in Clique modules

why:
  Prefer lib module

* Dissolve snapshot_misc.nim

details:
  .. into clique_verify.nim (the other source file clique_unused.nim
  is inactive)

* Hide unused AsyncLock in Clique descriptor

details:
  Unused here but was part of the Go reference implementation

* Remove fakeDiff flag from Clique descriptor

details:
  This flag was a kludge in the Go reference implementation used for the
  canonical tests. The tests have been adapted so there is no need for
  the fakeDiff flag and its implementation.

* Not observing minimum distance from epoch sync point

why:
  For compiling PoA state, the go implementation will walk back to the
  epoch header with at least 90000 blocks apart from the current header
  in the absence of other synchronisation points.

  Here just the nearest epoch header is used. The assumption is that all
  the checkpoints before have been vetted already regardless of the
  current branch.

details:
  The behaviour of using the nearest vs the minimum distance epoch is
  controlled by a flag and can be changed at run time.

* Analysing processing time (patch adds some debugging/visualisation support)

why:
  At the first half million blocks of the Goerli replay, blocks on the
  interval #194854..#196224 take exceptionally long to process, but not
  due to PoA processing.

details:
  It turns out that much time is spent in p2p/excecutor.processBlock()
  where the elapsed transaction execution time is significantly greater
  for many of these blocks.

  Between the 1371 blocks #194854..#196224 there are 223 blocks with more
  than 1/2 seconds execution time whereas there are only 4 such blocks
  before and 13 such after this range up to #504192.

* fix debugging symbol in clique_desc (causes CI failing)

* Fixing canonical reference tests

why:
  Two errors were introduced earlier but ovelooked:
   1. "Remove fakeDiff flag .." patch was incomplete
   2. "Not observing minimum distance .." introduced problem w/tests 23/24

details:
  Fixing 2. needed to revert the behaviour by setting the
  applySnapsMinBacklog flag for the Clique descriptor. Also a new
  test was added to lock the new behaviour.

* Remove cruft

why:
  Clique/PoA processing was intended to take place somewhere in
  executor/process_block.processBlock() but was decided later to run
  from chain/persist_block.persistBlock() instead.

* Update API comment

* ditto
2021-07-30 15:06:51 +01:00
jangko 5159ad7aac
preparation for London hard fork
This preparation is needed for subsequent
EIPs included in London.

- Add London to Fork enum
- Block number to fork
- Parsing London fork in chain config
- Prepare gas costs table for London
- Prepare EVM opcode dispatcher for London
- Block rewards for London
- Prepare hive script for London
2021-06-29 07:34:45 +07:00
Jamie Lokier aee0fe39d2
EVM: Remove `vm_types2` everywhere, use common forks list instead
File `vm_types2` is obsolete.  Remove this file and divert all imports to the
common forks list outside the EVM, or in some cases they don't need it anyway.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 15:36:31 +01:00
Jamie Lokier 613f06e61c
Whisper: Remove all the main Whisper code (config, startup, RPC etc)
This is the main patch which removes Whisper code from `nimbus-eth1` code.
It removes all configuration, help, startup, JSON-RPC calls and most types.

Note, there is still Whisper functionality in `nim-eth`.  Also, the "wrapper"
under `wrappers/` isn't dealt with by this change, but it's not built by
default (and might not currently work).

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-01 18:12:48 +01:00
Jamie Lokier 774f697c73
Whisper: Disable Whisper (Shh) protocol by default
This commit turns Whisper off by default, without changing anything else.

So this can be cherry-picked if you just want to disable Whisper without
removing it.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-01 18:12:47 +01:00
jangko 396f3e9909
add missing poaEngine configuration in config.nim
later we will use real engine configuration it if become available to us
2021-05-24 14:35:47 +07:00
jangko b82061bf46
don't mix customBootNodes and bootNodes usage 2021-05-20 14:04:17 +07:00
jangko a0d10f5728
drop PublicNetwork enum usage and replace it with NetworkId
we cannot limit the `--networkid` switch to values available in
`PublicNetwork` enum. it should able to accept very wide range of
custom NetworkId.
2021-05-20 14:04:16 +07:00
jangko f2491e6307
fixes crappy custom genesis and chain config parser
instead of using stdlib/json, now we switch to json_serialization
the result is much tidier code and more robust when parsing
optional fields.

fixes #635
2021-05-13 16:04:08 +07:00
jangko a57ac65c8c
fixes --customnetwork parser
now it can ignore an optional fork e.g. MuirGlacier
2021-05-12 19:05:32 +07:00
jangko 97f4226171
update berlin fork number in config.nim
also update test_forkid because of berlin changes
2021-05-12 17:24:27 +07:00
jangko e6d7d6188c
`processArguments` now can have alternate OptParser instead of fixed one
the `processArguments` now have overloaded proc, one with opt param and one without.
the OptParser now can be passed to `opt` param.
this is useful in scenario where in test code we need to simulate something
without using real command line arguments.
2021-04-30 12:56:19 +07:00