Commit Graph

1229 Commits

Author SHA1 Message Date
Jordan Hrycaj 73b628491d
Clique snapshots reorg (#1169)
* Add persistent snapshot size logging

why:
  Suspecting too much space used

snapshot statistic:
  [..]
  blockNumber=2214912 nSnaps=2236 snapsTotal=1.14m
  blockNumber=2215936 nSnaps=2237 snapsTotal=1.14m
  [..]
  Persisting blocks fromBlock=2216449 toBlock=2216640
  36458496	datadir-nimbus-goerlish/data/nimbus/

* Replace legacy `lru_cache` by `keyed_queue`

why:
  `keyed_queue` generalises `lru_cache`

snapshot statistic:
  [..]
  blockNumber=2234368 nSnaps=2259 snapsTotal=1.15m
  blockNumber=2235392 nSnaps=2260 snapsTotal=1.15m
  [..]
  Persisting blocks fromBlock=2235649 toBlock=2235840
  37627288	datadir-nimbus-goerlish/data/nimbus/

* Increase persistent snapshot storage interval by 300%

snapshot statistic:
      [..]
      blockNumber=2232320 nSnaps=620 snapsTotal=0.30m
      blockNumber=2236416 nSnaps=621 snapsTotal=0.30m
      [..]
      Persisting blocks fromBlock=2237185 toBlock=2237376
      37627288	datadir-nimbus-goerlish/data/nimbus/

* Cull legacy debugging environment for clique

why:
  Chronicles provides a better choice (when properly set up)
2022-07-21 19:16:28 +01:00
Jordan Hrycaj 5d98f68c09
Sync update to work with sepolia reorgs (#1168)
* Error return in `persistBlocks()` on initial `VmState` roblem

why:
  previously threw an exception

* Updated sync mode option

why:
 using enum rather than bool => space for more

* Added sync mode `full`, re-factued legacy sync

also:
  rebased

* Fix typo (crashes `pesistBlocks()` otherwise)

also:
  rebase to master

* Reduce log ticker noise by suppressing duplicate messages

* Clarify staged queue overflow handling

why:
  backtrack/re-org mode in `stageItem()` should be detected by both,
  the global indicator or the work item where it might have moved into.

also:
  rebased
2022-07-21 13:14:41 +01:00
jangko 4721fc7a54
add CORS support for HTTP local services(json-rpc, engine-api, graphql)
There is also a stub for websocket CORS handler. We still need to add
non immediate response management to websock.
2022-07-19 16:12:36 +07:00
jangko e6938af437
apply jwt auth to rpcHttpServer and update jwt auth of rpcWebsocketServer
fixes #967
2022-07-18 16:56:44 +07:00
jangko 6cfaaf5b45
bump json-rpc, secp256k1, blscurve, and accompanying fixes 2022-07-12 20:01:02 +07:00
Jordan Hrycaj 1a5e2e53dd
treat `persistBlocks()` exception as failure return code, unless `Defect` (#1155) 2022-07-06 16:00:51 +01:00
jangko 2b4baff8ec
move block validation from execution payload generator to engine api 2022-07-04 20:36:30 +07:00
jangko 709d8ef255
fix markCanonicalChain bug 2022-07-04 19:32:14 +07:00
jangko 1509dea39d
tidy up engine api 2022-07-04 19:32:13 +07:00
Jordan Hrycaj feda401e85
remove unused entry (#1151)
why:
  temporary value, `nim-eth` master did not compile cleanly
2022-07-04 12:47:15 +01:00
Kim De Mey fb12793482
Bump nim-eth and nim-bearssl and accompanying fixes (#1150) 2022-07-04 09:38:02 +02:00
Jordan Hrycaj 7293a54e58
Added sepolia specs (#1148)
* Added sepolia specs

* temporarily avoid latest `master` branch from `nim-eth1`

why:
  Currently does not cleanly compile after the `bearssl` split api update.
2022-07-01 21:16:26 +01:00
Jordan Hrycaj 134fe26997
Store proved snap accounts (#1145)
* Relocated `IntervalSets` to nim-stew repo

* Accumulate accounts on temporary kv-DB

why:
  Explore the data as returned from snap/1. Will be converted to a
  `eth/db` next.

details:
  Verify and accumulate per/state-root accounts downloaded via snap.

also:
  Some unit tests

* Replace `Table` by `TrieDatabaseRef` for accounts accumulator

* update ticker statistics

details:
  mean/variance based counter update

* allow persistent db for proved accounts

* rebase, and globally activate unit test

* fix statistics
2022-07-01 12:42:17 +01:00
jangko e274b347ae
add haveBlockAndState to BaseChainDB and use it in engine api 2022-06-30 16:11:43 +07:00
KonradStaniec fdb048ed21
Add getLogs implementation (#1143)
* Add eth_getLogs json-rpc endpoint in Fluffy and Nimbus
2022-06-29 17:44:08 +02:00
jangko 4a50b00c37
rpc: change getStorageAt param type to HexDataStr
if using HexQuantityStr, it will fail when given hex string with leading zeros
2022-06-27 17:47:59 +07:00
jangko 3f99ece6c5
rpc: change receipt status field type to HexQuantityStr 2022-06-27 13:01:32 +07:00
jangko b80eca0718
json-rpc: able to query finalized block and safe block header
engine-api:
- store safe block hash and finalized block hash

engine-api test:
- fix test case related to safe block hash and finalized block hash
2022-06-27 11:15:54 +07:00
jangko d07ef2ee56
fix engine api and angine api test 2022-06-22 08:10:01 +07:00
jangko 00a43234d7
fix enable rpc logic in makeConfig 2022-06-17 07:54:13 +07:00
Jordan Hrycaj c123e1eb93
Updated account scheduler (#1124)
* Using `IntervalSet` type data for `LeafRange`

* Updated log ticker

* Update to `eth67`

details:
  Disabled by default, use `ENABLE_LEGACY_ETH66=0` to enable
  No support for `Get/NodeData` dialogue via eth, anymore

* Dissolved fetch/common.nim

details;
  the log/ticker part becomes ticker.nim
  the interval range management is merged into fetch.nim

* Updated account scheduler

why:
  The previous scheduler fetched each account once (for different state
  roots.) The updated scheduler re-calibrates after a change of the state
  root and potentially (until told otherwise) fetches all possible
  accounts.

* Fix `high(P)` fringe cases in `IntervalSet` handling

why:
  The `high(P)` value for a point type `P` cannot be represented with
  half open intervals `[a,b)` for a,b points of `P`. So this single value
  needs extra treatment which was slightly wrong.

* Updated docu/comments

also:
  rebased

* Update scheduler

details:
  Change the `pivot` management when creating new accounts lists. It is
  strictly increasing (and wrapping around) depending on last updated
  accounts list.
2022-06-16 09:58:50 +01:00
jangko 69a1000d77
more engine api test 2022-06-15 15:56:45 +07:00
jangko 77be2f66d2
handle PoA block difficulty during block creation
- in vmState
- in txpool
- in sealing engine

fix #1105
2022-06-15 07:55:00 +07:00
jangko a37f8b17e2
fix rpc and websocket server config if they share the same port with engine api
also fixes json-rpc-engine-api server and websocket-engine-api server shutdown code,
checking if they actually created or not at startup.

fix #1119
2022-06-13 19:32:19 +07:00
Nikolay Mitev c1f7503402 Use asyncSpawn instead of deprecated asyncCheck 2022-06-09 12:51:06 +03:00
Nikolay Mitev 84a89e918a Add nimbus_addPeer rpc call 2022-06-09 12:51:06 +03:00
Jordan Hrycaj ee77d704bc
Non-adjacent intervals set management (#1117)
why:
  extracted from snap as separate helper utility

also:
  should go to `stew`, later on
2022-06-07 15:55:42 +01:00
Jordan Hrycaj 76f6de8059
Normalise snap objects (#1114)
* Fix/recover download flag

why:
  The fetch indicator used to control the data download somehow got
  lost during re-org.

* Updated chronicles/logger topics

* Reorganised run state flags

why:
  The original code used a pair of boolean flags `(stopped,stopThisState)`
  which was translated to three states running, stoppedPending, and
  stopped. It is currently not clear whether collapsing some states was
  correct. So the original logic has been re-stored, albeit wrapped into
  directives like `isStopped()` etc.

also:
  Moving some function bodies in `worker.nim`

* Moved `reply_data.nim` and `validate_trienode.nim` to sub-directory `fetch_trie`

why:
  Only used in `fetch_trie.nim`.

* Move `fetch_*` file and directory objects to `fetch` subdirectory

why:
  Only used in `fetch.nim`

* Added start/stop and/or setup/release methods for all sub-modules

why:
  good housekeeping

also:
  updated getters/setters for ctrl states
  updated trace messages
2022-06-06 14:42:08 +01:00
jangko 5bd134e2f0
more engine api tests 2022-06-01 20:32:07 +07:00
Kim De Mey 6d8b25a5f5
Update header accumulator test to start from Mainnet genesis (#1111) 2022-06-01 15:21:22 +02:00
jangko 9843c9428f
fix related to engine api alpha.9 2022-05-29 11:23:03 +07:00
jangko 8827e04bc0
remove unused coinbase from sealing engine
header.coinbase is handled by txpool, clique, and engine api.
so the sealing engine no need to touch it anymore.
2022-05-29 10:29:56 +07:00
jangko 3f0994f80b
fix clique_sealer: reset block header coinbase
if we are not voting, coinbase should be filled with zero
because other subsystem e.g txpool can produce block header
with non zero coinbase. if that coinbase is one of the signer
and the nonce is zero, that signer will be vote out from
signer list
2022-05-29 10:26:11 +07:00
jangko 8b0d700b45
fix EVM stack.[] bug 2022-05-25 12:04:47 +07:00
Jordan Hrycaj 96bb09457e
Snap sync rename objects (#1099)
* Disentangle `collect` module from `reply_data`

why:
  Now the module visible from `collect` for fetching data is `peer/fetch`
  only.

* Merge `SnapPeerHunt` into `collect`

why:
  This part needs to be known by `collect`, only

* rename collect => worker

* Dissolve `sync_fetch_xdesc` module into `common`

why:
  Descriptor is only used in `common` and `fetch_trie`

* rename `snap/peer` directory => `snap/worker`

* rename `SnapSync` -> `Worker`, `SnapPeer` -> `WorkerBuddy`

* moved `snap/base_desc.nim` -> `snap/worker/worker_desc.nim`

* Unified opaque object ref naming in `worker_desc.nim`

details:
  indicated my inheriting module (exactly one, always)
2022-05-24 09:07:39 +01:00
Jordan Hrycaj ba940a5ce7
Snap sync simplify object inheritance (#1098)
* Reorg SnapPeerBase descriptor, notably start/stop flags

details:
  Instead of using three boolean flags startedFetch, stopped, and
  stopThisState a single enum type is used with values SyncRunningOk,
  SyncStopRequest, and SyncStopped.

* Restricting snap to eth66 and later

why:
  Id-tracked request/response wire protocol can handle overlapped
  responses when requests are sent in row.

* Align function names with source code file names

why:
  Easier to reconcile when following the implemented logic.

* Update trace logging (want file locations)

why:
  The macros previously used hid the relevant file location (when
  `chroniclesLineNumbers` turned on.) It rather printed the file
  location of the template that was wrapping `trace`.

* Use KeyedQueue table instead of sequence

why:
  Quick access, easy configuration as LRU or FIFO with max entries
  (currently LRU.)

* Dissolve `SnapPeerEx` object extension into `SnapPeer`

why;
  It is logically cleaner and more obvious not to inherit from
  `SnapPeerBase` but to specify opaque field object references of the
  merged `SnapPeer` object. These can then be locally inherited.

* Dissolve `SnapSyncEx` object extension into `SnapSync`

why;
  It is logically cleaner and more obvious not to inherit from
  `SnapSyncEx` but to specify opaque field object references of
  the `SnapPeer` object. These can then be locally inherited.

  Also, in the re-factored code here the interface descriptor
  `SnapSyncCtx` inherited `SnapSyncEx` which was sub-optimal (OO
  inheritance makes it easier to work with call back functions.)
2022-05-23 17:53:19 +01:00
Kim De Mey a69b16abff
Cleanup some imports and fix some warnings in the process (#1096) 2022-05-22 22:44:15 +02:00
Jordan Hrycaj 575c69e6ba
Objects inheritance reorg for snap sync (#1091)
* new: time_helper, types

* new: path_desc

* new: base_desc

* Re-organised objects inheritance

why:
  Previous code used macros to instantiate opaque object references. This
  has been re-implemented with OO inheritance based logic.

* Normalised trace macros

* Using distinct types for Hash256 aliases

why:
  Better control of the meaning of the hashes, all or the same format

caveat:
  The protocol handler DSL used by eth66.nim and snap1.nim uses the
  underlying type Hash256 and cannot handle the distinct alias in
  rlp and chronicles/log macros. So Hash256 is used directly (does
  not change readability as the type is clear by parameter names.)
2022-05-17 12:09:49 +01:00
Jordan Hrycaj 62d31d6f1d
Normalise sync handler prototypes (#1087)
* Use type name eth and snap (rather than snap1)

* Prettified snap/eth handler trace messages

* Regrouped sync sources

details:
  Snap storage related sources are moved to common directory.
  Option --new-sync renamed to --snap-sync

also:
  Normalised logging for secondary/non-protocol handlers.

* Merge protocol wrapper files => protocol.nim

details:
  Merge wrapper sync/protocol_ethxx.nim and sync/protocol_snapxx.nim
  into single file snap/protocol.nim

* Comments cosmetics

* Similar start logic for blockchain_sync.nim and sync/snap.nim

* Renamed p2p/blockchain_sync.nim -> sync/fast.nim
2022-05-13 17:30:10 +01:00
Jordan Hrycaj 569d426ea8
Restore not waiting for fastBlockchainSync() to succeed on start up (#1077)
why:
  Accidentally wrapped into waitFor() directive with reviving jl/sync
  branch.

also:
  Decorate eth/66 and snap/1 protocol trace messages with protocol
  type and version
2022-05-10 09:02:34 +01:00
Jordan Hrycaj 58e0543920
Squashed snap-sync-preview patch (#1076)
* Squashed snap-sync-preview patch

why:
  Providing end results makes it easier to have an overview.

  Collected patch set comments are available as nimbus/sync/ChangeLog.md
  in chronological order, oldest first.

* Removed some cruft and obsolete imports, normalised logging
2022-05-09 15:04:48 +01:00
Jordan Hrycaj 3b9f0e3980
Proper Uint256 parser for JSON genesis (#1071)
* Update exception tracinig

* Use lazy JSON parser

why:
  Uint246 type is not directly supported by the JSON serializer

* Json parser update for stringified UInt256 integers

why:
  Now available

* update sub-module branch reference
2022-05-06 09:02:28 +01:00
jangko 6fcd63cb58
modify hive simulators to run in CI 2022-04-21 12:01:18 +07:00
jangko f2f204293e
first step into styleCheck fixes 2022-04-14 08:39:50 +07:00
jangko 38c548f25f
set default clique period to 1 seconds 2022-04-14 07:20:42 +07:00
jangko 25302f4fd0
non blocking startup
- remove `waitFor` and store async result in Future[T] for future execution
  instead of blocking other services from starting and running.
  This also fixes very long delay Ctrl-C issue. Fixes #585

- always create sealing engine instance even though there is no
  signer. other services such engine api need it.
2022-04-14 07:20:42 +07:00
jangko ee8ef7f58f
catch ValueError exception in JwtSharedkey.fromHex 2022-04-14 07:15:03 +07:00
jangko 4d126f2461
hive: add ethereum/engine simulator
some test with multiple client still need polishing.
merge test using specially crafted blocks need to be added.
2022-04-13 08:05:58 +07:00
jangko 83bdde55aa
handle reorg case in sealing engine
if engine api receive forkChoiceUpdated and turn out
reorg happened, sealing engine must update txpool
head.

so in the next iteration when the txpool asked to
produce a fresh block, it will produce it in the
current 'correct' chain and abandon side chain.
2022-04-13 08:05:17 +07:00
Jordan Hrycaj ded38128b5
Jordan/remove unit tests txpool legacy stuff (#1048)
* Update/clarify docu

* Remove legacy stuff for unit tests
2022-04-08 15:05:30 +01:00