Commit Graph

741 Commits

Author SHA1 Message Date
Jordan Hrycaj 0387afb7b1
Remove local block body rlp fix (#1555)
* Remove local block body rlp fix

why:
  Fix moved to `nim-eth` module

* Update nim-eth bumper
2023-04-21 20:08:18 +01:00
jangko 918c1309c8
fix processTransaction's gasLimit 2023-04-19 12:37:18 +07:00
Jordan Hrycaj 0a3bc102eb
Pre functional snap to full sync (#1546)
* Update sync scheduler pool mode

why:
  The pool mode allows to loop over active peers one after another. This
  is ideal for soft re-starting peers. As this is a two tier experience
  (start/stop, setup/release) the loop must be run twice. This is
  controlled by a more rigid re-definition of how to use the `poolMode`
  flag.

* Mitigate RLP serialiser deficiency

why:
  Currently, serialising the `BlockBody` in not conevrtible and need
  to be checked in the `eth` module. Currently a local fix for the
  wire protocol applies. Unit tests will stay (after this local solution
  will have been removed.)

* Code cosmetics and massage

details:
  Main part is `types.toStr()` as a unified function for logging block
  numbers.

* Allow to use a logical genesis replacement (start of history)

why:
  Snap sync will set up an arbitrary pivot at a block number different
  from zero. In fact, the higher the block number the better.

details:
  A non-genesis start of history will currently only affect the score
  values which were derived from the difficulty.

* Provide function to store the snap pivot block header in chain db

why:
  Together with the start of history facility, this allows to proceed
  with full syncing once snap has finished.

details:
  Snap db storage was switched from a sub-tables to the flat chain db.

* Provide database completeness and sanity checker

details:
  For debugging on smaller databases, only

* Implement snap -> full sync switch
2023-04-14 23:28:57 +01:00
Jordan Hrycaj 9facab91cb
Prepare snap client for continuing with full sync (#1534)
* Somewhat tighten error handling

why:
  Zombie state is invoked when the current peer turns out to be useless
  for further communication. While there is a chance to further talk
  to a peer about another topic (aka healing) after some protocol failure,
  it makes no sense to do so after a network problem.

  The latter state is explained bu the `peerDegraded` flag that goes
  together with the `zombie` state flag. A degraded peer is dropped
  immediately.

* Remove `--sync-mode=snapCtx` option, always start snap in recovery mode

why:
  No need for a snap sync option without recovery mode, can be achieved
  by deleting the database.

* Code cosmetics, typos, prettify logging, debugging helper, etc.

* Split off snap sync sub-mode handler into separate modules

details:
  The original `worker.nim` source has become a multiplexer for several
  snap sync sub-modes `full` and `snap`. The source modules of the
  incarnations of a particular sync sub-mode are places into the
  `worker/play` directory.

* Update ticker for snap and full sync logging
2023-04-06 20:42:07 +01:00
Jordan Hrycaj c01045c246
Update snap client account healing (#1521)
* Update nearby/neighbour leaf nodes finder

details:
  Update return error codes so that in the case that there is no more
  leaf node beyond the search direction, the particular error code
  `NearbyBeyondRange` is returned.

* Compile largest interval range containing only this leaf point

why:
  Will be needed in snap sync for adding single leaf nodes to the range
  of already allocated nodes.

* Reorg `hexary_inspect.nim`

why:
 Merged the nodes collecting algorithm for persistent and in-memory
 into a single generic function `hexary_inspect.inspectTrieImpl()`

* Update fetching accounts range failure handling in `rangeFetchAccounts()`

why:
  Rejected response leads now to fetching for another account range. Only
  repeated failures (or all done) terminate the algorithm.

* Update accounts healing

why:
+ Fixed looping over a bogus node response that could not inserted into
  the database. As a solution, these nodes are locally registered and not
  asked for in this download cycle.
+ Sub-optimal handling of interval range for a healed account leaf node.
  Now the maximal range interval containing this node is registered as
  processed which leafs to de-fragementation of the processed (and
  unprocessed) range list(s). So *gap* ranges which are known not to
  cover any account leaf node are not asked for on the network, anymore.
+ Sporadically remove empty interval ranges (if any)

* Update logging, better variable names
2023-03-25 10:44:48 +00:00
Jordan Hrycaj 33023aaf39
Update snap server client test scenario (#1518)
* Redesign snap1 message GetTrieNodes argument prototypes

why:
  A list of sub-objects `seq[SnapTriePath]` is more intuitive to work with
  than an opaque definition `seq[seq[Blob]]` because the inner object
  `SnapTriePath` object has a dedicated inner structure (for how to
  interprete `seq[Blob]`.)

* Collect some public constants into `constants.nim` file

* Reorg `hexary_paths.nim`

why:
+ Collecting nodes following a partial path properly ending at an
  extension node failed to collect this last node.
+ Merged the nodes collecting algorithm for persistent and in-memory
  into a single generic function `hexary_paths.rootPathExtend()`

info:
  Extracted common tasks to `hexary_nodes_helper.nim`

* Implement `StorageRanges` message handler for snap/1 protocol
2023-03-22 20:11:49 +00:00
jangko 34adb86167
fix macro assembler and reenable disabled tests
now macro assembler support merge fork, shanghai, etc without using ugly hack.
also each assembler test have their own `setup` section that can access
`vmState` and perform various custom setup.
2023-03-22 18:18:37 +07:00
jangko 2a3c67b4be
refactor touched accounts, selfdestruct, and log
simplify EVM and delegete those things to accounts cache.
also no more manual state clearing, accounts cache will be
responsible for both collecting touched account and perform
state clearing.
2023-03-21 08:14:06 +07:00
jangko 633f135c72
align accounts cache with EIP158/161
Some nomenclature used in accounts cache are not what described
in EIP158/161, therefore causing confusion and introduce bugs.
Now it should be fixed.
2023-03-18 18:13:06 +07:00
Adam Spitz 3ca6288c3b
Bumped the tests. Withdrawals tests pass now. 2023-03-18 05:17:07 -04:00
Adam Spitz 8d9b2522ea
More on withdrawals (#1508)
* Gwei conversion should use u256 because u64 can overflow.

* Make withdrawals follow the EIP-158 state-clearing rules.

(i.e. Empty accounts should be deleted.)

* Allow the zero address in normalizeNumber.

(Necessary for one of the new withdrawals-related tests.)

* Another fix with a withdrawals-related test.
2023-03-17 14:16:24 -04:00
Jordan Hrycaj 15d0ccb39c
Prepare snap server client test scenario cont4 (#1507)
* Add state root to node steps path register `RPath` or `XPath`

why:
  Typically, the first node in the path register is the state root. There
  are occasions, when the path register is empty (i.e. there are no node
  references) which typically applies to a zero node key.

  In order to find the next node key greater than zero, the state root is
  is needed which is now part of the `RPath` or `XPath` data types.

* Extracted hexary tree debugging functions into separate files

* Update empty path fringe case for left/right node neighbour

why:
  When starting at zero, the node steps path register would be empty. So
  will any path that is before the fist non-zero link of a state root (if
  it is a `Branch` node.)

  The `hexaryNearbyRight()` or `hexaryNearbyLeft()` function required a
  non-zero node steps path register.  Now the first node is to be advanced
  starting at the first state root link if necessary.

* Simplify/reorg neighbour node finder

why:
  There was too mach code repetition for the cases
  * persistent or in-memory database
  * left or right move

details:
  Most algorithms apply for persistent and in-memory alike. Using
  templates/generic functions most of these algorithms can be stated
  in a unified way

* Update storage slots snap/1 handler

details:
  Minor changes to be more debugging friendly.

* Fix detection of full database for snap sync

* Docu: Snap sync test & debugging scenario
2023-03-17 14:46:50 +00:00
Adam Spitz ec2bd4a9c5
More work on withdrawals (#1503)
* Gwei conversion should use u256 because u64 can overflow.

* Make withdrawals follow the EIP-158 state-clearing rules.

(i.e. Empty accounts should be deleted.)

* Allow the zero address in normalizeNumber.

(Necessary for one of the new withdrawals-related tests.)
2023-03-16 16:34:47 -04:00
Adam Spitz 11771f8444
Recreating some of the old stateless code that had bitrotted. (#1493)
Note that we're not really using it yet; I'm just refactoring and
adding new code, working in small steps.
2023-03-10 17:16:42 -05:00
Adam Spitz c921135e2e
Refactored to introduce distinct types AccountsTrie and StorageTrie. (#1490)
This is in preparation for working on stateless mode; it's useful to
have a clearer and more type-safe interface for accessing accounts
versus slots.
2023-03-10 13:42:37 -05:00
Jordan Hrycaj 2f7f2dba2d
Prepare snap server client test scenario cont3 (#1491)
* Handle last/all node(s) proof conditions at leaf node extractor

detail:
  Flag whether the maximum extracted node is the last one in database
  No proof needed if the full tree was extracted

* Clean up some helpers & definitions

details:
  Move entities to more plausible locations, e.g. `Account` object need
  not be dealt with in the range extractor as it applies to any kind of
  leaf data.

* Fix next/prev database walk fringe condition

details:
  First check needed might be for a leaf node which was done too late.

* Homogenise snap/1 protocol function prototypes

why:
  The range arguments `origin` and `limit` data types differed in various
  function prototypes (`Hash256` vs. `openArray[byte]`.)

* Implement `GetStorageRange` handler

* Implement server timeout for leaf node retrieval

why:
  This feature leaves control on the server for probably costly action
  invoked by the network

* Implement maximal reply size for snap service

why:
  This feature leaves control on the server for probably costly action
  invoked by the network.
2023-03-10 17:10:30 +00:00
Adam Spitz d8a1adacaa
More work on withdrawals (#1482)
* Part of EIP-4895: add withdrawals processing to block processing.

* Refactoring: extracted the engine API handler bodies into procs.

Intending to implement the V2 versions next. (I need the bodies to be
in separate procs so that multiple versions can use them.)

* Working on Engine API changes for Shanghai.

* Updated nim-web3, resolved ambiguity in Hash256 type.

* Updated nim-eth3 to point to master, now that I've merged that.

* I'm confused about what's going on with engine_client.

But let's try resolving this Hash256 ambiguity.

* Still trying to fix this conflict with the Hash256 types.

* Does this work now that nimbus-eth2 has been updated?

* Corrected blockValue in getPayload responses back to UInt256.

c834f67a37

* Working on getting the withdrawals-related tests to pass.

* Fixing more of those Hash256 ambiguities.

(I'm not sure why the nim-web3 library introduced a conflicting type
named Hash256, but right now I just want to get this code to compile again.)

* Bumped a couple of libraries to fix some error messages.

* Needed to get "make fluffy-tools" to pass, too.

* Getting "make nimbus_verified_proxy" to build.
2023-03-09 18:40:55 -05:00
Jordan Hrycaj fe3a6d67c6
Prepare snap server client test scenario cont2 (#1487)
* Clean up some function prototypes

why:
  Simplify polymorphic prototype variances for easier maintenance.

* Fix fringe condition crash when importing bogus RLP node

why:
  Accessing non-list RLP entry as a list causes `Defect`

* Fix left boundary proof at range extractor

why:
  Was insufficient. The main problem was that there was no unit test for
  the validity of the generated left boundary.

* Handle incomplete left boundary proofs early

why:
  Attempt to do it later leads to overly complex code in order to prevent
  looping when the same peer repeats to send the same incomplete proof.

  Contrary, gaps in the leaf sequence can be handled gracefully with
  registering the gaps

* Implement a manual pivot setup mechanism for snap sync

why:
  For a test scenario it is convenient to set the pivot to something
  lower than the beacon header from the consensus layer. This does not
  need rely on any RPC mechanism.

details:
  The file containing the pivot specs is specified by the
  `--sync-ctrl-file` option. It is regularly parsed for updates.

* Fix calculation error

why:
  Prevent from calculating negative square root
2023-03-07 14:23:22 +00:00
Jordan Hrycaj 10ad7867e4
Prepare snap server client test scenario cont1 (#1485)
* Renaming androgynous sub-object names according to where they belong

why:
  These objects are not explicitly dealt with. They give meaning to
  some generic wrapper objects. Naming them after their origin may
  help troubleshooting.

* Redefine proof nodes list data type for `snap/1` wire protocol

why:
  The current specification suffered from the fact that the basic data
  type for a proof node is an RLP encoded hexary node. This slightly
  confused the encoding/decoding magic.

details:
  This is the second attempt, now wrapping the `seq[Blob]` into a
  wrapper object of `seq[SnapProof]` for a distinct alias sequence.

  In the previous attempt, `SnapProof` was a wrapper object holding the
  `Blob` with magic applied to the `seq[]`. This needed the `append`
  mixin to strip the outer wrapper that was applied to the `Blob` already
  when it was passed as argument.

* Fix some prototype inconsistency

why:
  For easy reading, `getAccountRange()` handler return code should
  resemble the `accoundRange()` anruments prototype.
2023-03-03 20:01:59 +00:00
Jordan Hrycaj bf53226c2c
Minor updates for testing and cosmetics (#1476)
* Fix locked database file annoyance with unit tests on Windows

why:
  Need to clean up old files first from previous session as files remain
  locked despite closing of database.

* Fix initialisation order

detail:
  Apparently this has no real effect as the ticker is only initialised
  here but started later.

  This possible bug has been in all for a while and was running with the
  previous compiler and libraries.

* Better naming of data fields for sync descriptors

details:
* BuddyRef[S,W]: buddy.data -> buddy.only
* CtxRef[S]: ctx.data -> ctx.pool
2023-02-23 13:13:02 +00:00
jangko f5c12062bd
bump submodules to reduce compiler warnings 2023-02-22 12:21:41 +07:00
Adam Spitz fad3ed64cf
Time based forking (#1465)
* Refactoring in preparation for time-based forking.

* Timestamp-based hard-fork-transition.

* Workaround SideEffect issue / compiler bug for both failing locations in Portal history code

---------

Co-authored-by: kdeme <kim.demey@gmail.com>
2023-02-16 12:40:07 +01:00
Jordan Hrycaj b793f0de8d
Snap sync extractor and sub range proofs cont1 (#1468)
* Redefine `seq[Blob]` => `seq[SnapProof]` for `snap/1` protocol

why:
  Proof nodes are traded as `Blob` type items rather than Nim objects. So
  the RLP transcoder must not extra wrap proofs which are of type
  seq[Blob]. Without custom encoding one would produce a
  `list(blob(item1), blob(item2) ..)` instead of `list(item1, item2 ..)`.

* Limit leaf extractor by RLP size rather than number of items

why:
  To be used serving `snap/1` requests, the result of function
  `hexaryRangeLeafsProof()` is limited by the maximal space
  needed to serialise the result which will be part of the
  `snap/1` repsonse.

* Let the range extractor `hexaryRangeLeafsProof()` return RLP list sizes

why:
  When collecting accounts, the size oft the accounts list when encoded
  as RLP is continually updated. So the summed up value is available
  anyway. For the proof nodes list, there are not many (~ 10) so summing
  up is not expensive here.
2023-02-15 10:14:40 +00:00
Jordan Hrycaj df1217b7ca
Silence compiler gossip after nim upgrade cont3 (#1466)
* Removed some Windows specific unit test annoyances

details:
+ Short put()/get() cycles on persistent database have a race condition
  with vendor rocksdb. On a specific (and slow) qemu/win7 a 50ms `sleep()`
  in between will mostly do the job (i.e. unless heavy CPU load.) This
  issue was not observed on github/ci.
+ Removed annoyances when qemu/Win7 keeps the rocksdb database files
  locked even after closing the db. The problem is solved by strictly
  using fresh names for each test. No assumption made to be able to
  properly clean up. This issue was not observed on github/ci.

* Silence some compiler gossip -- part 7, misc/non(sync or graphql)

details:
  Adding some missing exception annotation
2023-02-14 20:27:17 +00:00
jangko 28129d1df2
evm: fix premature position truncation of byte op 2023-02-13 21:20:52 +07:00
Jordan Hrycaj c2fc46a99a
Snap sync extractor test sub range proofs (#1460)
* Unit tests to verify calculations based on hard coded constants

why:
  Sizes of RLP encoded objects are available at run time only.

* Changed argument order for `hexaryRangeLeafsProof()` prototype

why:
  Better to read as a stand-alone function (arguments were optimised
  for functional pipelines)

* Run sub-range proof tests for extracted ranges
2023-02-02 13:27:09 +00:00
jangko 0f065e91b5
bump submodules to further reduce compiler warnings 2023-02-02 09:29:02 +07:00
Jordan Hrycaj 6ca6bcd96f
Snap sync fix trie interpolation fringe condition (#1457)
* Cosmetics

details:
+ Update doc generator
+ Fix key type representation in `hexary_desc` for debugging
+ Redefine `isImportOk()` as template for better `check()` line reporting

* Fix fringe condition when interpolating Merkle-Patricia tries

details:
  Small change with profound effect fixing some pathological condition
  that haunted the unit test set on large data sers. There is still one
  condition left which might well be due to an incomplete data set.

* Unit test proof nodes for node range extractor

* Unit tests to run on full extraction set

why:
  Left over from troubleshooting, range length was only 5
2023-02-01 18:56:06 +00:00
Kim De Mey 73e93f1f11
Reduce Nim 1.6 compiler warnings & hints
* Reduce Nim 1.6 compiler warnings/hints for Fluffy and Nimbus proxy

Mostly raises Defect removals, TaintedString removal and some
unnecessary imports.

Also updating the copyright years alongside.

* Further reduce Nim 1.6 compiler warnings/hints for Nimbus
2023-01-31 13:38:08 +01:00
Jordan Hrycaj 6b9f3c9ac5
Silence compiler gossip after nim upgrade cont1 (#1455)
* Silence some compiler gossip -- part 5, common

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Silence some compiler gossip -- part 6, db, rpc, utils

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Silence some compiler gossip -- part 7, randomly collected source files

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Silence some compiler gossip -- part 8, assorted tests

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Clique update

why:
  More impossible exceptions (undoes temporary fix from previous PR)
2023-01-31 01:32:17 +00:00
Jordan Hrycaj 197d2b16dd
Snap sync interval range extractor (#1449)
* Update comments and test noise

* Fix boundary proofs

why:
  Where neither used in production, nor unit tested. For production, other
  methods apply to test leaf range integrity directly based of the proof
  nodes.

* Added `hexary_range()`: interval range + proof extractor

details:
+ Will be used for `snap/1` protocol handler
+ Unit tests added (also for testing left boundary proof)

todo:
  Need to verify completeness of proof nodes

* Reduce some nim 1.6 compiler noise

* Stop unit test gossip for ci tests
2023-01-30 17:50:58 +00:00
Kim De Mey 8523d8b40e
Bump submodules and fix related issues (#1448)
* Updated to the latest nim-eth, nim-rocksdb, nim-web3

* Bump nimbus-eth2 module and fix related issues

Temporarily disabling Portal beacon light client network as it is
a lot of copy pasted code that did not yet take into account
forks. This will require a bigger rework and was not yet tested
in an actual network anyhow.

* More nimbus fixes after module bumps

---------

Co-authored-by: Adam Spitz <adamspitz@status.im>
Co-authored-by: jangko <jangko128@gmail.com>
2023-01-27 15:57:48 +01:00
Kim De Mey a669b51ec5
Bump Nim to 1.6 and resolve the related issues (#1445)
Two unresolved items currently:
- Three tests that are temporarily disabled as they fail in the
macro_assembler code, which seems to be due to an ambigious
identifier Stop (Ops and chronos ServerCommand enum).
- i386 CI disabled as it fails at Nim compilation already. Failed
tests where already ignored for this target.
2023-01-26 13:37:19 +01:00
Jordan Hrycaj 1b441a485b
Replace coded unit test PRNG for Clique test address generation (#1446)
why:
  Clique relies on the even/odd position of an address after sorting. For
  address generation, the Nim PRNG was used which seems to have changed
  with Nim 1.6.11 (Linux, Windoes only.)

  As a replace, the Posix.1-2001 example (two-liner calculation) generator
  is used.
2023-01-24 14:52:02 +00:00
Jordan Hrycaj e093fa452d
Declutter snap sync unit tests (#1444)
* Extracted RocksDB timing unit tests into separate file

why:
  make space for more in main module :)

* Extracted `inspectionRunner()` unit tests into separate file

why:
  make space for more in main module :)

* Extracted `storagesRunner()` unit tests into separate file

why:
  make space for more in main module :)

* Extracted pivot checkpoint store/retrieval unit tests into separate file

why:
  make space for more in main module :)

* Extract helper functions into separate source file

* Extracted account import unit tests into separate file

why:
  make space for more in main module :)

* Rename `test_decompose()` => `test_NodeRangeDecompose()`

why:
  There will be more functions with `test_NodeRange` prefix.
2023-01-23 16:09:12 +00:00
Kim De Mey 9b1193c402
Revert "Updated to the latest nim-eth, nim-rocksdb, nim-web3. (#1441)" (#1443)
This reverts commit 1c1a3352b3.
2023-01-23 10:50:17 +01:00
Adam Spitz 1c1a3352b3
Updated to the latest nim-eth, nim-rocksdb, nim-web3. (#1441) 2023-01-21 15:04:22 +01:00
jangko 6814140c63
unify coinbase state clearing of gst, evmstate, and t8n
it is troublesome if we have to fix it in three places
in case of a bug, it also reduce code duplication.
2023-01-14 17:17:55 +07:00
jangko e2601e7c54
bump eth_tests 2023-01-14 15:56:08 +07:00
jangko 192646ca8a
fix test_blockchain_json lastBlockHash comparison 2023-01-14 15:56:08 +07:00
jangko a232c7eb1d
reduce code duplication in multiple test runners 2023-01-13 13:04:55 +07:00
jangko e36d2f432a
fix default --key-store location if --data-dir is custom 2022-12-22 11:17:04 +07:00
Etan Kissling bab49dcd7e update for `excessDataGas` type chaneg 2022-12-14 11:04:13 +02:00
Etan Kissling 22338b7870 bump `nim-eth` for `eip4844` support
The `BlockHeader` structure in `nim-eth` was updated with support for
EIP-4844 (danksharding). To enable the `nim-eth` bump, the ingress of
`BlockHeader` structures has been hardened to reject headers that have
the new `excessDataGas` field until proper EIP4844 support exists.
https://github.com/status-im/nim-eth/pull/570
2022-12-14 11:04:13 +02:00
jangko 13e5b1a76b
fix persistHeaderToDb, allow POS block header to be canonical 2022-12-10 09:02:30 +07:00
jangko 7bab58731b
simplify test_blockchain_json a bit 2022-12-09 15:26:53 +07:00
jangko b16b35caea
cleanup unused txpool unused code 2022-12-09 11:26:14 +07:00
Jordan Hrycaj 3766eddf5a
Some updates to the envelope module (#1353)
details:
+ Add detailed error return codes
+ Remove cruft
+ Some prototype wrappers
2022-12-06 20:13:31 +00:00
Jordan Hrycaj 85de03fd6e
Rename and update dismantle => hexaryEnvelopeDecompose() (#1351)
* Rename and update dismantle => hexaryEnvelopeDecompose()

why:
+ As for naming, a positive connotation is prefered
+ The unit tests were really insufficient
+ The function result was wrong on a few boundry conditions

detail:
+ Extracted the function from `hexary_paths.nim` and re-implemented
  it together with other envelope functions => `hexary_envelope.nim`
+ Re-wrote docu for `hexaryEnvelopeDecompose()`

* Relaxed right condition for `hexaryEnvelopeDecompose()` range argument

why;
  Previously, the right point of the argument interval had to be a path
  to an allocated leaf node. While this is typically a given for accounts,
  it is easier to require an arbitrary range of paths (or keys) with
  the requirement of a `boundary proof` for left and right (i.e. enough
  nodes in the database to find the end points.)

also:
  Bug fixes for related functions (typos, missing conditions etc.)

* Add missing unit tests include file
2022-12-06 17:35:56 +00:00
jangko a26a9f9ece
fix txpool: using consensus rule to prepare header 2022-12-06 16:43:00 +07:00