Commit Graph

75 Commits

Author SHA1 Message Date
Jordan Hrycaj de2c13e136
Update snap offline tests (#1199)
* Re-implemented `hexaryFollow()` in a more general fashion

details:
+ New name for re-implemented `hexaryFollow()` is `hexaryPath()`
+ Renamed `rTreeFollow()` as `hexaryPath()`

why:
  Returning similarly organised structures, the results of the
  `hexaryPath()` functions become comparable when running over
  the persistent and the in-memory databases.

* Added traversal functionality for persistent ChainDB

* Using `Account` values as re-packed Blob

* Repack samples as compressed data files

* Produce test data

details:
+ Can force pivot state root switch after minimal coverage.
+ For emulating certain network behaviour, downloading accounts stops for
  a particular pivot state root if 30% (some static number) coverage is
  reached. Following accounts are downloaded for a later pivot state root.
2022-08-24 14:44:18 +01:00
Jordan Hrycaj 73b628491d
Clique snapshots reorg (#1169)
* Add persistent snapshot size logging

why:
  Suspecting too much space used

snapshot statistic:
  [..]
  blockNumber=2214912 nSnaps=2236 snapsTotal=1.14m
  blockNumber=2215936 nSnaps=2237 snapsTotal=1.14m
  [..]
  Persisting blocks fromBlock=2216449 toBlock=2216640
  36458496	datadir-nimbus-goerlish/data/nimbus/

* Replace legacy `lru_cache` by `keyed_queue`

why:
  `keyed_queue` generalises `lru_cache`

snapshot statistic:
  [..]
  blockNumber=2234368 nSnaps=2259 snapsTotal=1.15m
  blockNumber=2235392 nSnaps=2260 snapsTotal=1.15m
  [..]
  Persisting blocks fromBlock=2235649 toBlock=2235840
  37627288	datadir-nimbus-goerlish/data/nimbus/

* Increase persistent snapshot storage interval by 300%

snapshot statistic:
      [..]
      blockNumber=2232320 nSnaps=620 snapsTotal=0.30m
      blockNumber=2236416 nSnaps=621 snapsTotal=0.30m
      [..]
      Persisting blocks fromBlock=2237185 toBlock=2237376
      37627288	datadir-nimbus-goerlish/data/nimbus/

* Cull legacy debugging environment for clique

why:
  Chronicles provides a better choice (when properly set up)
2022-07-21 19:16:28 +01:00
Kim De Mey fb12793482
Bump nim-eth and nim-bearssl and accompanying fixes (#1150) 2022-07-04 09:38:02 +02:00
Jordan Hrycaj 134fe26997
Store proved snap accounts (#1145)
* Relocated `IntervalSets` to nim-stew repo

* Accumulate accounts on temporary kv-DB

why:
  Explore the data as returned from snap/1. Will be converted to a
  `eth/db` next.

details:
  Verify and accumulate per/state-root accounts downloaded via snap.

also:
  Some unit tests

* Replace `Table` by `TrieDatabaseRef` for accounts accumulator

* update ticker statistics

details:
  mean/variance based counter update

* allow persistent db for proved accounts

* rebase, and globally activate unit test

* fix statistics
2022-07-01 12:42:17 +01:00
Jordan Hrycaj c123e1eb93
Updated account scheduler (#1124)
* Using `IntervalSet` type data for `LeafRange`

* Updated log ticker

* Update to `eth67`

details:
  Disabled by default, use `ENABLE_LEGACY_ETH66=0` to enable
  No support for `Get/NodeData` dialogue via eth, anymore

* Dissolved fetch/common.nim

details;
  the log/ticker part becomes ticker.nim
  the interval range management is merged into fetch.nim

* Updated account scheduler

why:
  The previous scheduler fetched each account once (for different state
  roots.) The updated scheduler re-calibrates after a change of the state
  root and potentially (until told otherwise) fetches all possible
  accounts.

* Fix `high(P)` fringe cases in `IntervalSet` handling

why:
  The `high(P)` value for a point type `P` cannot be represented with
  half open intervals `[a,b)` for a,b points of `P`. So this single value
  needs extra treatment which was slightly wrong.

* Updated docu/comments

also:
  rebased

* Update scheduler

details:
  Change the `pivot` management when creating new accounts lists. It is
  strictly increasing (and wrapping around) depending on last updated
  accounts list.
2022-06-16 09:58:50 +01:00
jangko 77be2f66d2
handle PoA block difficulty during block creation
- in vmState
- in txpool
- in sealing engine

fix #1105
2022-06-15 07:55:00 +07:00
Jordan Hrycaj ee77d704bc
Non-adjacent intervals set management (#1117)
why:
  extracted from snap as separate helper utility

also:
  should go to `stew`, later on
2022-06-07 15:55:42 +01:00
jangko f2f204293e
first step into styleCheck fixes 2022-04-14 08:39:50 +07:00
Jordan Hrycaj ded38128b5
Jordan/remove unit tests txpool legacy stuff (#1048)
* Update/clarify docu

* Remove legacy stuff for unit tests
2022-04-08 15:05:30 +01:00
Jordan Hrycaj fd6fcc6cc4
Jordan/dissolve txpool jobs queue (#1047)
* Prepare unit tests for running without tx-pool job queue

why:
  Most of the job queue logic can be emulated. This adapts to a few
  pathological test cases.

* Replace tx-pool job queue logic with in-place actions

why:
  This additional execution layer is not needed, anymore which has been
  learned from working with the integration/hive tests.

details:
  Execution of add or deletion jobs are executed in-place. Some actions
  -- as in smartHead() -- have been combined for facilitating the correct
  order actions

* Update production functions, remove txpool legacy stuff
2022-04-08 09:38:47 +01:00
jangko fdbabf0b80
unify eip1559TxNormalization
result.gasPrice = baseFee + min(result.maxPriorityFee, result.maxFee - baseFee)
cannot be simplified into
result.gasPrice = min(result.maxPriorityFee + baseFee, result.maxFee)
the expression is not commutative
2022-04-05 17:22:46 +07:00
Jordan Hrycaj f11b1a8d8e
Fix typo in tx_head.nim delta calculator
why:
  Causes havoc in most bit fringe cases.

details:
  When setting the head forward, the delta was wrongly registered from
  the static "left" end (which limits the loop) rather than the moving
  "right" end.
2022-04-04 17:07:48 +01:00
jangko 2746f52cc8
txPool: add PoS feeRecipient setter 2022-04-04 09:01:39 +07:00
jangko ef1a68867b
txpool: fix wrong baseFee usage in runTx and runTxCommit 2022-04-03 13:23:10 +07:00
jangko 00e2ec7d38
txpool: fix eip1559TxNormalization bug 2022-04-03 13:23:10 +07:00
jangko 61d7a9acea
fix txpool prevRandao setter 2022-04-03 13:23:10 +07:00
Jordan Hrycaj 84ff179cd9
Jordan/fix txs sort order (#1025)
* Fix database sort order for local txs

why:
  For convenience, packed txs were stored in the block sorted by
  rank->nonce. Using local accounts, the greedy grabber uses the sort
  order (local,non-local)->rank->nonce which leads to a wrong calculation
  of the txRoot.

* Housekeeping

details:
  Replaced a couple of local eip1559TxNormalization() functions by a
  single public
2022-03-31 17:17:22 +01:00
Jordan Hrycaj 2eb3f414d8
Support for local accounts (#1019)
* Support for local accounts

why:
  Accounts tagged local will be packed with priority over untagged
  accounts

* Added functions for queuing txs and simultaneously setting account locality

why:
  Might be a popular task, in particular for unconditionally adding txs to
  a local (aka prioritised account) via "xp.addLocal(tx,true)"

caveat:
  Untested yet

* fix typo

* backup

* No baseFee for pre-London tx in verifier

why:
  The packer would wrongly discard valid legacy txs.
2022-03-31 09:21:36 +01:00
Jordan Hrycaj 534fb528a4
Update terminal total difficulty handling (#992)
why:
  Testing against a replay unit test for Devnet4 made it necessary to
  adjust the TTD handling. Without updated, importing fails at block #5646
  which is the parent of the terminal PoW block. Similar considerations
  apply for Devnet5 and Kiln.
2022-03-15 17:21:41 +00:00
jangko 667cb6d750
EIP 4399 changes: random -> prevRandao 2022-02-27 14:37:35 +07:00
jangko ec59c691aa
EIP-3675: disable reward for coinbase and uncles miner 2022-02-08 20:23:41 +07:00
jangko 28cdfcaf6b
fix EIP-4399 'random' opcode
- fix previous implementation of EIP-4399
- now `random` opcode can be used with evmc_enabled
2022-02-08 20:23:40 +07:00
jangko 71aa7e4b5c EIP-4399 implementation of nim-vm
what's new:
- `RANDOM` OPCODE
- `random` field of BlockHeader(previously `mixDigest`)
- `PostMerge` temporary name
2022-02-01 18:11:14 +02:00
jangko 01980041d5
setup nimbus tx pool for app wide usage
currently the txpool is used at:
- sealer engine
- graphql ethapi
- json rpc ethapi
- ws rpc ethapi
2022-01-25 11:38:49 +07:00
Zahary Karadjov 137eb97766 Initial implementation of the merge spec
Includes a simple test harness for the merge interop M1 milestone

This aims to enable connecting nimbus-eth2 to nimbus-eth1 within
the testing protocol described here:

https://github.com/status-im/nimbus-eth2/blob/amphora-merge-interop/docs/interop_merge.md

To execute the work-in-progress test, please run:

In terminal 1:
tests/amphora/launch-nimbus.sh

In terminal 2:
tests/amphora/check-merge-test-vectors.sh
2022-01-24 09:44:39 +02:00
Jordan Hrycaj 103656dbb5 TxPool implementation
details:
  For documentation, see comments in the file tx_pool.nim.

  For prettified manual pages run 'make docs' in the nimbus directory and
  point your web browser to the newly created 'docs' directory.
2022-01-22 08:26:57 +02:00
Jordan Hrycaj 9545767c53 update ec_recover
why:
  Previous version was based on lru_cache which is ugly. This module is
  based on the stew/keyed_queue library module.

other:
  There are still some other modules rely on lru_cache which should be
  removed.
2022-01-22 08:26:57 +02:00
Jamie Lokier 74253c88e3
Arrow Glacier fork
Add the new [Arrow Glacier fork](https://eips.ethereum.org/EIPS/eip-4345).

Only the difficulty calculation is changed, but as a new fork it still affects
a number of places in the code.

To the best of my knowledge the change is only scheduled on Mainnet.

In addition:

- The fork date comments in `chain_config.nim` have been checked against the
  real networks, set consistently in UTC instead of random timezones, and made
  neater.  Maybe we'll keep these when transferring config to a file someday.

- It's added to forkid hash tests (EIP-2124/EIP-2364), of course.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-12-10 13:40:51 +00:00
Jordan Hrycaj 55f7a4425f
Jordan/pow cache management (#888)
* PoW wrapper for verification & mining

why:
  It eases data management of per-Epoch lookup tables. Also some unit
  tests show limits of usefulness on non-specialised machines for
  mining besides developing tests.

details:
  For PoW verification, this patch provides a pretty wrapper hiding the
  details of the ethash/Hashimoto lookup cache management.

  For mining on my development system without special hardware, the
  underlying ethash functions are prohibitively slow. It takes
   * ~20 minutes to prepare the full ethash/Hashimoto lookup dataset
   * a second to run ~25k nonce tests (in the mining loop)

  The mining part might be of some use for generating test data for
  the tx-pool, though.

* Using PowRef as replacement for EpochHashCache + hashimotoLight()

* Fix typo (CI failed)

why:
  was below log level when testing locally

* fix canonical naming
2021-12-10 08:49:57 +00:00
jangko 521f29c0a0
remove unused calcGasLimit code
we have new calcGasLimit tested in sealing engine.
so we can safely remove the old unused calcGasLimit
2021-08-24 18:30:52 +07:00
jangko 7dbc44f88c
implement simple PoA sealing engine
the goal of this module is to pass hive/smoke/clique test
and also support for hive/ethereum/rpc test

fixes #801
2021-08-24 14:49:13 +07:00
jangko 18b26a0089
implement calcEIP1559GasLimit
CalcGasLimit1559 calculates the next block gas limit under 1559 rules.
this function is needed in upcoming sealing engine implementation
2021-08-18 20:23:38 +07:00
Jordan Hrycaj 4713bd4cf4
#768 Moved/re-implemented ecRecover() from Clique sources to utils/ec_recover
why:
  The same functionality was differently implemented in one or the
  other form.

details:
  Caching and non-caching variants available
2021-08-05 12:27:10 +07:00
Jordan Hrycaj cfe955c962
Feature/implement poa processing (#748)
* re-shuffled Clique functions

why:
  Due to the port from the go-sources, the interface logic is not optimal
  for nimbus. The main visible function is currently snapshot() and most
  of the _procurement_ of this function result has been moved to a
  sub-directory.

* run eip-225 Clique test against p2p/chain.persistBlocks()

why:
  Previously, loading the test block chains was fugdged with the purpose
  only to fill the database. As it is now clear how nimbus works on
  Goerli, the same can be achieved with a more realistic scenario.

details:
  Eventually these tests will be pre-cursor to the reply tests for the
  Goerli chain supporting TDD approach with more simple cases.

* fix exception annotations for executor module

why:
  needed for exception tracking

details:
  main annoyance are vmState methods (in state.nim) which potentially
  throw a base level Exception (a proc would only throws CatchableError)

* split p2p/chain into sub-modules and fix exception annotations

why:
  make space for implementing PoA stuff

* provide over-loadable Clique PRNG

why:
  There is a PRNG provided for generating reproducible number sequences.
  The functions which employ the PRNG to generate time slots were ported
  ported from the go-implementation. They are currently unused.

* implement trusted signer assembly in p2p/chain.persistBlocks()

details:
  * PoA processing moved there at the end of a transaction. Currently,
   there is no action (eg. transaction rollback) if this fails.
  * The unit tests with staged blocks work ok. In particular, there should
    be tests with to-be-rejected blocks.
  * TODO: 1.Optimise throughput/cache handling; 2.Verify headers

* fix statement cast in pool.nim

* added table features to LRU cache

why:
  Clique uses the LRU cache using a mixture of volatile online items
  from the LRU cache and database checkpoints for hard synchronisation.
  For performance, Clique needs more table like features.

details:
  First, last, and query key added, as well as efficient random delete
  added. Also key-item pair iterator added for debugging.

* re-factored LRU snapshot caching

why:
  Caching was sub-optimal (aka. bonkers) in that it skipped over memory
  caches in many cases and so mostly rebuild the snapshot from the
  last on-disk checkpoint.

details;
  The LRU snapshot toValue() handler has been moved into the module
  clique_snapshot. This is for the fact that toValue() is not supposed
  to see the whole LRU cache database. So there must be a higher layer
  working with the the whole LRU cache and the on-disk checkpoint
  database.

also:
  some clean up

todo:
  The code still assumes that the block headers are valid in itself. This
  is particular important when an epoch header (aka re-sync header) is
  processed as it must contain the PoA result of all previous headers.

  So blocks need to be verified when they come in before used for PoA
  processing.

* fix some snapshot cache fringe cases

why:
  Must not index empty sequences in clique_snapshot module
2021-07-14 16:13:27 +01:00
jangko db8988fe64
EIP-1559: Fee market change for ETH 1.0 chain
Transaction and BlockHeader already updated in nim-eth repo
to support EIP-1559

EIP-1559 header validation and gasLimit validation
already implemented in previous commit

This commit deals with block validation:
- Effective gasPrice per EIP-1559
- new miner reward based on priorityFee
2021-06-30 20:30:39 +07:00
jangko 7600046a11
EIP-1559: unify PoA and PoW gasLimit and baseFee validation
Turn out both EthHash and Clique are using the same gasLimit
validation.

They also share the same EIP-1559 baseFee validation.
2021-06-30 20:21:45 +07:00
jangko b4221381d6
EIP-3554: Difficulty Bomb Delay to December 2021 2021-06-29 07:36:41 +07:00
Jordan Hrycaj 61e460c125 Most snapshot unit tests work
details:
  three test cases still fail which are skipped
  test suite is linked to all_tests list
2021-06-17 08:03:57 +01:00
Jordan Hrycaj e5947f4db6 Deep copy semantics for LRU cache
why:
  follows standard nim semantics

details:
  changed Table to TableRef in previous patch which was the
  wrong choice (see andri's comment.)
2021-05-26 11:12:52 +01:00
Jordan Hrycaj 7b72109afa Use sorted RLP serialisation for LRU cache
why:
  previously, table data were stored with the table iterator. while
  loading a table with permuted entries will always reconstruct equivalent
  tables (in the sense of `==`), serialisation data are not comparable.

  this patch produces always the same serialised data for equivalent
  tables.
2021-05-26 07:58:12 +01:00
Jordan Hrycaj b83b47e541 LRU cache tests makeover
why:
  source-local unit tests would hardly be triggered by github CI as rightly
  criticised with the last patch.

details:
  source-local unit tests have been moved to tests folder.

  this version also contains rlp serialisation code so rlp encode/decode
  will apply tranparently. this is not needed in p2p/validate but will be
   useful with the clique protocol.
2021-05-26 07:58:12 +01:00
Jordan Hrycaj 179cc75c32 Update LRU complexity comment
why:
  hash tables might worst-case degrade into linear mode, so it is not
  strictly O(1)
2021-05-24 07:57:21 +01:00
Jordan Hrycaj 1965ea027b updated LRU cache to run with complexity O(1)
why:
  to be used in Clique consensus protocol which suggests 4k cache entries.
  the previous implementation used OrderTable[] which has complexity O(n)
  for deleting entries.
2021-05-24 07:57:21 +01:00
Jordan Hrycaj a5e0bb6139 use general lru_cache for EpochHashCache
why:
  generic implementation will be also be used elsewhere
2021-05-24 07:57:21 +01:00
jangko f2491e6307
fixes crappy custom genesis and chain config parser
instead of using stdlib/json, now we switch to json_serialization
the result is much tidier code and more robust when parsing
optional fields.

fixes #635
2021-05-13 16:04:08 +07:00
jangko 207065746c
reduce more warnings 2020-07-21 13:25:27 +07:00
andri lim af02a3b1b2
reduce unused import warnings 2020-04-15 19:05:57 +07:00
andri lim d2b0ca62b2
fix GlacierMuir to MuirGlacier typo 2020-04-12 18:13:22 +07:00
andri lim 87bae2bb78
switch to new toFork 2020-04-12 18:02:59 +07:00
andri lim 6591893158
change FkGlacierMuir to FkMuirGlacier [skip ci] 2020-04-11 16:02:15 +07:00