Commit Graph

17 Commits

Author SHA1 Message Date
Jordan Hrycaj 221e6c9e2f
Unified database frontend integration (#1670)
* Nimbus folder environment update

details:
* Integrated `CoreDbRef` for the sources in the `nimbus` sub-folder.
* The `nimbus` program does not compile yet as it needs the updates
  in the parallel `stateless` sub-folder.

* Stateless environment update

details:
* Integrated `CoreDbRef` for the sources in the `stateless` sub-folder.
* The `nimbus` program compiles now.

* Premix environment update

details:
* Integrated `CoreDbRef` for the sources in the `premix` sub-folder.

* Fluffy environment update

details:
* Integrated `CoreDbRef` for the sources in the `fluffy` sub-folder.

* Tools environment update

details:
* Integrated `CoreDbRef` for the sources in the `tools` sub-folder.

* Nodocker environment update

details:
* Integrated `CoreDbRef` for the sources in the
  `hive_integration/nodocker` sub-folder.

* Tests environment update

details:
* Integrated `CoreDbRef` for the sources in the `tests` sub-folder.
* The unit tests compile and run cleanly now.

* Generalise `CoreDbRef` to any `select_backend` supported database

why:
  Generalisation was just missed due to overcoming some compiler oddity
  which was tied to rocksdb for testing.

* Suppress compiler warning for `newChainDB()`

why:
  Warning was added to this function which must be wrapped so that
  any `CatchableError` is re-raised as `Defect`.

* Split off persistent `CoreDbRef` constructor into separate file

why:
  This allows to compile a memory only database version without linking
  the backend library.

* Use memory `CoreDbRef` database by default

detail:
 Persistent DB constructor needs to import `db/core_db/persistent

why:
 Most tests use memory DB anyway. This avoids linking `-lrocksdb` or
 any other backend by default.

* fix `toLegacyBackend()` availability check

why:
  got garbled after memory/persistent split.

* Clarify raw access to MPT for snap sync handler

why:
  Logically, `kvt` is not the raw access for the hexary trie (although
  this holds for the legacy database)
2023-08-04 12:10:09 +01:00
Jordan Hrycaj 322f1c2e9e
Unified database frontend (#1661)
* Remove 32bit os support from `custom_network` unit test

also:
* Fix compilation annoyance #1648
* Fix unit test on Kiln (changed `merge` logic?)

* Hide unused sources do not compile

why:
* Get them out of the way before major update
* Import and function prototype mismatch -- maybe some changes got out
  of scope.

* Re-implemented `db_chain` as `core_db`

why:
  Hiding `TrieDatabaseRef` and `HexaryTrie` by default allows to replace
  the current db wrapper by some other one, e.g. Aristo

* Support compiler exception warnings for CoreDbRef base methods.

* Allow `pairs()` iterator on all memory based key-value tables

why:
  Previously only available for capture recorder.

* Backport `chain_db.nim` changes into its re-implementation `core_apps.nim`

* Fix exception annotation
2023-07-31 14:43:38 +01:00
Jordan Hrycaj f40a066cc6
Update snap sync ready to succeed at lab test (#1556)
* Extract RocksDB timing tests from snap unit tests as separate module

why:
  Declutter, make space for more snap related unit tests.

* Renamed `undumpNextGroup()` => `undumpBlocks()`

why:
  Source file name is called `undump_blocks.nim` which should be sort
  of in sync with the method name(s).

* Implement snap/1 server method `getByteCodes()`

* Implement snap/1 client method `getByteCodes()`

* Implement faculty for handling contract code fetching via snap/1

* Provide persistent storage for contract code records

* Implement contract code snap sync fetch & store

* Code massage, cosmetics

* Unit tests for verifying snap sync snapshot dump

details:
  Use `undump_kvp.dumpAllDb()` to dump any database.
2023-04-21 22:11:04 +01:00
Adam Spitz fad3ed64cf
Time based forking (#1465)
* Refactoring in preparation for time-based forking.

* Timestamp-based hard-fork-transition.

* Workaround SideEffect issue / compiler bug for both failing locations in Portal history code

---------

Co-authored-by: kdeme <kim.demey@gmail.com>
2023-02-16 12:40:07 +01:00
jangko 53e71e8837
better hardForkTransition usage 2022-12-05 15:46:37 +07:00
jangko 94a94c5b65 implement better hardfork management 2022-12-02 13:51:42 +07:00
Kim De Mey 74a83c1229
Bump nim-eth and remove all now unneeded p2p related imports (#1273) 2022-10-20 10:34:59 +02:00
jangko a48cc04ea7
add gray glacier difficulty calculator
fixes #1264
2022-10-19 09:51:27 +07:00
Jacek Sieka c2ed731fa5
eth: adapt to smaller eth_types (#1210) 2022-09-03 20:15:35 +02:00
Jordan Hrycaj de2c13e136
Update snap offline tests (#1199)
* Re-implemented `hexaryFollow()` in a more general fashion

details:
+ New name for re-implemented `hexaryFollow()` is `hexaryPath()`
+ Renamed `rTreeFollow()` as `hexaryPath()`

why:
  Returning similarly organised structures, the results of the
  `hexaryPath()` functions become comparable when running over
  the persistent and the in-memory databases.

* Added traversal functionality for persistent ChainDB

* Using `Account` values as re-packed Blob

* Repack samples as compressed data files

* Produce test data

details:
+ Can force pivot state root switch after minimal coverage.
+ For emulating certain network behaviour, downloading accounts stops for
  a particular pivot state root if 30% (some static number) coverage is
  reached. Following accounts are downloaded for a later pivot state root.
2022-08-24 14:44:18 +01:00
jangko 2b4baff8ec
move block validation from execution payload generator to engine api 2022-07-04 20:36:30 +07:00
Jordan Hrycaj 134fe26997
Store proved snap accounts (#1145)
* Relocated `IntervalSets` to nim-stew repo

* Accumulate accounts on temporary kv-DB

why:
  Explore the data as returned from snap/1. Will be converted to a
  `eth/db` next.

details:
  Verify and accumulate per/state-root accounts downloaded via snap.

also:
  Some unit tests

* Replace `Table` by `TrieDatabaseRef` for accounts accumulator

* update ticker statistics

details:
  mean/variance based counter update

* allow persistent db for proved accounts

* rebase, and globally activate unit test

* fix statistics
2022-07-01 12:42:17 +01:00
jangko f2f204293e
first step into styleCheck fixes 2022-04-14 08:39:50 +07:00
Jordan Hrycaj 737236fd6e
Enable JWT authentication for websockets (#1039)
* Enable JWT authentication for websockets

details:
  Currently, this is optional and only enabled when the jwtsecret option
  is set.

  There is a default mechanism to generate a JWT secret if it is not
  explicitly stated. This mechanism is currently unused.

* Make JWT authentication compulsory for websockets

* Fix unit test entry point + cosmetics

* Update JSON-RPC link

* Improvements as suggested by Mamy
2022-04-06 15:11:13 +01:00
Jordan Hrycaj a64da329e8
Jordan/ttd debugging helpers (#1013)
* De-noisify some Clique logging

why:
  Too annoying when syncing against Goerly

* Replay devnet# and kiln sessions

why:
  Compiled as local program, the unit test was used for TDD.
2022-03-28 09:35:26 +01:00
Jordan Hrycaj ed0e882387
Exported blobs and some scripts to parallel project nimbus-eth1-blobs (#995)
why:
  TDD data and test script that are not needed for CI are externally held.
  This saves space.

also:
  Added support for test-custom_networks.nim to run import Devnet5 dump.
2022-03-16 09:13:17 +00:00
Jordan Hrycaj 215e9856d3
Jordan/custom network (#962)
* Rearrange/rename test_kintsugu => test_custom_network

why:
  Debug, fix and test more general problems related to running
  nimbus on a custom network.

* Update UInt265/Json parser for --custom-network command line option

why:
  As found out with the Kintsugi configuration, block number and balance
  have the same Nim type which led to misunderstandings. This patch makes
  sure that UInt265 encoded string values "0x11" decodes to 17, and "b"
  and "11" to 11.

* Refactored genesis.toBlock() => genesis.toBlockHeader()

why:
  The function toBlock(g,db) may return different results depending on
  whether the db descriptor argument is nil, or initialised. This is due
  to the db.config data sub-descriptor which may give various outcomes
  for the baseFee field of the genesis header.

  Also, the version where db is non-nil initialised is used internally
  only. So the public rewrite toBlockHeader() that replaces the toBlock()
  function expects a full set of NetworkParams.

* update comments

* Rename toBlockHeader() => toGenesisHeader()

why:
  Polymorphic prototype used for BaseChainDB or NetworkParams argument.
  With a BaseChainDB descriptor argument, the name shall imply that the
  header is generated from the config fields rather than fetched from
  the database.

* Added command line option --static-peers-file

why:
  Handy feature to keep peer nodes in a file, similar to the
  --bootstrap-file option.
2022-02-11 16:28:39 +00:00