Commit Graph

32 Commits

Author SHA1 Message Date
Jacek Sieka 0b32078c4b
Consolidate block type for block processing (#2325)
This PR consolidates the split header-body sequences into a single EthBlock
sequence and cleans up the fallout from that which significantly reduces
block processing overhead during import thanks to less garbage collection
and fewer copies of things all around.

Notably, since the number of headers must always match the number of bodies,
we also get rid of a pointless degree of freedom that in the future could
introduce unnecessary bugs.

* only read header and body from era file
* avoid several unnecessary copies along the block processing way
* simplify signatures, cleaning up unused arguemnts and returns
* use `stew/assign2` in a few strategic places where the generated
  nim assignent is slow and add a few `move` to work around poor
  analysis in nim 1.6 (will need to be revisited for 2.0)

```
stats-20240607_2223-a814aa0b.csv vs stats-20240608_0714-21c1d0a9.csv
                       bps_x     bps_y     tps_x        tps_y    bpsd    tpsd    timed
block_number
(498305, 713245]    1,540.52  1,809.73  2,361.58  2775.340189  17.63%  17.63%  -14.92%
(713245, 928185]      730.36    865.26  1,715.90  2028.973852  18.01%  18.01%  -15.21%
(928185, 1143126]     663.03    789.10  2,529.26  3032.490771  19.79%  19.79%  -16.28%
(1143126, 1358066]    393.46    508.05  2,152.50  2777.578119  29.13%  29.13%  -22.50%
(1358066, 1573007]    370.88    440.72  2,351.31  2791.896052  18.81%  18.81%  -15.80%
(1573007, 1787947]    283.65    335.11  2,068.93  2441.373402  17.60%  17.60%  -14.91%
(1787947, 2002888]    287.29    342.11  2,078.39  2474.179448  18.99%  18.99%  -15.91%
(2002888, 2217828]    293.38    343.16  2,208.83   2584.77457  17.16%  17.16%  -14.61%
(2217828, 2432769]    140.09    167.86  1,081.87  1296.336926  18.82%  18.82%  -15.80%

blocks: 1934464, baseline: 3h13m1s, contender: 2h43m47s
bpsd (mean): 19.55%
tpsd (mean): 19.55%
Time (total): -29m13s, -15.14%
```
2024-06-09 16:32:20 +02:00
Jordan Hrycaj e9eae4df70
Core db disable legacy api n remove distinct tries (#2299)
* CoreDb: Remove crufty second/off-site KVT

why:
  Was used to allow late `Clique` to store directly to disk

* CoreDb: Remove prune flag related functionality

why:
  Is completely legacy stuff

* CoreDb: Remove dependence on legacy API (tests unsupported yet)

why:
  Does not fully support Aristo

* Re-factoring `state_db` using new API

details:
  Only minimum changes needed to compile `nimbus`

* Update tests and aux modules

* Turn off legacy API and remove `distinct_tries`

comment:
  The legacy API has now cruft status, will be removed soon

* Fix copyright years

* Update rpc for verified proxy

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2024-06-05 20:52:04 +00:00
Jacek Sieka c876729c4d
Add some basic rocksdb options to command line (#2286)
These options are there mainly to drive experiments, and are therefore
hidden.

One thing that this PR brings in is an initial set of caches and buffers for rocksdb - the set that I've been using during various performance tests to get to a viable baseline performance level.
2024-06-05 17:08:29 +02:00
Jordan Hrycaj ee9aea171d
Culling legacy DB and accounts cache (#2197)
details:
+ Compiles nimbus all_tests
+ Failing tests have been commented out
2024-05-20 10:17:51 +00:00
Jordan Hrycaj a02a915039
Provide public default db symbol (#2050)
* CoreDb: Provide default db backend symbols

why:
  Handy for running `Aristo` against standard tests

note:
  These defaults are currently set to legacy DB types. The must be
  enabled manually in `db/core_db.nim`.

* Provide `Aristo` for macro assembler related tests

caveat:
  Some tests use `initStorageTrie()` which lets `Aristo` bail out. The
  test need to run on `distinct_ledgers` (or something like) rather than
  `distinct_tries`.

* Tests: Misc modules that can run on `Aristo` as well

* NoHive: Module that can run on `Aristo` as well

* Fix copyright year

* ditto
2024-02-23 09:17:24 +00:00
jangko b0000eed8b
Add check copyright year linter to CI 2023-11-01 10:41:20 +07:00
Jordan Hrycaj 221e6c9e2f
Unified database frontend integration (#1670)
* Nimbus folder environment update

details:
* Integrated `CoreDbRef` for the sources in the `nimbus` sub-folder.
* The `nimbus` program does not compile yet as it needs the updates
  in the parallel `stateless` sub-folder.

* Stateless environment update

details:
* Integrated `CoreDbRef` for the sources in the `stateless` sub-folder.
* The `nimbus` program compiles now.

* Premix environment update

details:
* Integrated `CoreDbRef` for the sources in the `premix` sub-folder.

* Fluffy environment update

details:
* Integrated `CoreDbRef` for the sources in the `fluffy` sub-folder.

* Tools environment update

details:
* Integrated `CoreDbRef` for the sources in the `tools` sub-folder.

* Nodocker environment update

details:
* Integrated `CoreDbRef` for the sources in the
  `hive_integration/nodocker` sub-folder.

* Tests environment update

details:
* Integrated `CoreDbRef` for the sources in the `tests` sub-folder.
* The unit tests compile and run cleanly now.

* Generalise `CoreDbRef` to any `select_backend` supported database

why:
  Generalisation was just missed due to overcoming some compiler oddity
  which was tied to rocksdb for testing.

* Suppress compiler warning for `newChainDB()`

why:
  Warning was added to this function which must be wrapped so that
  any `CatchableError` is re-raised as `Defect`.

* Split off persistent `CoreDbRef` constructor into separate file

why:
  This allows to compile a memory only database version without linking
  the backend library.

* Use memory `CoreDbRef` database by default

detail:
 Persistent DB constructor needs to import `db/core_db/persistent

why:
 Most tests use memory DB anyway. This avoids linking `-lrocksdb` or
 any other backend by default.

* fix `toLegacyBackend()` availability check

why:
  got garbled after memory/persistent split.

* Clarify raw access to MPT for snap sync handler

why:
  Logically, `kvt` is not the raw access for the hexary trie (although
  this holds for the legacy database)
2023-08-04 12:10:09 +01:00
jangko 94a94c5b65 implement better hardfork management 2022-12-02 13:51:42 +07:00
jangko 709d8ef255
fix markCanonicalChain bug 2022-07-04 19:32:14 +07:00
jangko f2f204293e
first step into styleCheck fixes 2022-04-14 08:39:50 +07:00
jangko 69f2a0f95a
config: replace stdlib parseOpt with nim-confutils
fixes #581
2021-09-18 17:34:46 +07:00
jangko c9cfebfa97
config: rearrange getConfiguration usage
avoid using getConfiguration inside object construction and
replace it with passing suitable param
2021-09-08 08:07:10 +07:00
jangko 165f9fea2e
reduce warnings 2020-07-21 13:15:06 +07:00
kdeme b4b096ef5e Add PersistBlockTest for block 1352922, fixes #363 2019-08-14 20:56:57 +02:00
Jacek Sieka 2763bd0dd5
std_shims -> stew 2019-07-07 12:12:01 +02:00
andri lim a0126a7458 add byzantium block validation test 2019-05-15 17:03:36 +03:00
andri lim f3c740498c
implement EIP155 tx.getSender 2019-04-18 15:33:17 +07:00
andri lim e658ac0a30
add block 2.675.000 test data 2019-04-18 13:49:06 +07:00
andri lim c852199481
add test data 2019-04-15 11:17:24 +07:00
andri lim 847ab79082
add dao test data 2019-04-07 06:55:25 +07:00
andri lim b8133e9f1c
add test data 2019-04-06 21:27:06 +07:00
andri lim 268695003b
fix writePaddedResult regression 2019-03-19 16:43:38 +07:00
andri lim fee2a97b52 add problematic block tests 2019-03-13 15:49:54 +02:00
andri lim 56f81ae821
add notable blocks into test 2019-03-11 20:18:28 +07:00
andri lim 8fbcbfb9b3
add some notable blocks into test 2019-02-26 08:45:32 +07:00
andri lim c657f23d73 fixes persistBlockTestGen and update tests 2019-02-07 11:01:43 +01:00
andri lim 4466deff37 fixes path in tests 2019-02-06 18:42:03 +01:00
Yuriy Glukhov 481c6cf4ed Use nim-eth (#224) 2019-02-05 20:15:50 +01:00
andri lim 281b7dffac implement debug_setHead 2019-01-06 13:17:08 +02:00
andri lim 74d728f962 pass additional 15 general state tests 2019-01-06 13:16:45 +02:00
andri lim 96b18cfce3 add select_backend module 2019-01-06 11:43:38 +02:00
andri lim 3a10f99694 add persistblock test 2019-01-06 11:43:38 +02:00