status-im
/
nimbus-eth1
mirror of https://github.com/status-im/nimbus-eth1.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth1/nimbus/db
History
Jacek Sieka df4a21c910
Store cached hash at the layer corresponding to the source data (#2492) 
When lazily verifying state roots, we may end up with an entire state
without roots that gets computed for the whole database - in the current
design, that would result in hashes for the entire trie being held in
memory.

Since the hash depends only on the data in the vertex, we can store it
directly at the top-most level derived from the verticies it depends on
- be that memory or database - this makes the memory usage broadly
linear with respect to the already-existing in-memory change set stored
in the layers.

It also ensures that if we have multiple forks in memory, hashes get
cached in the correct layer maximising reuse between forks.

The same layer numbering scheme as elsewhere is reused, where -2 is the
backend, -1 is the balancer, then 0+ is the top of the stack and stack.

A downside of this approach is that we create many small batches - a
future improvement could be to collect all such writes in a single
batch, though the memory profile of this approach should be examined
first (where is the batch kept, exactly?).
 		2024-07-18 09:13:56 +02:00
..
aristo Store cached hash at the layer corresponding to the source data (#2492) 2024-07-18 09:13:56 +02:00
core_db No ext update (#2494) 2024-07-16 19:47:59 +00:00
era1_db Consolidate block type for block processing (#2325) 2024-06-09 16:32:20 +02:00
kvt Separate config for core db and ledger (#2479) 2024-07-12 13:12:25 +00:00
ledger Not using deprecated functions in config anymore (#2495) 2024-07-17 02:57:19 +00:00
.gitignore Database architecture diagram & module overview (#2065) 2024-03-08 18:42:46 +00:00
README.md Aristo resume off line syncing on pre loaded database (#2203) 2024-05-22 13:41:14 +00:00
access_list.nim avoid initTable (#2328) 2024-06-10 11:05:30 +02:00
aristo.nim Core db reorg (#2444) 2024-07-03 15:50:27 +00:00
core_db.nim Cleanup unused raises in evm/state and other obsolete informations (#2243) 2024-05-30 09:03:54 +00:00
era1_db.nim era: simplify, instant startup (#2218) 2024-05-26 08:24:13 +02:00
kvstore_rocksdb.nim Bump RocksDb version and enable autoClose on opt types to prevent memory leaks (#2427) 2024-07-02 13:44:09 +08:00
kvt.nim Core db reorg (#2444) 2024-07-03 15:50:27 +00:00
ledger.nim Separate constructor helpers for core db and ledger (#2480) 2024-07-12 19:32:31 +00:00
opts.nim Fix memory usage spikes during sync, give memory to rocksdb (#2413) 2024-06-25 13:39:53 +02:00
storage_types.nim don't rewrite hash->slot map (#2463) 2024-07-09 17:25:43 +02:00
transient_storage.nim avoid initTable (#2328) 2024-06-10 11:05:30 +02:00

README.md

Nimbus-eth1 -- Ethereum execution layer database architecture

Last update: 2024-03-08

The following diagram gives a simplified view how components relate with regards to the data storage management.

An arrow between components a and b (as in a->b) is meant to be read as a relies directly on b, or a is served by b. For classifying the functional type of a component in the below diagram, the abstraction type is enclosed in brackets after the name of a component.

  • (application)
    This is a group of software modules at the top level of the hierarchy. In the diagram below, the EVM is used as an example. Another application might be the RPC service.

  • (API)
    The API classification is used for a thin software layer hiding a set of different drivers where only one driver is active for the same API instance. It servers as sort of a logical switch.

  • (concentrator)
    The concentrator merges several sub-module instances and provides their collected services as a single unified instance. There is not much additional logic implemented besides what the sub-modules provide.

  • (driver)
    The driver instances are sort of the lower layer workhorses. The implement logic for solving a particular problem, providing a typically well defined service, etc.

  • (engine)
    This is a bottom level driver in the below diagram.

                           +-------------------+
                       | EVM (application) |
                       +-------------------+
                               |     |
                               v     |
   +-----------------------------+   |
   |   State DB (concentrator)   |   |
   +-----------------------------+   |
       |                       |     |
       v                       |     |
   +------------------------+  |     |
   |      Ledger (API)      |  |     |
   +------------------------+  |     |
       |              |        |     |
       v              |        |     |
   +--------------+   |        |     |
   | ledger cache |   |        |     |
   |   (driver)   |   |        |     |
   +--------------+   |        |     |
       |              v        |     |
       |   +----------------+  |     |
       |   |   Common       |  |     |
       |   | (concentrator) |  |     |
       |   +----------------+  |     |
       |             |         |     |
       v             v         v     v
   +---------------------------------------+
   |               Core DB (API)           |
   +---------------------------------------+
                     |
                     v
   +---------------------------------------+
   |    Aristo DB (driver,concentrator)    |
   +---------------------------------------+
             |             |
             v             v
   +--------------+  +---------------------+
   | Kvt (driver) |  | Aristo MPT (driver) |
   +--------------+  +---------------------+
             |             |
             v             v
   +---------------------------------------+
   |         Rocks DB (engine)             |
   +---------------------------------------+

Here is a list of path references for the components with some explanation. The sources for the components are not always complete but indicate the main locations where to start looking at.

  • Aristo DB (driver)

    • Sources:
      ./nimbus/db/core_db/backend/aristo_*

    • Synopsis:
      Combines both, the Kvt and the Aristo driver sub-modules providing an interface similar to the legacy DB (concentrator) module.

  • Aristo MPT (driver)

    • Sources:
      ./nimbus/db/aristo*

    • Synopsis:
      Revamped implementation of a hexary Merkle Patricia Tree.

  • Common (concentrator)

    • Sources:
      ./nimbus/common*

    • Synopsis:
      Collected information for running block chain execution layer applications.

  • Core DB (API)

    • Sources:
      ./nimbus/db/core_db*

    • Synopsis:
      Database abstraction layer. Unless for legacy applications, there should be no need to reach out to the layers below.

  • EVM (application)

    • Sources:
      ./nimbus/core/executor/* ./nimbus/evm/*

    • Synopsis:
      An implementation of the Ethereum Virtual Machine.

  • Hexary DB (driver)

  • Key-value table (driver)

    • Sources:
      ./vendor/nim-eth/eth/trie/db.nim

    • Synopsis:
      Key value table interface to be used directly for key-value storage or by the Hexary DB (driver) module for storage. Some magic is applied in order to treat hexary data accordingly (based on key length.)

  • Kvt (driver)

  • Ledger (API)

  • ledger cache (driver)

    • Sources:
      ./nimbus/db/ledger/accounts_ledger.nim
      ./nimbus/db/ledger/backend/accounts_ledger*
      ./nimbus/db/ledger/distinct_ledgers.nim

    • Synopsis:
      Management of accounts and storage data. This is a re-write of the legacy DB (driver) which is supposed to work with all Core DB (API) backends.

  • legacy DB (concentrator)

  • Rocks DB (engine)

    • Sources:
      ./vendor/nim-rocksdb/*

    • Synopsis:
      Persistent storage engine.

  • State DB (concentrator)

    • Sources:
      ./nimbus/evm/state.nim
      ./nimbus/evm/types.nim

    • Synopsis:
      Integrated collection of modules and methods relevant for the EVM.