nimbus-eth1/nimbus/db
Jacek Sieka c364426422
Smaller in-database representations (#2436)
These representations use ~15-20% less data compared to the status quo,
mainly by removing redundant zeroes in the integer encodings - a
significant effect of this change is that the various rocksdb caches see
better efficiency since more items fit in the same amount of space.

* use RLP encoding for `VertexID` and `UInt256` wherever it appears
* pack `VertexRef`/`PayloadRef` more tightly
2024-07-02 20:25:06 +02:00
..
aristo Smaller in-database representations (#2436) 2024-07-02 20:25:06 +02:00
core_db Smaller in-database representations (#2436) 2024-07-02 20:25:06 +02:00
era1_db Consolidate block type for block processing (#2325) 2024-06-09 16:32:20 +02:00
kvt Update storage tree admin (#2419) 2024-06-27 09:01:26 +00:00
ledger Small cleanups (#2435) 2024-07-01 14:07:39 +02:00
.gitignore Database architecture diagram & module overview (#2065) 2024-03-08 18:42:46 +00:00
README.md Aristo resume off line syncing on pre loaded database (#2203) 2024-05-22 13:41:14 +00:00
access_list.nim avoid initTable (#2328) 2024-06-10 11:05:30 +02:00
aristo.nim Update storage tree admin (#2419) 2024-06-27 09:01:26 +00:00
core_db.nim Cleanup unused raises in evm/state and other obsolete informations (#2243) 2024-05-30 09:03:54 +00:00
era1_db.nim era: simplify, instant startup (#2218) 2024-05-26 08:24:13 +02:00
kvstore_rocksdb.nim Bump RocksDb version and enable autoClose on opt types to prevent memory leaks (#2427) 2024-07-02 13:44:09 +08:00
kvt.nim Coredb use stackable api for aristo backend (#2060) 2024-02-29 21:10:24 +00:00
ledger.nim Update storage tree admin (#2419) 2024-06-27 09:01:26 +00:00
opts.nim Fix memory usage spikes during sync, give memory to rocksdb (#2413) 2024-06-25 13:39:53 +02:00
storage_types.nim Use simpler schema when writing transactions, receipts, and withdrawals (#2420) 2024-06-29 12:43:17 +07:00
transient_storage.nim avoid initTable (#2328) 2024-06-10 11:05:30 +02:00

README.md

Nimbus-eth1 -- Ethereum execution layer database architecture

Last update: 2024-03-08

The following diagram gives a simplified view how components relate with regards to the data storage management.

An arrow between components a and b (as in a->b) is meant to be read as a relies directly on b, or a is served by b. For classifying the functional type of a component in the below diagram, the abstraction type is enclosed in brackets after the name of a component.

  • (application)
    This is a group of software modules at the top level of the hierarchy. In the diagram below, the EVM is used as an example. Another application might be the RPC service.

  • (API)
    The API classification is used for a thin software layer hiding a set of different drivers where only one driver is active for the same API instance. It servers as sort of a logical switch.

  • (concentrator)
    The concentrator merges several sub-module instances and provides their collected services as a single unified instance. There is not much additional logic implemented besides what the sub-modules provide.

  • (driver)
    The driver instances are sort of the lower layer workhorses. The implement logic for solving a particular problem, providing a typically well defined service, etc.

  • (engine)
    This is a bottom level driver in the below diagram.

                           +-------------------+
                           | EVM (application) |
                           +-------------------+
                                   |     |
                                   v     |
       +-----------------------------+   |
       |   State DB (concentrator)   |   |
       +-----------------------------+   |
           |                       |     |
           v                       |     |
       +------------------------+  |     |
       |      Ledger (API)      |  |     |
       +------------------------+  |     |
           |              |        |     |
           v              |        |     |
       +--------------+   |        |     |
       | ledger cache |   |        |     |
       |   (driver)   |   |        |     |
       +--------------+   |        |     |
           |              v        |     |
           |   +----------------+  |     |
           |   |   Common       |  |     |
           |   | (concentrator) |  |     |
           |   +----------------+  |     |
           |             |         |     |
           v             v         v     v
       +---------------------------------------+
       |               Core DB (API)           |
       +---------------------------------------+
                         |
                         v
       +---------------------------------------+
       |    Aristo DB (driver,concentrator)    |
       +---------------------------------------+
                 |             |
                 v             v
       +--------------+  +---------------------+
       | Kvt (driver) |  | Aristo MPT (driver) |
       +--------------+  +---------------------+
                 |             |
                 v             v
       +---------------------------------------+
       |         Rocks DB (engine)             |
       +---------------------------------------+
    

Here is a list of path references for the components with some explanation. The sources for the components are not always complete but indicate the main locations where to start looking at.

  • Aristo DB (driver)

    • Sources:
      ./nimbus/db/core_db/backend/aristo_*

    • Synopsis:
      Combines both, the Kvt and the Aristo driver sub-modules providing an interface similar to the legacy DB (concentrator) module.

  • Aristo MPT (driver)

    • Sources:
      ./nimbus/db/aristo*

    • Synopsis:
      Revamped implementation of a hexary Merkle Patricia Tree.

  • Common (concentrator)

    • Sources:
      ./nimbus/common*

    • Synopsis:
      Collected information for running block chain execution layer applications.

  • Core DB (API)

    • Sources:
      ./nimbus/db/core_db*

    • Synopsis:
      Database abstraction layer. Unless for legacy applications, there should be no need to reach out to the layers below.

  • EVM (application)

    • Sources:
      ./nimbus/core/executor/* ./nimbus/evm/*

    • Synopsis:
      An implementation of the Ethereum Virtual Machine.

  • Hexary DB (driver)

  • Key-value table (driver)

    • Sources:
      ./vendor/nim-eth/eth/trie/db.nim

    • Synopsis:
      Key value table interface to be used directly for key-value storage or by the Hexary DB (driver) module for storage. Some magic is applied in order to treat hexary data accordingly (based on key length.)

  • Kvt (driver)

  • Ledger (API)

  • ledger cache (driver)

    • Sources:
      ./nimbus/db/ledger/accounts_ledger.nim
      ./nimbus/db/ledger/backend/accounts_ledger*
      ./nimbus/db/ledger/distinct_ledgers.nim

    • Synopsis:
      Management of accounts and storage data. This is a re-write of the legacy DB (driver) which is supposed to work with all Core DB (API) backends.

  • legacy DB (concentrator)

  • Rocks DB (engine)

    • Sources:
      ./vendor/nim-rocksdb/*

    • Synopsis:
      Persistent storage engine.

  • State DB (concentrator)

    • Sources:
      ./nimbus/evm/state.nim
      ./nimbus/evm/types.nim

    • Synopsis:
      Integrated collection of modules and methods relevant for the EVM.