df4a21c910
When lazily verifying state roots, we may end up with an entire state without roots that gets computed for the whole database - in the current design, that would result in hashes for the entire trie being held in memory. Since the hash depends only on the data in the vertex, we can store it directly at the top-most level derived from the verticies it depends on - be that memory or database - this makes the memory usage broadly linear with respect to the already-existing in-memory change set stored in the layers. It also ensures that if we have multiple forks in memory, hashes get cached in the correct layer maximising reuse between forks. The same layer numbering scheme as elsewhere is reused, where -2 is the backend, -1 is the balancer, then 0+ is the top of the stack and stack. A downside of this approach is that we create many small batches - a future improvement could be to collect all such writes in a single batch, though the memory profile of this approach should be examined first (where is the batch kept, exactly?). |
||
---|---|---|
.. | ||
aristo | ||
core_db | ||
era1_db | ||
kvt | ||
ledger | ||
.gitignore | ||
README.md | ||
access_list.nim | ||
aristo.nim | ||
core_db.nim | ||
era1_db.nim | ||
kvstore_rocksdb.nim | ||
kvt.nim | ||
ledger.nim | ||
opts.nim | ||
storage_types.nim | ||
transient_storage.nim |
README.md
Nimbus-eth1 -- Ethereum execution layer database architecture
Last update: 2024-03-08
The following diagram gives a simplified view how components relate with regards to the data storage management.
An arrow between components a and b (as in a->b) is meant to be read as a relies directly on b, or a is served by b. For classifying the functional type of a component in the below diagram, the abstraction type is enclosed in brackets after the name of a component.
-
(application)
This is a group of software modules at the top level of the hierarchy. In the diagram below, the EVM is used as an example. Another application might be the RPC service. -
(API)
The API classification is used for a thin software layer hiding a set of different drivers where only one driver is active for the same API instance. It servers as sort of a logical switch. -
(concentrator)
The concentrator merges several sub-module instances and provides their collected services as a single unified instance. There is not much additional logic implemented besides what the sub-modules provide. -
(driver)
The driver instances are sort of the lower layer workhorses. The implement logic for solving a particular problem, providing a typically well defined service, etc. -
(engine)
This is a bottom level driver in the below diagram.+-------------------+ | EVM (application) | +-------------------+ | | v | +-----------------------------+ | | State DB (concentrator) | | +-----------------------------+ | | | | v | | +------------------------+ | | | Ledger (API) | | | +------------------------+ | | | | | | v | | | +--------------+ | | | | ledger cache | | | | | (driver) | | | | +--------------+ | | | | v | | | +----------------+ | | | | Common | | | | | (concentrator) | | | | +----------------+ | | | | | | v v v v +---------------------------------------+ | Core DB (API) | +---------------------------------------+ | v +---------------------------------------+ | Aristo DB (driver,concentrator) | +---------------------------------------+ | | v v +--------------+ +---------------------+ | Kvt (driver) | | Aristo MPT (driver) | +--------------+ +---------------------+ | | v v +---------------------------------------+ | Rocks DB (engine) | +---------------------------------------+
Here is a list of path references for the components with some explanation. The sources for the components are not always complete but indicate the main locations where to start looking at.
-
-
Sources:
./nimbus/db/core_db/backend/aristo_* -
Synopsis:
Combines both, the Kvt and the Aristo driver sub-modules providing an interface similar to the legacy DB (concentrator) module.
-
-
-
Sources:
./nimbus/db/aristo* -
Synopsis:
Revamped implementation of a hexary Merkle Patricia Tree.
-
-
-
Sources:
./nimbus/common* -
Synopsis:
Collected information for running block chain execution layer applications.
-
-
-
Sources:
./nimbus/db/core_db* -
Synopsis:
Database abstraction layer. Unless for legacy applications, there should be no need to reach out to the layers below.
-
-
-
Sources:
./nimbus/core/executor/* ./nimbus/evm/* -
Synopsis:
An implementation of the Ethereum Virtual Machine.
-
-
-
Sources:
./vendor/nim-eth/eth/trie/hexary.nim -
Synopsis:
Implementation of an MPT, see compact Merkle Patricia Tree.
-
-
-
Sources:
./vendor/nim-eth/eth/trie/db.nim -
Synopsis:
Key value table interface to be used directly for key-value storage or by the Hexary DB (driver) module for storage. Some magic is applied in order to treat hexary data accordingly (based on key length.)
-
-
-
Sources:
./nimbus/db/kvt* -
Synopsis:
Key value table interface for the Aristo DB (driver) module. Contrary to the Key-value table (driver), it is not used for MPT data.
-
-
-
Sources:
./nimbus/db/ledger* -
Synopsis:
Abstraction layer for either the legacy cache (driver) accounts cache (which works with the legacy DB (driver) backend only) or the ledger cache (driver) re-write which is supposed to work with all Core DB (API) backends.
-
-
-
Sources:
./nimbus/db/ledger/accounts_ledger.nim
./nimbus/db/ledger/backend/accounts_ledger*
./nimbus/db/ledger/distinct_ledgers.nim -
Synopsis:
Management of accounts and storage data. This is a re-write of the legacy DB (driver) which is supposed to work with all Core DB (API) backends.
-
-
-
Sources:
./nimbus/db/core_db/backend/legacy_* -
Synopsis:
Legacy database abstraction. It mostly forwards requests directly to the to the Key-value table (driver) and/or the hexary DB (driver).
-
-
-
Sources:
./vendor/nim-rocksdb/* -
Synopsis:
Persistent storage engine.
-
-
-
Sources:
./nimbus/evm/state.nim
./nimbus/evm/types.nim -
Synopsis:
Integrated collection of modules and methods relevant for the EVM.
-