nimbus-eth1/nimbus/db/core_db
Jacek Sieka 9521582005
avoid closure environment for mpt methods (#2408)
An instance of `CoreDbMptRef` is created for and stored in every account
- when we are processing blocks and have many accounts in memory, this
closure environment takes up hundreds of mb of memory (around block 5M,
it is the 4:th largest memory consumer!) - incidentally, this also
removes a circular reference in the setup that causes the
`AristoCodeDbMptRef` to linger in memory much longer than it
has to which is the core reason why it takes so much.

The real solution here is to remove the methods indirection entirely,
but this PR provides relief until that has been done.

Similar treatment is given to some of the other core api functions to
avoid circulars there too.
2024-06-24 07:56:41 +02:00
..
backend avoid closure environment for mpt methods (#2408) 2024-06-24 07:56:41 +02:00
base avoid closure environment for mpt methods (#2408) 2024-06-24 07:56:41 +02:00
README.md Core db+aristo provides tracer funtionality (#2089) 2024-03-21 10:45:57 +00:00
TODO.md Aristo resume off line syncing on pre loaded database (#2203) 2024-05-22 13:41:14 +00:00
base.nim avoid closure environment for mpt methods (#2408) 2024-06-24 07:56:41 +02:00
base_iterators.nim Coredb maintenance (#2398) 2024-06-19 14:13:12 +00:00
base_iterators_persistent.nim Coredb maintenance (#2398) 2024-06-19 14:13:12 +00:00
core_apps.nim Remove AccountStateDB (#2368) 2024-06-16 10:21:02 +07:00
memory_only.nim Coeredb related clean up and maint fixes (#2360) 2024-06-14 11:19:48 +00:00
persistent.nim Coeredb related clean up and maint fixes (#2360) 2024-06-14 11:19:48 +00:00

README.md

Core database replacement wrapper object

This wrapper replaces the TrieDatabaseRef and its derivatives by the new object CoreDbRef.

Relations to current TrieDatabaseRef implementation

Here are some incomplete translations for objects and constructors.

Object types:

Legacy notation CoreDbRef based replacement
ChainDB (don't use/avoid)
ChainDbRef CoreDbRef
TrieDatabaseRef CoreDbKvtRef
HexaryTrie CoreDbMptRef
SecureHexaryTrie CoreDbPhkRef
DbTransaction CoreDbTxRef
TransactionID CoreDbTxID

Constructors:

Legacy notation CoreDbRef based replacement
trieDB newChainDB("..") newCoreDbRef(LegacyDbPersistent,"..")
newMemoryDB() newCoreDbRef(LegacyDbMemory)
--
initHexaryTrie(db,..) db.mpt(..) (no pruning)
db.mptPrune(..) (w/pruning true/false)
--
initSecureHexaryTrie(db,..) db.phk(..) (no pruning)
db.phkPrune(..) (w/pruning true/false)
--
newCaptureDB(db,memDB) db.capture() (see below)

Usage of the replacement wrapper

Objects pedigree:

    CoreDbRef                        -- base descriptor
     | | |
     | | +--- CoreDbCtxRef           -- MPT context descriptor
     | |        | |
     | |        | +-- CoreDbMptRef   -- hexary trie instance
     | |        | |    :                    :
     | |        | +-- CoreDbMptRef   -- hexary trie instance
     | |        |
     | |        |
     | |        +---- CoreDbPhkRef   -- pre-hashed key hexary trie instance
     | |        |      :                    :
     | |        +---- CoreDbPhkRef   -- pre-hashed key hexary trie instance
     | |
     | |
     | +------ CoreDbKvtRef          -- single static key-value table
     |
     |
     +-------- CoreDbCaptRef         -- tracer support descriptor

Instantiating legacy standard database object descriptors works as follows:

    let
      db = newCoreDbRef(..)           # new base descriptor
      mpt = db.mpt(..)                # hexary trie/Merkle Patricia Tree
      phk = db.phk(..)                # pre-hashed key hexary trie/MPT
      kvt = db.kvt                    # key-value table

Tracer support setup by hiding the current CoreDbRef behind a replacement:

    let
      capture = db.capture()
      db = capture.recorder           # use the recorder in place of db
    ...

    for key,value in capture.recorder.kvt:
     ...                              # process recorded data