status-im
/
nimbus-eth1
mirror of https://github.com/status-im/nimbus-eth1.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth1/nimbus/db/ledger/base/api_tracking.nim

141 lines
5.2 KiB
Nim
Raw Normal View History

Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
# Nimbus
Core db update storage root management for sub tries (#1964) * Aristo: Re-phrase `LayerDelta` and `LayerFinal` as object references why: Avoids copying in some cases * Fix copyright header * Aristo: Verify `leafTie.root` function argument for `merge()` proc why: Zero root will lead to inconsistent DB entry * Aristo: Update failure condition for hash labels compiler `hashify()` why: Node need not be rejected as long as links are on the schedule. In that case, `redo[]` is to become `wff.base[]` at a later stage. This amends an earlier fix, part of #1952 by also testing against the target nodes of the `wff.base[]` sets. * Aristo: Add storage root glue record to `hashify()` schedule why: An account leaf node might refer to a non-resolvable storage root ID. Storage root node chains will end up at the storage root. So the link `storage-root->account-leaf` needs an extra item in the schedule. * Aristo: fix error code returned by `fetchPayload()` details: Final error code is implied by the error code form the `hikeUp()` function. * CoreDb: Discard `createOk` argument in API `getRoot()` function why: Not needed for the legacy DB. For the `Arsto` DB, a lazy approach is implemented where a stprage root node is created on-the-fly. * CoreDb: Prevent `$$` logging in some cases why: Logging the function `$$` is not useful when it is used for internal use, i.e. retrieving an an error text for logging. * CoreDb: Add `tryHashFn()` to API for pretty printing why: Pretty printing must not change the hashification status for the `Aristo` DB. So there is an independent API wrapper for getting the node hash which never updated the hashes. * CoreDb: Discard `update` argument in API `hash()` function why: When calling the API function `hash()`, the latest state is always wanted. For a version that uses the current state as-is without checking, the function `tryHash()` was added to the backend. * CoreDb: Update opaque vertex ID objects for the `Aristo` backend why: For `Aristo`, vID objects encapsulate a numeric `VertexID` referencing a vertex (rather than a node hash as used on the legacy backend.) For storage sub-tries, there might be no initial vertex known when the descriptor is created. So opaque vertex ID objects are supported without a valid `VertexID` which will be initalised on-the-fly when the first item is merged. * CoreDb: Add pretty printer for opaque vertex ID objects * Cosmetics, printing profiling data * CoreDb: Fix segfault in `Aristo` backend when creating MPT descriptor why: Missing initialisation error * CoreDb: Allow MPT to inherit shared context on `Aristo` backend why: Creates descriptors with different storage roots for the same shared `Aristo` DB descriptor. * Cosmetics, update diagnostic message items for `Aristo` backend * Fix Copyright year
2024-01-11 19:11:38 +00:00
# Copyright (c) 2023-2024 Status Research & Development GmbH
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or
# http://www.apache.org/licenses/LICENSE-2.0)
# * MIT license ([LICENSE-MIT](LICENSE-MIT) or
# http://opensource.org/licenses/MIT)
# at your option. This file may not be copied, modified, or distributed
# except according to those terms.
{.push raises: [].}
import
Coredb use stackable api for aristo backend (#2060) * Aristo/Kvt: Provide function hooks APIs why: These APIs can be used for installing tracers, profiling functoinality, and other niceties on the databases. * Aristo: Provide optional API profiling details: It basically is a re-implementation of the `CoreDb` profiling implementation * Kvt: Provide optional API profiling similar to `Aristo` * CoreDb: Re-implementing profiling using `aristo_profile` * Ledger: Re-implementing profiling using `aristo_profile` * CoreDb: Update unit tests for maintainability * update copyright dates
2024-02-29 21:10:24 +00:00
std/[strutils, times],
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
eth/common,
stew/byteutils,
Coredb use stackable api for aristo backend (#2060) * Aristo/Kvt: Provide function hooks APIs why: These APIs can be used for installing tracers, profiling functoinality, and other niceties on the databases. * Aristo: Provide optional API profiling details: It basically is a re-implementation of the `CoreDb` profiling implementation * Kvt: Provide optional API profiling similar to `Aristo` * CoreDb: Re-implementing profiling using `aristo_profile` * Ledger: Re-implementing profiling using `aristo_profile` * CoreDb: Update unit tests for maintainability * update copyright dates
2024-02-29 21:10:24 +00:00
../../aristo/aristo_profile,
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
../../core_db,
"."/base_desc
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
type
LedgerFnInx* = enum
## Profiling table index
SummaryItem = "total"
LdgBlessFn = "LedgerRef.init"
Misc updates for full sync (#2140) * Code cosmetics * Aristo+Kvt: Fix api wrappers why: Api setup killed the backend descriptor when backend mapping was disabled. * Aristo: Implement masked profiling entries why: Database backend should be listed but not counted in tally * CoreDb: Simplify backend() methods why: DBMS backend access Was provided very early and over engineered. Now there are only two backend machines, one for `Kvt` and the other one for an `Mpt` available only via new API. * CoreDb: Code cleanup regarding descriptor types * CoreDb: Refactor/redefine `persistent()` methods why: There were `persistent()` methods for any type of caching storage facilities `Kvt`, `Mpt`, `Phk`, and `Acc`. Now there is only a single `persistent()` method storing all facilities in tandem (similar to how transactions work.) For non shared `Kvt` tables, there is now an extra storage method `saveOffSite()`. * CoreDb lingo update: `trie` becomes `column` why: Notion of a `trie` is pretty much hidden by the new `CoreDb` api. Revealed are sort of database columns for accounts an storage data, any of which have an internal state represented by a Keccack hash. So a `trie` or `MPT` becomes a `column` and a `rootHash` becomes a column state. * Aristo: rename backend filed `filters` => `journal` * Update full sync logging details: + Disable eth handler noise while syncing + Log journal depth (if available) * Fix copyright year * Fix cruft and unwanted imports
2024-04-19 18:37:27 +00:00
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
LdgAccessListFn = "accessList"
LdgAccountExistsFn = "accountExists"
LdgAddBalanceFn = "addBalance"
LdgAddLogEntryFn = "addLogEntry"
LdgBeginSavepointFn = "beginSavepoint"
LdgClearStorageFn = "clearStorage"
LdgClearTransientStorageFn = "clearTransientStorage"
LdgCollectWitnessDataFn = "collectWitnessData"
LdgCommitFn = "commit"
Misc updates for full sync (#2140) * Code cosmetics * Aristo+Kvt: Fix api wrappers why: Api setup killed the backend descriptor when backend mapping was disabled. * Aristo: Implement masked profiling entries why: Database backend should be listed but not counted in tally * CoreDb: Simplify backend() methods why: DBMS backend access Was provided very early and over engineered. Now there are only two backend machines, one for `Kvt` and the other one for an `Mpt` available only via new API. * CoreDb: Code cleanup regarding descriptor types * CoreDb: Refactor/redefine `persistent()` methods why: There were `persistent()` methods for any type of caching storage facilities `Kvt`, `Mpt`, `Phk`, and `Acc`. Now there is only a single `persistent()` method storing all facilities in tandem (similar to how transactions work.) For non shared `Kvt` tables, there is now an extra storage method `saveOffSite()`. * CoreDb lingo update: `trie` becomes `column` why: Notion of a `trie` is pretty much hidden by the new `CoreDb` api. Revealed are sort of database columns for accounts an storage data, any of which have an internal state represented by a Keccack hash. So a `trie` or `MPT` becomes a `column` and a `rootHash` becomes a column state. * Aristo: rename backend filed `filters` => `journal` * Update full sync logging details: + Disable eth handler noise while syncing + Log journal depth (if available) * Fix copyright year * Fix cruft and unwanted imports
2024-04-19 18:37:27 +00:00
LdgContractCollisionFn = "contractCollision"
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
LdgDeleteAccountFn = "deleteAccount"
LdgDisposeFn = "dispose"
Misc updates for full sync (#2140) * Code cosmetics * Aristo+Kvt: Fix api wrappers why: Api setup killed the backend descriptor when backend mapping was disabled. * Aristo: Implement masked profiling entries why: Database backend should be listed but not counted in tally * CoreDb: Simplify backend() methods why: DBMS backend access Was provided very early and over engineered. Now there are only two backend machines, one for `Kvt` and the other one for an `Mpt` available only via new API. * CoreDb: Code cleanup regarding descriptor types * CoreDb: Refactor/redefine `persistent()` methods why: There were `persistent()` methods for any type of caching storage facilities `Kvt`, `Mpt`, `Phk`, and `Acc`. Now there is only a single `persistent()` method storing all facilities in tandem (similar to how transactions work.) For non shared `Kvt` tables, there is now an extra storage method `saveOffSite()`. * CoreDb lingo update: `trie` becomes `column` why: Notion of a `trie` is pretty much hidden by the new `CoreDb` api. Revealed are sort of database columns for accounts an storage data, any of which have an internal state represented by a Keccack hash. So a `trie` or `MPT` becomes a `column` and a `rootHash` becomes a column state. * Aristo: rename backend filed `filters` => `journal` * Update full sync logging details: + Disable eth handler noise while syncing + Log journal depth (if available) * Fix copyright year * Fix cruft and unwanted imports
2024-04-19 18:37:27 +00:00
LdgGetAccessListFn = "getAcessList"
Culling legacy DB and accounts cache (#2197) details: + Compiles nimbus all_tests + Failing tests have been commented out
2024-05-20 10:17:51 +00:00
LdgGetAccountFn = "getAccount"
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
LdgGetAndClearLogEntriesFn = "getAndClearLogEntries"
LdgGetBalanceFn = "getBalance"
LdgGetCodeFn = "getCode"
LdgGetCodeHashFn = "getCodeHash"
LdgGetCodeSizeFn = "getCodeSize"
LdgGetCommittedStorageFn = "getCommittedStorage"
LdgGetNonceFn = "getNonce"
LdgGetStorageFn = "getStorage"
LdgGetStorageRootFn = "getStorageRoot"
LdgGetTransientStorageFn = "getTransientStorage"
Remove AccountStateDB (#2368) * Remove AccountStateDB AccountStateDB should no longer be used. It's usage have been reduce to read only operations. Replace it with LedgerRef to reduce maintenance burden. * remove extra spaces Co-authored-by: tersec <tersec@users.noreply.github.com> --------- Co-authored-by: tersec <tersec@users.noreply.github.com>
2024-06-16 03:21:02 +00:00
LdgGetAthAccountFn = "getEthAccount"
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
LdgInAccessListFn = "inAccessList"
LdgIncNonceFn = "incNonce"
LdgIsDeadAccountFn = "isDeadAccount"
LdgIsEmptyAccountFn = "isEmptyAccount"
LdgIsTopLevelCleanFn = "isTopLevelClean"
LdgLogEntriesFn = "logEntries"
LdgMakeMultiKeysFn = "makeMultiKeys"
LdgPersistFn = "persist"
Misc updates for full sync (#2140) * Code cosmetics * Aristo+Kvt: Fix api wrappers why: Api setup killed the backend descriptor when backend mapping was disabled. * Aristo: Implement masked profiling entries why: Database backend should be listed but not counted in tally * CoreDb: Simplify backend() methods why: DBMS backend access Was provided very early and over engineered. Now there are only two backend machines, one for `Kvt` and the other one for an `Mpt` available only via new API. * CoreDb: Code cleanup regarding descriptor types * CoreDb: Refactor/redefine `persistent()` methods why: There were `persistent()` methods for any type of caching storage facilities `Kvt`, `Mpt`, `Phk`, and `Acc`. Now there is only a single `persistent()` method storing all facilities in tandem (similar to how transactions work.) For non shared `Kvt` tables, there is now an extra storage method `saveOffSite()`. * CoreDb lingo update: `trie` becomes `column` why: Notion of a `trie` is pretty much hidden by the new `CoreDb` api. Revealed are sort of database columns for accounts an storage data, any of which have an internal state represented by a Keccack hash. So a `trie` or `MPT` becomes a `column` and a `rootHash` becomes a column state. * Aristo: rename backend filed `filters` => `journal` * Update full sync logging details: + Disable eth handler noise while syncing + Log journal depth (if available) * Fix copyright year * Fix cruft and unwanted imports
2024-04-19 18:37:27 +00:00
LdgRawRootHashFn = "rawRootHash"
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
LdgRipemdSpecialFn = "ripemdSpecial"
LdgRollbackFn = "rollback"
LdgRootHashFn = "rootHash"
LdgSafeDisposeFn = "safeDispose"
LdgSelfDestruct6780Fn = "selfDestruct6780"
Misc updates for full sync (#2140) * Code cosmetics * Aristo+Kvt: Fix api wrappers why: Api setup killed the backend descriptor when backend mapping was disabled. * Aristo: Implement masked profiling entries why: Database backend should be listed but not counted in tally * CoreDb: Simplify backend() methods why: DBMS backend access Was provided very early and over engineered. Now there are only two backend machines, one for `Kvt` and the other one for an `Mpt` available only via new API. * CoreDb: Code cleanup regarding descriptor types * CoreDb: Refactor/redefine `persistent()` methods why: There were `persistent()` methods for any type of caching storage facilities `Kvt`, `Mpt`, `Phk`, and `Acc`. Now there is only a single `persistent()` method storing all facilities in tandem (similar to how transactions work.) For non shared `Kvt` tables, there is now an extra storage method `saveOffSite()`. * CoreDb lingo update: `trie` becomes `column` why: Notion of a `trie` is pretty much hidden by the new `CoreDb` api. Revealed are sort of database columns for accounts an storage data, any of which have an internal state represented by a Keccack hash. So a `trie` or `MPT` becomes a `column` and a `rootHash` becomes a column state. * Aristo: rename backend filed `filters` => `journal` * Update full sync logging details: + Disable eth handler noise while syncing + Log journal depth (if available) * Fix copyright year * Fix cruft and unwanted imports
2024-04-19 18:37:27 +00:00
LdgSelfDestructFn = "selfDestruct"
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
LdgSelfDestructLenFn = "selfDestructLen"
LdgSetBalanceFn = "setBalance"
LdgSetCodeFn = "setCode"
LdgSetNonceFn = "setNonce"
LdgSetStorageFn = "setStorage"
LdgSetTransientStorageFn = "setTransientStorage"
Misc updates for full sync (#2140) * Code cosmetics * Aristo+Kvt: Fix api wrappers why: Api setup killed the backend descriptor when backend mapping was disabled. * Aristo: Implement masked profiling entries why: Database backend should be listed but not counted in tally * CoreDb: Simplify backend() methods why: DBMS backend access Was provided very early and over engineered. Now there are only two backend machines, one for `Kvt` and the other one for an `Mpt` available only via new API. * CoreDb: Code cleanup regarding descriptor types * CoreDb: Refactor/redefine `persistent()` methods why: There were `persistent()` methods for any type of caching storage facilities `Kvt`, `Mpt`, `Phk`, and `Acc`. Now there is only a single `persistent()` method storing all facilities in tandem (similar to how transactions work.) For non shared `Kvt` tables, there is now an extra storage method `saveOffSite()`. * CoreDb lingo update: `trie` becomes `column` why: Notion of a `trie` is pretty much hidden by the new `CoreDb` api. Revealed are sort of database columns for accounts an storage data, any of which have an internal state represented by a Keccack hash. So a `trie` or `MPT` becomes a `column` and a `rootHash` becomes a column state. * Aristo: rename backend filed `filters` => `journal` * Update full sync logging details: + Disable eth handler noise while syncing + Log journal depth (if available) * Fix copyright year * Fix cruft and unwanted imports
2024-04-19 18:37:27 +00:00
LdgStateFn = "state"
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
LdgSubBalanceFn = "subBalance"
LdgAccountsIt = "accounts"
LdgAdressesIt = "addresses"
LdgCachedStorageIt = "cachedStorage"
LdgPairsIt = "pairs"
LdgStorageIt = "storage"
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
# ------------------------------------------------------------------------------
# Private helpers
# ------------------------------------------------------------------------------
func oaToStr(w: openArray[byte]): string =
w.toHex.toLowerAscii
# ------------------------------------------------------------------------------
# Public API logging helpers
# ------------------------------------------------------------------------------
func toStr*(w: EthAddress): string =
w.oaToStr
func toStr*(w: Hash256): string =
w.data.oaToStr
Core db disable legacy api n remove distinct tries (#2299) * CoreDb: Remove crufty second/off-site KVT why: Was used to allow late `Clique` to store directly to disk * CoreDb: Remove prune flag related functionality why: Is completely legacy stuff * CoreDb: Remove dependence on legacy API (tests unsupported yet) why: Does not fully support Aristo * Re-factoring `state_db` using new API details: Only minimum changes needed to compile `nimbus` * Update tests and aux modules * Turn off legacy API and remove `distinct_tries` comment: The legacy API has now cruft status, will be removed soon * Fix copyright years * Update rpc for verified proxy --------- Co-authored-by: Jacek Sieka <jacek@status.im>
2024-06-05 20:52:04 +00:00
when declared(CoreDbMptRef):
func toStr*(w: CoreDbMptRef): string =
Coredb maintenance (#2398) * CoreDb: remove PHK tries why: There is no general use anymore for an MPT with a pre-hashed key. It was used to resemble the `SecureHexaryTrie` logic from the legacy DB. The only pace where this is needed is the `Leger` which uses a a distinct MPT version anyway (see `distinct_ledgers.nim`.) * Rename `CoreDx*` -> `CoreDb*` why: The naming `CoreDx*` was used to differentiate the new CoreDb API from the legacy API which had descriptors named `CoreDb*`.
2024-06-19 14:13:12 +00:00
if w.CoreDbMptRef.isNil: "MptRef(nil)" else: "MptRef"
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
func toStr*(w: Blob): string =
if 0 < w.len and w.len < 5: "<" & w.oaToStr & ">"
else: "Blob[" & $w.len & "]"
func toStr*(w: seq[Log]): string =
"Logs[" & $w.len & "]"
Coredb use stackable api for aristo backend (#2060) * Aristo/Kvt: Provide function hooks APIs why: These APIs can be used for installing tracers, profiling functoinality, and other niceties on the databases. * Aristo: Provide optional API profiling details: It basically is a re-implementation of the `CoreDb` profiling implementation * Kvt: Provide optional API profiling similar to `Aristo` * CoreDb: Re-implementing profiling using `aristo_profile` * Ledger: Re-implementing profiling using `aristo_profile` * CoreDb: Update unit tests for maintainability * update copyright dates
2024-02-29 21:10:24 +00:00
func toStr*(ela: Duration): string =
aristo_profile.toStr(ela)
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
# ------------------------------------------------------------------------------
# Public API logging framework
# ------------------------------------------------------------------------------
Coredb use stackable api for aristo backend (#2060) * Aristo/Kvt: Provide function hooks APIs why: These APIs can be used for installing tracers, profiling functoinality, and other niceties on the databases. * Aristo: Provide optional API profiling details: It basically is a re-implementation of the `CoreDb` profiling implementation * Kvt: Provide optional API profiling similar to `Aristo` * CoreDb: Re-implementing profiling using `aristo_profile` * Ledger: Re-implementing profiling using `aristo_profile` * CoreDb: Update unit tests for maintainability * update copyright dates
2024-02-29 21:10:24 +00:00
template beginApi*(ldg: LedgerRef; s: static[LedgerFnInx]) =
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
const api {.inject,used.} = s # Generally available
Coredb use stackable api for aristo backend (#2060) * Aristo/Kvt: Provide function hooks APIs why: These APIs can be used for installing tracers, profiling functoinality, and other niceties on the databases. * Aristo: Provide optional API profiling details: It basically is a re-implementation of the `CoreDb` profiling implementation * Kvt: Provide optional API profiling similar to `Aristo` * CoreDb: Re-implementing profiling using `aristo_profile` * Ledger: Re-implementing profiling using `aristo_profile` * CoreDb: Update unit tests for maintainability * update copyright dates
2024-02-29 21:10:24 +00:00
let baStart {.inject.} = getTime() # Local use only
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
template endApiIf*(ldg: LedgerRef; code: untyped) =
Coredb use stackable api for aristo backend (#2060) * Aristo/Kvt: Provide function hooks APIs why: These APIs can be used for installing tracers, profiling functoinality, and other niceties on the databases. * Aristo: Provide optional API profiling details: It basically is a re-implementation of the `CoreDb` profiling implementation * Kvt: Provide optional API profiling similar to `Aristo` * CoreDb: Re-implementing profiling using `aristo_profile` * Ledger: Re-implementing profiling using `aristo_profile` * CoreDb: Update unit tests for maintainability * update copyright dates
2024-02-29 21:10:24 +00:00
when CoreDbEnableApiProfiling:
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
let elapsed {.inject,used.} = getTime() - baStart
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
aristo_profile.update(ldg.profTab, api.ord, elapsed)
Coredb use stackable api for aristo backend (#2060) * Aristo/Kvt: Provide function hooks APIs why: These APIs can be used for installing tracers, profiling functoinality, and other niceties on the databases. * Aristo: Provide optional API profiling details: It basically is a re-implementation of the `CoreDb` profiling implementation * Kvt: Provide optional API profiling similar to `Aristo` * CoreDb: Re-implementing profiling using `aristo_profile` * Ledger: Re-implementing profiling using `aristo_profile` * CoreDb: Update unit tests for maintainability * update copyright dates
2024-02-29 21:10:24 +00:00
if ldg.trackApi:
when not CoreDbEnableApiProfiling: # otherwise use variable above
let elapsed {.inject,used.} = getTime() - baStart
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
code
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
# ------------------------------------------------------------------------------
# Public helpers
# ------------------------------------------------------------------------------
Coredb use stackable api for aristo backend (#2060) * Aristo/Kvt: Provide function hooks APIs why: These APIs can be used for installing tracers, profiling functoinality, and other niceties on the databases. * Aristo: Provide optional API profiling details: It basically is a re-implementation of the `CoreDb` profiling implementation * Kvt: Provide optional API profiling similar to `Aristo` * CoreDb: Re-implementing profiling using `aristo_profile` * Ledger: Re-implementing profiling using `aristo_profile` * CoreDb: Update unit tests for maintainability * update copyright dates
2024-02-29 21:10:24 +00:00
func init*(T: type LedgerProfListRef): T =
T(list: newSeq[LedgerProfData](1 + high(LedgerFnInx).ord))
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
# ------------------------------------------------------------------------------
# End
# ------------------------------------------------------------------------------