status-im
/
nimbus-eth1
mirror of https://github.com/status-im/nimbus-eth1.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth1/nimbus/db/ledger/base.nim

390 lines
14 KiB
Nim
Raw Normal View History

Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
# Nimbus
Fix styles (#2046) * Fix styles * Fix copyright year
2024-02-21 16:04:59 +00:00
# Copyright (c) 2023-2024 Status Research & Development GmbH
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or
# http://www.apache.org/licenses/LICENSE-2.0)
# * MIT license ([LICENSE-MIT](LICENSE-MIT) or
# http://opensource.org/licenses/MIT)
# at your option. This file may not be copied, modified, or distributed
# except according to those terms.
## Unify different ledger management APIs.
{.push raises: [].}
import
eth/common,
Cache code and invalid jump destination tables (fixes #2268) (#2404) It is common for many accounts to share the same code - at the database level, code is stored by hash meaning only one copy exists per unique program but when loaded in memory, a copy is made for each account. Further, every time we execute the code, it must be scanned for invalid jump destinations which slows down EVM exeuction. Finally, the extcodesize call causes code to be loaded even if only the size is needed. This PR improves on all these points by introducing a shared CodeBytesRef type whose code section is immutable and that can be shared between accounts. Further, a dedicated `len` API call is added so that the EXTCODESIZE opcode can operate without polluting the GC and code cache, for cases where only the size is requested - rocksdb will in this case cache the code itself in the row cache meaning that lookup of the code itself remains fast when length is asked for first. With 16k code entries, there's a 90% hit rate which goes up to 99% during the 2.3M attack - the cache significantly lowers memory consumption and execution time not only during this event but across the board.
2024-06-21 07:44:10 +00:00
../../evm/code_bytes,
Cleanup stateless and block witness code. (#2295) * Cleanup unneeded stateless and block witness code. Keeping MultiKeys which is used in the eth_getProofsByBlockNumber RPC endpoint which is needed for the Fluffy state network bridge. * Rename generateWitness flag to collectWitnessData to better describe what the flag does. We only collect the keys of the touched accounts and storage slots but no block witness generation is supported for now. * Move remaining stateless code into nimbus directory. * Add vmstate parameter to ChainRef to fix test. * Exclude *.in from check copyright year --------- Co-authored-by: jangko <jangko128@gmail.com>
2024-06-08 08:05:00 +00:00
../../stateless/multi_keys,
Fix default pruning for ledger and update core db and ledger logging (#1861) * Make sure that storage tries are not pruned (by default) on the new Ledger API why: Pruning might kill some unwanted entries from storage tries ending up with an unstable database leading to crashes. * Implement `CoreDb` and `LedgerRef` API tracing details: + Locally enabled at compile time via constants `ProvideCoreDbLegacyAPI` and `EnableApiTracking` in either `base.nim` source + If enabled it can be selectively turned on/off via public switches in the `CoreDb` descriptor. * Allow suppressing opportunistic `ifNecessaryGetXxx()` functions why: Better troubleshooting when the system crashes (assertions will then most probably happen outside an `async` function.)
2023-10-25 14:03:09 +00:00
../core_db,
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
./base/[api_tracking, base_desc]
const
EnableApiTracking = false
## When enabled, API functions are logged. Tracking is enabled by setting
## the `trackApi` flag to `true`.
EnableApiProfiling = true
## Enable functions profiling (only if `EnableApiTracking` is set `true`.)
apiTxt = "Ledger API"
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
type
ReadOnlyStateDB* = distinct LedgerRef
export
Cache code and invalid jump destination tables (fixes #2268) (#2404) It is common for many accounts to share the same code - at the database level, code is stored by hash meaning only one copy exists per unique program but when loaded in memory, a copy is made for each account. Further, every time we execute the code, it must be scanned for invalid jump destinations which slows down EVM exeuction. Finally, the extcodesize call causes code to be loaded even if only the size is needed. This PR improves on all these points by introducing a shared CodeBytesRef type whose code section is immutable and that can be shared between accounts. Further, a dedicated `len` API call is added so that the EXTCODESIZE opcode can operate without polluting the GC and code cache, for cases where only the size is requested - rocksdb will in this case cache the code itself in the row cache meaning that lookup of the code itself remains fast when length is asked for first. With 16k code entries, there's a 90% hit rate which goes up to 99% during the 2.3M attack - the cache significantly lowers memory consumption and execution time not only during this event but across the board.
2024-06-21 07:44:10 +00:00
code_bytes,
Coredb use stackable api for aristo backend (#2060) * Aristo/Kvt: Provide function hooks APIs why: These APIs can be used for installing tracers, profiling functoinality, and other niceties on the databases. * Aristo: Provide optional API profiling details: It basically is a re-implementation of the `CoreDb` profiling implementation * Kvt: Provide optional API profiling similar to `Aristo` * CoreDb: Re-implementing profiling using `aristo_profile` * Ledger: Re-implementing profiling using `aristo_profile` * CoreDb: Update unit tests for maintainability * update copyright dates
2024-02-29 21:10:24 +00:00
LedgerFnInx,
LedgerProfListRef,
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
LedgerRef,
Coredb use stackable api for aristo backend (#2060) * Aristo/Kvt: Provide function hooks APIs why: These APIs can be used for installing tracers, profiling functoinality, and other niceties on the databases. * Aristo: Provide optional API profiling details: It basically is a re-implementation of the `CoreDb` profiling implementation * Kvt: Provide optional API profiling similar to `Aristo` * CoreDb: Re-implementing profiling using `aristo_profile` * Ledger: Re-implementing profiling using `aristo_profile` * CoreDb: Update unit tests for maintainability * update copyright dates
2024-02-29 21:10:24 +00:00
LedgerSpRef
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
Fix default pruning for ledger and update core db and ledger logging (#1861) * Make sure that storage tries are not pruned (by default) on the new Ledger API why: Pruning might kill some unwanted entries from storage tries ending up with an unstable database leading to crashes. * Implement `CoreDb` and `LedgerRef` API tracing details: + Locally enabled at compile time via constants `ProvideCoreDbLegacyAPI` and `EnableApiTracking` in either `base.nim` source + If enabled it can be selectively turned on/off via public switches in the `CoreDb` descriptor. * Allow suppressing opportunistic `ifNecessaryGetXxx()` functions why: Better troubleshooting when the system crashes (assertions will then most probably happen outside an `async` function.)
2023-10-25 14:03:09 +00:00
const
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
LedgerEnableApiTracking* = EnableApiTracking
LedgerEnableApiProfiling* = EnableApiTracking and EnableApiProfiling
LedgerApiTxt* = apiTxt
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
Coredb use stackable api for aristo backend (#2060) * Aristo/Kvt: Provide function hooks APIs why: These APIs can be used for installing tracers, profiling functoinality, and other niceties on the databases. * Aristo: Provide optional API profiling details: It basically is a re-implementation of the `CoreDb` profiling implementation * Kvt: Provide optional API profiling similar to `Aristo` * CoreDb: Re-implementing profiling using `aristo_profile` * Ledger: Re-implementing profiling using `aristo_profile` * CoreDb: Update unit tests for maintainability * update copyright dates
2024-02-29 21:10:24 +00:00
proc ldgProfData*(db: CoreDbRef): LedgerProfListRef {.gcsafe.}
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
# ------------------------------------------------------------------------------
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
# Logging/tracking helpers (some public)
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
# ------------------------------------------------------------------------------
Fix default pruning for ledger and update core db and ledger logging (#1861) * Make sure that storage tries are not pruned (by default) on the new Ledger API why: Pruning might kill some unwanted entries from storage tries ending up with an unstable database leading to crashes. * Implement `CoreDb` and `LedgerRef` API tracing details: + Locally enabled at compile time via constants `ProvideCoreDbLegacyAPI` and `EnableApiTracking` in either `base.nim` source + If enabled it can be selectively turned on/off via public switches in the `CoreDb` descriptor. * Allow suppressing opportunistic `ifNecessaryGetXxx()` functions why: Better troubleshooting when the system crashes (assertions will then most probably happen outside an `async` function.)
2023-10-25 14:03:09 +00:00
when EnableApiTracking:
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
when EnableApiProfiling:
{.warning: "*** Provided API profiling for Ledger (disabled by default)".}
else:
{.warning: "*** Provided API logging for Ledger (disabled by default)".}
Fix default pruning for ledger and update core db and ledger logging (#1861) * Make sure that storage tries are not pruned (by default) on the new Ledger API why: Pruning might kill some unwanted entries from storage tries ending up with an unstable database leading to crashes. * Implement `CoreDb` and `LedgerRef` API tracing details: + Locally enabled at compile time via constants `ProvideCoreDbLegacyAPI` and `EnableApiTracking` in either `base.nim` source + If enabled it can be selectively turned on/off via public switches in the `CoreDb` descriptor. * Allow suppressing opportunistic `ifNecessaryGetXxx()` functions why: Better troubleshooting when the system crashes (assertions will then most probably happen outside an `async` function.)
2023-10-25 14:03:09 +00:00
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
import
std/times,
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
chronicles
Fix default pruning for ledger and update core db and ledger logging (#1861) * Make sure that storage tries are not pruned (by default) on the new Ledger API why: Pruning might kill some unwanted entries from storage tries ending up with an unstable database leading to crashes. * Implement `CoreDb` and `LedgerRef` API tracing details: + Locally enabled at compile time via constants `ProvideCoreDbLegacyAPI` and `EnableApiTracking` in either `base.nim` source + If enabled it can be selectively turned on/off via public switches in the `CoreDb` descriptor. * Allow suppressing opportunistic `ifNecessaryGetXxx()` functions why: Better troubleshooting when the system crashes (assertions will then most probably happen outside an `async` function.)
2023-10-25 14:03:09 +00:00
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
func `$`(e: Duration): string {.used.} = e.toStr
func `$`(c: CoreDbMptRef): string {.used.} = c.toStr
func `$`(l: seq[Log]): string {.used.} = l.toStr
func `$`(h: Hash256): string {.used.} = h.toStr
func `$`(a: EthAddress): string {.used.} = a.toStr
Optional accounts cache module for creating genesis (#1897) * Split off `ReadOnlyStateDB` from `AccountStateDB` from `state_db.nim` why: Apart from testing, applications use `ReadOnlyStateDB` as an easy way to access the accounts ledger. This is well supported by the `Aristo` db, but writable mode is only parially supported. The writable AccountStateDB` object for modifying accounts is not used by production code. So, for lecgacy and testing apps, the full support of the previous `AccountStateDB` is now enabled by `import db/state_db/read_write` and the `import db/state_db` provides read-only mode. * Encapsulate `AccountStateDB` as `GenesisLedgerRef` or genesis creation why: `AccountStateDB` has poor support for `Aristo` and is not widely used in favour of `AccountsLedger` (which will be abstracted as `ledger`.) Currently, using other than the `AccountStateDB` ledgers within the `GenesisLedgerRef` wrapper is experimental and test only. Eventually, the wrapper should disappear so that the `Ledger` object (which encapsulates `AccountsCache` and `AccountsLedger`) will prevail. * For the `Ledger`, provide access to raw accounts `MPT` why: This gives to the `CoreDbMptRef` descriptor from the `CoreDb` (which is the legacy version of CoreDxMptRef`.) For the new `ledger` API, the accounts are based on the `CoreDxMAccRef` descriptor which uses a particular sub-system for accounts while legacy applications use the `CoreDbPhkRef` equivalent of the `SecureHexaryTrie`. The only place where this feature will currently be used is the `genesis.nim` source file. * Fix `Aristo` bugs, missing boundary checks, typos, etc. * Verify root vertex in `MPT` and account constructors why: Was missing so far, in particular the accounts constructor must verify `VertexID(1) * Fix include file
2023-11-20 11:51:43 +00:00
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
# Publicly available for API logging
template beginTrackApi*(ldg: LedgerRef; s: LedgerFnInx) =
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
when EnableApiTracking:
Coredb use stackable api for aristo backend (#2060) * Aristo/Kvt: Provide function hooks APIs why: These APIs can be used for installing tracers, profiling functoinality, and other niceties on the databases. * Aristo: Provide optional API profiling details: It basically is a re-implementation of the `CoreDb` profiling implementation * Kvt: Provide optional API profiling similar to `Aristo` * CoreDb: Re-implementing profiling using `aristo_profile` * Ledger: Re-implementing profiling using `aristo_profile` * CoreDb: Update unit tests for maintainability * update copyright dates
2024-02-29 21:10:24 +00:00
ldg.beginApi(s)
Fix default pruning for ledger and update core db and ledger logging (#1861) * Make sure that storage tries are not pruned (by default) on the new Ledger API why: Pruning might kill some unwanted entries from storage tries ending up with an unstable database leading to crashes. * Implement `CoreDb` and `LedgerRef` API tracing details: + Locally enabled at compile time via constants `ProvideCoreDbLegacyAPI` and `EnableApiTracking` in either `base.nim` source + If enabled it can be selectively turned on/off via public switches in the `CoreDb` descriptor. * Allow suppressing opportunistic `ifNecessaryGetXxx()` functions why: Better troubleshooting when the system crashes (assertions will then most probably happen outside an `async` function.)
2023-10-25 14:03:09 +00:00
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
template ifTrackApi*(ldg: LedgerRef; code: untyped) =
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
when EnableApiTracking:
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.endApiIf:
code
Fix default pruning for ledger and update core db and ledger logging (#1861) * Make sure that storage tries are not pruned (by default) on the new Ledger API why: Pruning might kill some unwanted entries from storage tries ending up with an unstable database leading to crashes. * Implement `CoreDb` and `LedgerRef` API tracing details: + Locally enabled at compile time via constants `ProvideCoreDbLegacyAPI` and `EnableApiTracking` in either `base.nim` source + If enabled it can be selectively turned on/off via public switches in the `CoreDb` descriptor. * Allow suppressing opportunistic `ifNecessaryGetXxx()` functions why: Better troubleshooting when the system crashes (assertions will then most probably happen outside an `async` function.)
2023-10-25 14:03:09 +00:00
# ------------------------------------------------------------------------------
# Public constructor helper
# ------------------------------------------------------------------------------
proc bless*(ldg: LedgerRef; db: CoreDbRef): LedgerRef =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgBlessFn
Fix default pruning for ledger and update core db and ledger logging (#1861) * Make sure that storage tries are not pruned (by default) on the new Ledger API why: Pruning might kill some unwanted entries from storage tries ending up with an unstable database leading to crashes. * Implement `CoreDb` and `LedgerRef` API tracing details: + Locally enabled at compile time via constants `ProvideCoreDbLegacyAPI` and `EnableApiTracking` in either `base.nim` source + If enabled it can be selectively turned on/off via public switches in the `CoreDb` descriptor. * Allow suppressing opportunistic `ifNecessaryGetXxx()` functions why: Better troubleshooting when the system crashes (assertions will then most probably happen outside an `async` function.)
2023-10-25 14:03:09 +00:00
when EnableApiTracking:
ldg.trackApi = db.trackLedgerApi
Coredb use stackable api for aristo backend (#2060) * Aristo/Kvt: Provide function hooks APIs why: These APIs can be used for installing tracers, profiling functoinality, and other niceties on the databases. * Aristo: Provide optional API profiling details: It basically is a re-implementation of the `CoreDb` profiling implementation * Kvt: Provide optional API profiling similar to `Aristo` * CoreDb: Re-implementing profiling using `aristo_profile` * Ledger: Re-implementing profiling using `aristo_profile` * CoreDb: Update unit tests for maintainability * update copyright dates
2024-02-29 21:10:24 +00:00
when LedgerEnableApiProfiling:
ldg.profTab = db.ldgProfData()
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, ldgType=ldg.ldgType
Fix default pruning for ledger and update core db and ledger logging (#1861) * Make sure that storage tries are not pruned (by default) on the new Ledger API why: Pruning might kill some unwanted entries from storage tries ending up with an unstable database leading to crashes. * Implement `CoreDb` and `LedgerRef` API tracing details: + Locally enabled at compile time via constants `ProvideCoreDbLegacyAPI` and `EnableApiTracking` in either `base.nim` source + If enabled it can be selectively turned on/off via public switches in the `CoreDb` descriptor. * Allow suppressing opportunistic `ifNecessaryGetXxx()` functions why: Better troubleshooting when the system crashes (assertions will then most probably happen outside an `async` function.)
2023-10-25 14:03:09 +00:00
ldg
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
# ------------------------------------------------------------------------------
# Public methods
# ------------------------------------------------------------------------------
Coredb use stackable api for aristo backend (#2060) * Aristo/Kvt: Provide function hooks APIs why: These APIs can be used for installing tracers, profiling functoinality, and other niceties on the databases. * Aristo: Provide optional API profiling details: It basically is a re-implementation of the `CoreDb` profiling implementation * Kvt: Provide optional API profiling similar to `Aristo` * CoreDb: Re-implementing profiling using `aristo_profile` * Ledger: Re-implementing profiling using `aristo_profile` * CoreDb: Update unit tests for maintainability * update copyright dates
2024-02-29 21:10:24 +00:00
proc ldgProfData*(db: CoreDbRef): LedgerProfListRef =
## Return profiling data table (only available in profiling mode). If
## available (i.e. non-nil), result data can be organised by the functions
## available with `aristo_profile`.
##
## Note that profiling these data have accumulated over several ledger
## sessions running on the same `CoreDb` instance.
##
when LedgerEnableApiProfiling:
if db.ledgerHook.isNil:
db.ledgerHook = LedgerProfListRef.init()
cast[LedgerProfListRef](db.ledgerHook)
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc accessList*(ldg: LedgerRef, eAddr: EthAddress) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgAccessListFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.accessList(eAddr)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc accessList*(ldg: LedgerRef, eAddr: EthAddress, slot: UInt256) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgAccessListFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.accessList(eAddr, slot)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, slot
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc accountExists*(ldg: LedgerRef, eAddr: EthAddress): bool =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgAccountExistsFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.accountExists(eAddr)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, result
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc addBalance*(ldg: LedgerRef, eAddr: EthAddress, delta: UInt256) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgAddBalanceFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.addBalance(eAddr, delta)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, delta
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc addLogEntry*(ldg: LedgerRef, log: Log) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgAddLogEntryFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.addLogEntry(log)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc beginSavepoint*(ldg: LedgerRef): LedgerSpRef =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgBeginSavepointFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.beginSavepoint()
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc clearStorage*(ldg: LedgerRef, eAddr: EthAddress) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgClearStorageFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.clearStorage(eAddr)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc clearTransientStorage*(ldg: LedgerRef) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgClearTransientStorageFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.clearTransientStorage()
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc collectWitnessData*(ldg: LedgerRef) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgCollectWitnessDataFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.collectWitnessData()
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc commit*(ldg: LedgerRef, sp: LedgerSpRef) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgCommitFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.commit(sp)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc deleteAccount*(ldg: LedgerRef, eAddr: EthAddress) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgDeleteAccountFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.deleteAccount(eAddr)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc dispose*(ldg: LedgerRef, sp: LedgerSpRef) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgDisposeFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.dispose(sp)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc getAndClearLogEntries*(ldg: LedgerRef): seq[Log] =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgGetAndClearLogEntriesFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.getAndClearLogEntries()
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc getBalance*(ldg: LedgerRef, eAddr: EthAddress): UInt256 =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgGetBalanceFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.getBalance(eAddr)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, result
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
Cache code and invalid jump destination tables (fixes #2268) (#2404) It is common for many accounts to share the same code - at the database level, code is stored by hash meaning only one copy exists per unique program but when loaded in memory, a copy is made for each account. Further, every time we execute the code, it must be scanned for invalid jump destinations which slows down EVM exeuction. Finally, the extcodesize call causes code to be loaded even if only the size is needed. This PR improves on all these points by introducing a shared CodeBytesRef type whose code section is immutable and that can be shared between accounts. Further, a dedicated `len` API call is added so that the EXTCODESIZE opcode can operate without polluting the GC and code cache, for cases where only the size is requested - rocksdb will in this case cache the code itself in the row cache meaning that lookup of the code itself remains fast when length is asked for first. With 16k code entries, there's a 90% hit rate which goes up to 99% during the 2.3M attack - the cache significantly lowers memory consumption and execution time not only during this event but across the board.
2024-06-21 07:44:10 +00:00
proc getCode*(ldg: LedgerRef, eAddr: EthAddress): CodeBytesRef =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgGetCodeFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.getCode(eAddr)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, result=result.toStr
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc getCodeHash*(ldg: LedgerRef, eAddr: EthAddress): Hash256 =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgGetCodeHashFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.getCodeHash(eAddr)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, result
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc getCodeSize*(ldg: LedgerRef, eAddr: EthAddress): int =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgGetCodeSizeFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.getCodeSize(eAddr)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, result
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
proc getCommittedStorage*(
ldg: LedgerRef;
eAddr: EthAddress;
slot: UInt256;
): UInt256 =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgGetCommittedStorageFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.getCommittedStorage(eAddr, slot)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, slot, result
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc getNonce*(ldg: LedgerRef, eAddr: EthAddress): AccountNonce =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgGetNonceFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.getNonce(eAddr)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, result
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc getStorage*(ldg: LedgerRef, eAddr: EthAddress, slot: UInt256): UInt256 =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgGetStorageFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.getStorage(eAddr, slot)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, slot, result
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc getStorageRoot*(ldg: LedgerRef, eAddr: EthAddress): Hash256 =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgGetStorageRootFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.getStorageRoot(eAddr)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, result
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
proc getTransientStorage*(
ldg: LedgerRef;
eAddr: EthAddress;
slot: UInt256;
): UInt256 =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgGetTransientStorageFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.getTransientStorage(eAddr, slot)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, slot, result
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
Rename hasCodeOrNonce to contractCollision
2024-04-16 02:31:10 +00:00
proc contractCollision*(ldg: LedgerRef, eAddr: EthAddress): bool =
ldg.beginTrackApi LdgContractCollisionFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.contractCollision(eAddr)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, result
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc inAccessList*(ldg: LedgerRef, eAddr: EthAddress): bool =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgInAccessListFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.inAccessList(eAddr)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, result
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc inAccessList*(ldg: LedgerRef, eAddr: EthAddress, slot: UInt256): bool =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgInAccessListFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.inAccessList(eAddr, slot)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, slot, result
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc incNonce*(ldg: LedgerRef, eAddr: EthAddress) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgIncNonceFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.incNonce(eAddr)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc isDeadAccount*(ldg: LedgerRef, eAddr: EthAddress): bool =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgIsDeadAccountFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.isDeadAccount(eAddr)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, result
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc isEmptyAccount*(ldg: LedgerRef, eAddr: EthAddress): bool =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgIsEmptyAccountFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.isEmptyAccount(eAddr)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, result
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc isTopLevelClean*(ldg: LedgerRef): bool =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgIsTopLevelCleanFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.isTopLevelClean()
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, result
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc logEntries*(ldg: LedgerRef): seq[Log] =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgLogEntriesFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.logEntries()
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, result
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
Fix styles (#2046) * Fix styles * Fix copyright year
2024-02-21 16:04:59 +00:00
proc makeMultiKeys*(ldg: LedgerRef): MultiKeysRef =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgMakeMultiKeysFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.makeMultiKeys()
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
Clear account cache after each block (#2411) When processing long ranges of blocks, the account cache grows unbounded which cause huge memory spikes. Here, we move the cache to a second-level cache after each block - the second-level cache is cleared on the next block after that which creates a simple LRU effect. There's a small performance cost of course, though overall the freed-up memory can now be reassigned to the rocksdb row cache which not only makes up for the loss but overall leads to a performance increase. The bump to 2gb of rocksdb row cache here needs more testing but is slightly less and loosely basedy on the savings from this PR and the circular ref fix in #2408 - another way to phrase this is that it's better to give rocksdb more breathing room than let the memory sit unused until circular ref collection happens ;)
2024-06-25 05:30:32 +00:00
proc persist*(ldg: LedgerRef, clearEmptyAccount = false, clearCache = false) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgPersistFn
Clear account cache after each block (#2411) When processing long ranges of blocks, the account cache grows unbounded which cause huge memory spikes. Here, we move the cache to a second-level cache after each block - the second-level cache is cleared on the next block after that which creates a simple LRU effect. There's a small performance cost of course, though overall the freed-up memory can now be reassigned to the rocksdb row cache which not only makes up for the loss but overall leads to a performance increase. The bump to 2gb of rocksdb row cache here needs more testing but is slightly less and loosely basedy on the savings from this PR and the circular ref fix in #2408 - another way to phrase this is that it's better to give rocksdb more breathing room than let the memory sit unused until circular ref collection happens ;)
2024-06-25 05:30:32 +00:00
ldg.ac.persist(clearEmptyAccount, clearCache)
ldg.ifTrackApi: debug apiTxt, api, elapsed, clearEmptyAccount, clearCache
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc ripemdSpecial*(ldg: LedgerRef) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgRipemdSpecialFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.ripemdSpecial()
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc rollback*(ldg: LedgerRef, sp: LedgerSpRef) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgRollbackFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.rollback(sp)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc safeDispose*(ldg: LedgerRef, sp: LedgerSpRef) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgSafeDisposeFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.safeDispose(sp)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc selfDestruct*(ldg: LedgerRef, eAddr: EthAddress) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgSelfDestructFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.selfDestruct(eAddr)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc selfDestruct6780*(ldg: LedgerRef, eAddr: EthAddress) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgSelfDestruct6780Fn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.selfDestruct6780(eAddr)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc selfDestructLen*(ldg: LedgerRef): int =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgSelfDestructLenFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.selfDestructLen()
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, result
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc setBalance*(ldg: LedgerRef, eAddr: EthAddress, balance: UInt256) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgSetBalanceFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.setBalance(eAddr, balance)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, balance
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc setCode*(ldg: LedgerRef, eAddr: EthAddress, code: Blob) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgSetCodeFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.setCode(eAddr, code)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, code=code.toStr
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc setNonce*(ldg: LedgerRef, eAddr: EthAddress, nonce: AccountNonce) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgSetNonceFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.setNonce(eAddr, nonce)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, nonce
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc setStorage*(ldg: LedgerRef, eAddr: EthAddress, slot, val: UInt256) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgSetStorageFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.setStorage(eAddr, slot, val)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, slot, val
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
proc setTransientStorage*(
ldg: LedgerRef;
eAddr: EthAddress;
slot: UInt256;
val: UInt256;
) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgSetTransientStorageFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.setTransientStorage(eAddr, slot, val)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, slot, val
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
Misc updates for full sync (#2140) * Code cosmetics * Aristo+Kvt: Fix api wrappers why: Api setup killed the backend descriptor when backend mapping was disabled. * Aristo: Implement masked profiling entries why: Database backend should be listed but not counted in tally * CoreDb: Simplify backend() methods why: DBMS backend access Was provided very early and over engineered. Now there are only two backend machines, one for `Kvt` and the other one for an `Mpt` available only via new API. * CoreDb: Code cleanup regarding descriptor types * CoreDb: Refactor/redefine `persistent()` methods why: There were `persistent()` methods for any type of caching storage facilities `Kvt`, `Mpt`, `Phk`, and `Acc`. Now there is only a single `persistent()` method storing all facilities in tandem (similar to how transactions work.) For non shared `Kvt` tables, there is now an extra storage method `saveOffSite()`. * CoreDb lingo update: `trie` becomes `column` why: Notion of a `trie` is pretty much hidden by the new `CoreDb` api. Revealed are sort of database columns for accounts an storage data, any of which have an internal state represented by a Keccack hash. So a `trie` or `MPT` becomes a `column` and a `rootHash` becomes a column state. * Aristo: rename backend filed `filters` => `journal` * Update full sync logging details: + Disable eth handler noise while syncing + Log journal depth (if available) * Fix copyright year * Fix cruft and unwanted imports
2024-04-19 18:37:27 +00:00
proc state*(ldg: LedgerRef): Hash256 =
ldg.beginTrackApi LdgStateFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.state()
Misc updates for full sync (#2140) * Code cosmetics * Aristo+Kvt: Fix api wrappers why: Api setup killed the backend descriptor when backend mapping was disabled. * Aristo: Implement masked profiling entries why: Database backend should be listed but not counted in tally * CoreDb: Simplify backend() methods why: DBMS backend access Was provided very early and over engineered. Now there are only two backend machines, one for `Kvt` and the other one for an `Mpt` available only via new API. * CoreDb: Code cleanup regarding descriptor types * CoreDb: Refactor/redefine `persistent()` methods why: There were `persistent()` methods for any type of caching storage facilities `Kvt`, `Mpt`, `Phk`, and `Acc`. Now there is only a single `persistent()` method storing all facilities in tandem (similar to how transactions work.) For non shared `Kvt` tables, there is now an extra storage method `saveOffSite()`. * CoreDb lingo update: `trie` becomes `column` why: Notion of a `trie` is pretty much hidden by the new `CoreDb` api. Revealed are sort of database columns for accounts an storage data, any of which have an internal state represented by a Keccack hash. So a `trie` or `MPT` becomes a `column` and a `rootHash` becomes a column state. * Aristo: rename backend filed `filters` => `journal` * Update full sync logging details: + Disable eth handler noise while syncing + Log journal depth (if available) * Fix copyright year * Fix cruft and unwanted imports
2024-04-19 18:37:27 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, result
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc subBalance*(ldg: LedgerRef, eAddr: EthAddress, delta: UInt256) =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgSubBalanceFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
ldg.ac.subBalance(eAddr, delta)
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, eAddr, delta
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
Implement getAccessList of db/ledger (#2055)
2024-02-29 11:16:47 +00:00
proc getAccessList*(ldg: LedgerRef): AccessList =
ldg.beginTrackApi LdgGetAccessListFn
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
result = ldg.ac.getAccessList()
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed
Implement getAccessList of db/ledger (#2055)
2024-02-29 11:16:47 +00:00
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
proc rootHash*(ldg: LedgerRef): KeccakHash =
ldg.state()
Misc updates for full sync (#2140) * Code cosmetics * Aristo+Kvt: Fix api wrappers why: Api setup killed the backend descriptor when backend mapping was disabled. * Aristo: Implement masked profiling entries why: Database backend should be listed but not counted in tally * CoreDb: Simplify backend() methods why: DBMS backend access Was provided very early and over engineered. Now there are only two backend machines, one for `Kvt` and the other one for an `Mpt` available only via new API. * CoreDb: Code cleanup regarding descriptor types * CoreDb: Refactor/redefine `persistent()` methods why: There were `persistent()` methods for any type of caching storage facilities `Kvt`, `Mpt`, `Phk`, and `Acc`. Now there is only a single `persistent()` method storing all facilities in tandem (similar to how transactions work.) For non shared `Kvt` tables, there is now an extra storage method `saveOffSite()`. * CoreDb lingo update: `trie` becomes `column` why: Notion of a `trie` is pretty much hidden by the new `CoreDb` api. Revealed are sort of database columns for accounts an storage data, any of which have an internal state represented by a Keccack hash. So a `trie` or `MPT` becomes a `column` and a `rootHash` becomes a column state. * Aristo: rename backend filed `filters` => `journal` * Update full sync logging details: + Disable eth handler noise while syncing + Log journal depth (if available) * Fix copyright year * Fix cruft and unwanted imports
2024-04-19 18:37:27 +00:00
Coredb maintenance (#2398) * CoreDb: remove PHK tries why: There is no general use anymore for an MPT with a pre-hashed key. It was used to resemble the `SecureHexaryTrie` logic from the legacy DB. The only pace where this is needed is the `Leger` which uses a a distinct MPT version anyway (see `distinct_ledgers.nim`.) * Rename `CoreDx*` -> `CoreDb*` why: The naming `CoreDx*` was used to differentiate the new CoreDb API from the legacy API which had descriptors named `CoreDb*`.
2024-06-19 14:13:12 +00:00
proc getMpt*(ldg: LedgerRef): CoreDbMptRef =
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
ldg.beginTrackApi LdgGetMptFn
Coredb maintenance (#2398) * CoreDb: remove PHK tries why: There is no general use anymore for an MPT with a pre-hashed key. It was used to resemble the `SecureHexaryTrie` logic from the legacy DB. The only pace where this is needed is the `Leger` which uses a a distinct MPT version anyway (see `distinct_ledgers.nim`.) * Rename `CoreDx*` -> `CoreDb*` why: The naming `CoreDx*` was used to differentiate the new CoreDb API from the legacy API which had descriptors named `CoreDb*`.
2024-06-19 14:13:12 +00:00
result = ldg.ac.rawTrie.CoreDbAccRef.getMpt
Core db implement ctx layer for mpt state admin (#2082) * CoreDb+Ledger: Update logging why: Use symbol `api` rather than `ctx` because the latter will be used as name for particular objects * CoreDb: Remove cruft * CoreDb: Remove `TxID` support why: It is nowhere used and ugly implemented. The upcoming context layer will be a cleaner alternative to use, instead should this particular functionality be needed. * CoreDb: Rearrange base methods in source code for better reading * CoreDb+Aristo: Update API closures for better reading & maintenance * CoreDb: Implement context layer for MPT why: On `Aristo` the context layer allows to manage different views on the same backend database. This is an abstraction of the legacy hexary trie which can be localised on a particular root nose. details: The `ctx` context provides the state (equiv. to state root) of the database for MPT and account descriptors. * Fix Copyright headers
2024-03-18 19:40:23 +00:00
ldg.ifTrackApi: debug apiTxt, api, elapsed, result
Optional accounts cache module for creating genesis (#1897) * Split off `ReadOnlyStateDB` from `AccountStateDB` from `state_db.nim` why: Apart from testing, applications use `ReadOnlyStateDB` as an easy way to access the accounts ledger. This is well supported by the `Aristo` db, but writable mode is only parially supported. The writable AccountStateDB` object for modifying accounts is not used by production code. So, for lecgacy and testing apps, the full support of the previous `AccountStateDB` is now enabled by `import db/state_db/read_write` and the `import db/state_db` provides read-only mode. * Encapsulate `AccountStateDB` as `GenesisLedgerRef` or genesis creation why: `AccountStateDB` has poor support for `Aristo` and is not widely used in favour of `AccountsLedger` (which will be abstracted as `ledger`.) Currently, using other than the `AccountStateDB` ledgers within the `GenesisLedgerRef` wrapper is experimental and test only. Eventually, the wrapper should disappear so that the `Ledger` object (which encapsulates `AccountsCache` and `AccountsLedger`) will prevail. * For the `Ledger`, provide access to raw accounts `MPT` why: This gives to the `CoreDbMptRef` descriptor from the `CoreDb` (which is the legacy version of CoreDxMptRef`.) For the new `ledger` API, the accounts are based on the `CoreDxMAccRef` descriptor which uses a particular sub-system for accounts while legacy applications use the `CoreDbPhkRef` equivalent of the `SecureHexaryTrie`. The only place where this feature will currently be used is the `genesis.nim` source file. * Fix `Aristo` bugs, missing boundary checks, typos, etc. * Verify root vertex in `MPT` and account constructors why: Was missing so far, in particular the accounts constructor must verify `VertexID(1) * Fix include file
2023-11-20 11:51:43 +00:00
Remove AccountStateDB (#2368) * Remove AccountStateDB AccountStateDB should no longer be used. It's usage have been reduce to read only operations. Replace it with LedgerRef to reduce maintenance burden. * remove extra spaces Co-authored-by: tersec <tersec@users.noreply.github.com> --------- Co-authored-by: tersec <tersec@users.noreply.github.com>
2024-06-16 03:21:02 +00:00
proc getEthAccount*(ldg: LedgerRef, eAddr: EthAddress): Account =
ldg.beginTrackApi LdgGetAthAccountFn
result = ldg.ac.getEthAccount(eAddr)
ldg.ifTrackApi: debug apiTxt, api, elapsed, result
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
# ------------------------------------------------------------------------------
# Public virtual read-only methods
# ------------------------------------------------------------------------------
Simplify AccountsLedgerRef complexity (#2239)
2024-05-29 11:06:49 +00:00
proc rootHash*(db: ReadOnlyStateDB): KeccakHash = db.LedgerRef.state()
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc getCodeHash*(db: ReadOnlyStateDB, eAddr: EthAddress): Hash256 {.borrow.}
proc getStorageRoot*(db: ReadOnlyStateDB, eAddr: EthAddress): Hash256 {.borrow.}
proc getBalance*(db: ReadOnlyStateDB, eAddr: EthAddress): UInt256 {.borrow.}
proc getStorage*(db: ReadOnlyStateDB, eAddr: EthAddress, slot: UInt256): UInt256 {.borrow.}
proc getNonce*(db: ReadOnlyStateDB, eAddr: EthAddress): AccountNonce {.borrow.}
Cache code and invalid jump destination tables (fixes #2268) (#2404) It is common for many accounts to share the same code - at the database level, code is stored by hash meaning only one copy exists per unique program but when loaded in memory, a copy is made for each account. Further, every time we execute the code, it must be scanned for invalid jump destinations which slows down EVM exeuction. Finally, the extcodesize call causes code to be loaded even if only the size is needed. This PR improves on all these points by introducing a shared CodeBytesRef type whose code section is immutable and that can be shared between accounts. Further, a dedicated `len` API call is added so that the EXTCODESIZE opcode can operate without polluting the GC and code cache, for cases where only the size is requested - rocksdb will in this case cache the code itself in the row cache meaning that lookup of the code itself remains fast when length is asked for first. With 16k code entries, there's a 90% hit rate which goes up to 99% during the 2.3M attack - the cache significantly lowers memory consumption and execution time not only during this event but across the board.
2024-06-21 07:44:10 +00:00
proc getCode*(db: ReadOnlyStateDB, eAddr: EthAddress): CodeBytesRef {.borrow.}
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc getCodeSize*(db: ReadOnlyStateDB, eAddr: EthAddress): int {.borrow.}
Rename hasCodeOrNonce to contractCollision
2024-04-16 02:31:10 +00:00
proc contractCollision*(db: ReadOnlyStateDB, eAddr: EthAddress): bool {.borrow.}
Ledger abstraction for accounts cache (#1824) * Provide TDD/debug facility for inspecting `persistBlocks()` working detail: + Make sure that the last block of a test sample is the first batch item in `persistBlocks()`. + Additionally, allow `AccountsCache` API tracing by setting the flag `extraTraceMessages = true` in the file `accounts_cache.nim` * Overload AccountsCache by abstraction wrapper details: Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 19:27:22 +00:00
proc accountExists*(db: ReadOnlyStateDB, eAddr: EthAddress): bool {.borrow.}
proc isDeadAccount*(db: ReadOnlyStateDB, eAddr: EthAddress): bool {.borrow.}
proc isEmptyAccount*(db: ReadOnlyStateDB, eAddr: EthAddress): bool {.borrow.}
proc getCommittedStorage*(db: ReadOnlyStateDB, eAddr: EthAddress, slot: UInt256): UInt256 {.borrow.}
func inAccessList*(db: ReadOnlyStateDB, eAddr: EthAddress): bool {.borrow.}
func inAccessList*(db: ReadOnlyStateDB, eAddr: EthAddress, slot: UInt256): bool {.borrow.}
func getTransientStorage*(db: ReadOnlyStateDB, eAddr: EthAddress, slot: UInt256): UInt256 {.borrow.}
# ------------------------------------------------------------------------------
# End
# ------------------------------------------------------------------------------