status-im
/
nimbus-eth1
mirror of https://github.com/status-im/nimbus-eth1.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth1/nimbus/db/core_db/base/validate.nim

157 lines
4.8 KiB
Nim
Raw Normal View History

Core db update storage root management for sub tries (#1964) * Aristo: Re-phrase `LayerDelta` and `LayerFinal` as object references why: Avoids copying in some cases * Fix copyright header * Aristo: Verify `leafTie.root` function argument for `merge()` proc why: Zero root will lead to inconsistent DB entry * Aristo: Update failure condition for hash labels compiler `hashify()` why: Node need not be rejected as long as links are on the schedule. In that case, `redo[]` is to become `wff.base[]` at a later stage. This amends an earlier fix, part of #1952 by also testing against the target nodes of the `wff.base[]` sets. * Aristo: Add storage root glue record to `hashify()` schedule why: An account leaf node might refer to a non-resolvable storage root ID. Storage root node chains will end up at the storage root. So the link `storage-root->account-leaf` needs an extra item in the schedule. * Aristo: fix error code returned by `fetchPayload()` details: Final error code is implied by the error code form the `hikeUp()` function. * CoreDb: Discard `createOk` argument in API `getRoot()` function why: Not needed for the legacy DB. For the `Arsto` DB, a lazy approach is implemented where a stprage root node is created on-the-fly. * CoreDb: Prevent `$$` logging in some cases why: Logging the function `$$` is not useful when it is used for internal use, i.e. retrieving an an error text for logging. * CoreDb: Add `tryHashFn()` to API for pretty printing why: Pretty printing must not change the hashification status for the `Aristo` DB. So there is an independent API wrapper for getting the node hash which never updated the hashes. * CoreDb: Discard `update` argument in API `hash()` function why: When calling the API function `hash()`, the latest state is always wanted. For a version that uses the current state as-is without checking, the function `tryHash()` was added to the backend. * CoreDb: Update opaque vertex ID objects for the `Aristo` backend why: For `Aristo`, vID objects encapsulate a numeric `VertexID` referencing a vertex (rather than a node hash as used on the legacy backend.) For storage sub-tries, there might be no initial vertex known when the descriptor is created. So opaque vertex ID objects are supported without a valid `VertexID` which will be initalised on-the-fly when the first item is merged. * CoreDb: Add pretty printer for opaque vertex ID objects * Cosmetics, printing profiling data * CoreDb: Fix segfault in `Aristo` backend when creating MPT descriptor why: Missing initialisation error * CoreDb: Allow MPT to inherit shared context on `Aristo` backend why: Creates descriptors with different storage roots for the same shared `Aristo` DB descriptor. * Cosmetics, update diagnostic message items for `Aristo` backend * Fix Copyright year
2024-01-11 19:11:38 +00:00
# Copyright (c) 2023-2024 Status Research & Development GmbH
Core db update providing additional results code interface (#1776) * Split `core_db/base.nim` into several sources * Rename `core_db/legacy.nim` => `core_db/legacy_db.nim` * Update `CoreDb` API, dual methods returning `Result[]` or plain value detail: Plain value methods implemet the legacy API, they defect on error results * Redesign `CoreDB` direct backend access why: Made the `backend` directive integral part of the API * Discontinue providing unused or otherwise available functions details: + setTransactionID() removed, not used and not easily replicable in Aristo + maybeGet() removed, available via direct backend access + newPhk() removed, never used & was experimental anyway * Update/reorg backend API why: + Added error print function `$$()` + General descriptor completion (and optional validation) via `bless()` * Update `Aristo`/`Kvt` exception handling why: Avoid `CatchableError` exceptions, rather pass them as error code where appropriate. * More `CoreDB` compliant `Aristo` and `Kvt` methods details: + Providing functions like `contains()`, `getVtxRc()` (returns `Result[]`). + Additional error code: `NotImplemented` * Rewrite/reorg of Aristo DB constructor why: Previously used global object `DefaultQidLayoutRef` as default initialiser. This object was created at compile time which lead to non-gc safe functions. * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/aristo/aristo_transcode.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> --------- Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-09-26 09:21:13 +00:00
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or
# http://www.apache.org/licenses/LICENSE-2.0)
# * MIT license ([LICENSE-MIT](LICENSE-MIT) or
# http://opensource.org/licenses/MIT)
# at your option. This file may not be copied, modified, or distributed except
# according to those terms.
{.push raises: [].}
import
./base_desc
type
EphemMethodsDesc =
Core db update api and fix tracer methods (#1816) * CoreDB: Re-org API details: Legacy API internally uses vertex ID for root node abstraction * Cosmetics: Move some unit test helpers to common sub-directory * Extract constant from `accouns_cache.nim` => `constants.nim` * Fix tracer methods why: Logger dump data were wrongly dumped from the production database. This caused an assert exception when iterating over the persistent database (instead of the memory logger.) This event in turn was enabled after fixing another inconsistency which just set up an empty iterator. Unit tests failed to detect that.
2023-10-11 19:09:11 +00:00
CoreDbBackendRef | CoreDbKvtBackendRef | CoreDbMptBackendRef |
Core db+ledger aristo backend update (#2006) * CoreDb: Improve API and API tracking why: Now logs state roots where appropriate * CoreDb: re-implement `CoreDbVidRef` => `CoreDbTrieRef` why: Instead of a root vertex ID wrapper, the purpose of this object type has been upgrades to a sub-trie prototype. caveat: `Aristo` backend not fully functional, yet. * CoreDb: Update `Aristo` backend why: Supports virtual sub-tries * CoreDb: Account address tracking for `StorageTrie` virtual tries details: Supported with API tracking/logging * CoreDb: Keep account address in payload object why: No need to provide extra address argument for `merge()`, also provides tracking possibility for account debugging. * Ledger: Update new API for `Aristo` specific storage trie handling * CoreDb+Ledger: Update unit tests * Fix copyright headers
2024-02-02 20:23:04 +00:00
CoreDbAccBackendRef | CoreDbTrieRef
Core db update providing additional results code interface (#1776) * Split `core_db/base.nim` into several sources * Rename `core_db/legacy.nim` => `core_db/legacy_db.nim` * Update `CoreDb` API, dual methods returning `Result[]` or plain value detail: Plain value methods implemet the legacy API, they defect on error results * Redesign `CoreDB` direct backend access why: Made the `backend` directive integral part of the API * Discontinue providing unused or otherwise available functions details: + setTransactionID() removed, not used and not easily replicable in Aristo + maybeGet() removed, available via direct backend access + newPhk() removed, never used & was experimental anyway * Update/reorg backend API why: + Added error print function `$$()` + General descriptor completion (and optional validation) via `bless()` * Update `Aristo`/`Kvt` exception handling why: Avoid `CatchableError` exceptions, rather pass them as error code where appropriate. * More `CoreDB` compliant `Aristo` and `Kvt` methods details: + Providing functions like `contains()`, `getVtxRc()` (returns `Result[]`). + Additional error code: `NotImplemented` * Rewrite/reorg of Aristo DB constructor why: Previously used global object `DefaultQidLayoutRef` as default initialiser. This object was created at compile time which lead to non-gc safe functions. * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/aristo/aristo_transcode.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> --------- Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-09-26 09:21:13 +00:00
MethodsDesc =
Core db update api and fix tracer methods (#1816) * CoreDB: Re-org API details: Legacy API internally uses vertex ID for root node abstraction * Cosmetics: Move some unit test helpers to common sub-directory * Extract constant from `accouns_cache.nim` => `constants.nim` * Fix tracer methods why: Logger dump data were wrongly dumped from the production database. This caused an assert exception when iterating over the persistent database (instead of the memory logger.) This event in turn was enabled after fixing another inconsistency which just set up an empty iterator. Unit tests failed to detect that.
2023-10-11 19:09:11 +00:00
CoreDxKvtRef |
CoreDxMptRef | CoreDxPhkRef | CoreDxAccRef |
CoreDxTxRef | CoreDxTxID |
Core db update providing additional results code interface (#1776) * Split `core_db/base.nim` into several sources * Rename `core_db/legacy.nim` => `core_db/legacy_db.nim` * Update `CoreDb` API, dual methods returning `Result[]` or plain value detail: Plain value methods implemet the legacy API, they defect on error results * Redesign `CoreDB` direct backend access why: Made the `backend` directive integral part of the API * Discontinue providing unused or otherwise available functions details: + setTransactionID() removed, not used and not easily replicable in Aristo + maybeGet() removed, available via direct backend access + newPhk() removed, never used & was experimental anyway * Update/reorg backend API why: + Added error print function `$$()` + General descriptor completion (and optional validation) via `bless()` * Update `Aristo`/`Kvt` exception handling why: Avoid `CatchableError` exceptions, rather pass them as error code where appropriate. * More `CoreDB` compliant `Aristo` and `Kvt` methods details: + Providing functions like `contains()`, `getVtxRc()` (returns `Result[]`). + Additional error code: `NotImplemented` * Rewrite/reorg of Aristo DB constructor why: Previously used global object `DefaultQidLayoutRef` as default initialiser. This object was created at compile time which lead to non-gc safe functions. * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/aristo/aristo_transcode.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> --------- Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-09-26 09:21:13 +00:00
CoreDxCaptRef
Core db update api and fix tracer methods (#1816) * CoreDB: Re-org API details: Legacy API internally uses vertex ID for root node abstraction * Cosmetics: Move some unit test helpers to common sub-directory * Extract constant from `accouns_cache.nim` => `constants.nim` * Fix tracer methods why: Logger dump data were wrongly dumped from the production database. This caused an assert exception when iterating over the persistent database (instead of the memory logger.) This event in turn was enabled after fixing another inconsistency which just set up an empty iterator. Unit tests failed to detect that.
2023-10-11 19:09:11 +00:00
ValidateDesc* = MethodsDesc | EphemMethodsDesc | CoreDbErrorRef
Core db update providing additional results code interface (#1776) * Split `core_db/base.nim` into several sources * Rename `core_db/legacy.nim` => `core_db/legacy_db.nim` * Update `CoreDb` API, dual methods returning `Result[]` or plain value detail: Plain value methods implemet the legacy API, they defect on error results * Redesign `CoreDB` direct backend access why: Made the `backend` directive integral part of the API * Discontinue providing unused or otherwise available functions details: + setTransactionID() removed, not used and not easily replicable in Aristo + maybeGet() removed, available via direct backend access + newPhk() removed, never used & was experimental anyway * Update/reorg backend API why: + Added error print function `$$()` + General descriptor completion (and optional validation) via `bless()` * Update `Aristo`/`Kvt` exception handling why: Avoid `CatchableError` exceptions, rather pass them as error code where appropriate. * More `CoreDB` compliant `Aristo` and `Kvt` methods details: + Providing functions like `contains()`, `getVtxRc()` (returns `Result[]`). + Additional error code: `NotImplemented` * Rewrite/reorg of Aristo DB constructor why: Previously used global object `DefaultQidLayoutRef` as default initialiser. This object was created at compile time which lead to non-gc safe functions. * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/aristo/aristo_transcode.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> --------- Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-09-26 09:21:13 +00:00
# ------------------------------------------------------------------------------
# Private helpers
# ------------------------------------------------------------------------------
Core db update api and fix tracer methods (#1816) * CoreDB: Re-org API details: Legacy API internally uses vertex ID for root node abstraction * Cosmetics: Move some unit test helpers to common sub-directory * Extract constant from `accouns_cache.nim` => `constants.nim` * Fix tracer methods why: Logger dump data were wrongly dumped from the production database. This caused an assert exception when iterating over the persistent database (instead of the memory logger.) This event in turn was enabled after fixing another inconsistency which just set up an empty iterator. Unit tests failed to detect that.
2023-10-11 19:09:11 +00:00
proc validateMethodsDesc(base: CoreDbBaseFns) =
Core db+ledger aristo backend update (#2006) * CoreDb: Improve API and API tracking why: Now logs state roots where appropriate * CoreDb: re-implement `CoreDbVidRef` => `CoreDbTrieRef` why: Instead of a root vertex ID wrapper, the purpose of this object type has been upgrades to a sub-trie prototype. caveat: `Aristo` backend not fully functional, yet. * CoreDb: Update `Aristo` backend why: Supports virtual sub-tries * CoreDb: Account address tracking for `StorageTrie` virtual tries details: Supported with API tracking/logging * CoreDb: Keep account address in payload object why: No need to provide extra address argument for `merge()`, also provides tracking possibility for account debugging. * Ledger: Update new API for `Aristo` specific storage trie handling * CoreDb+Ledger: Update unit tests * Fix copyright headers
2024-02-02 20:23:04 +00:00
doAssert not base.verifyFn.isNil
Core db update api and fix tracer methods (#1816) * CoreDB: Re-org API details: Legacy API internally uses vertex ID for root node abstraction * Cosmetics: Move some unit test helpers to common sub-directory * Extract constant from `accouns_cache.nim` => `constants.nim` * Fix tracer methods why: Logger dump data were wrongly dumped from the production database. This caused an assert exception when iterating over the persistent database (instead of the memory logger.) This event in turn was enabled after fixing another inconsistency which just set up an empty iterator. Unit tests failed to detect that.
2023-10-11 19:09:11 +00:00
doAssert not base.backendFn.isNil
doAssert not base.destroyFn.isNil
Core db update storage root management for sub tries (#1964) * Aristo: Re-phrase `LayerDelta` and `LayerFinal` as object references why: Avoids copying in some cases * Fix copyright header * Aristo: Verify `leafTie.root` function argument for `merge()` proc why: Zero root will lead to inconsistent DB entry * Aristo: Update failure condition for hash labels compiler `hashify()` why: Node need not be rejected as long as links are on the schedule. In that case, `redo[]` is to become `wff.base[]` at a later stage. This amends an earlier fix, part of #1952 by also testing against the target nodes of the `wff.base[]` sets. * Aristo: Add storage root glue record to `hashify()` schedule why: An account leaf node might refer to a non-resolvable storage root ID. Storage root node chains will end up at the storage root. So the link `storage-root->account-leaf` needs an extra item in the schedule. * Aristo: fix error code returned by `fetchPayload()` details: Final error code is implied by the error code form the `hikeUp()` function. * CoreDb: Discard `createOk` argument in API `getRoot()` function why: Not needed for the legacy DB. For the `Arsto` DB, a lazy approach is implemented where a stprage root node is created on-the-fly. * CoreDb: Prevent `$$` logging in some cases why: Logging the function `$$` is not useful when it is used for internal use, i.e. retrieving an an error text for logging. * CoreDb: Add `tryHashFn()` to API for pretty printing why: Pretty printing must not change the hashification status for the `Aristo` DB. So there is an independent API wrapper for getting the node hash which never updated the hashes. * CoreDb: Discard `update` argument in API `hash()` function why: When calling the API function `hash()`, the latest state is always wanted. For a version that uses the current state as-is without checking, the function `tryHash()` was added to the backend. * CoreDb: Update opaque vertex ID objects for the `Aristo` backend why: For `Aristo`, vID objects encapsulate a numeric `VertexID` referencing a vertex (rather than a node hash as used on the legacy backend.) For storage sub-tries, there might be no initial vertex known when the descriptor is created. So opaque vertex ID objects are supported without a valid `VertexID` which will be initalised on-the-fly when the first item is merged. * CoreDb: Add pretty printer for opaque vertex ID objects * Cosmetics, printing profiling data * CoreDb: Fix segfault in `Aristo` backend when creating MPT descriptor why: Missing initialisation error * CoreDb: Allow MPT to inherit shared context on `Aristo` backend why: Creates descriptors with different storage roots for the same shared `Aristo` DB descriptor. * Cosmetics, update diagnostic message items for `Aristo` backend * Fix Copyright year
2024-01-11 19:11:38 +00:00
doAssert not base.tryHashFn.isNil
Core db+ledger aristo backend update (#2006) * CoreDb: Improve API and API tracking why: Now logs state roots where appropriate * CoreDb: re-implement `CoreDbVidRef` => `CoreDbTrieRef` why: Instead of a root vertex ID wrapper, the purpose of this object type has been upgrades to a sub-trie prototype. caveat: `Aristo` backend not fully functional, yet. * CoreDb: Update `Aristo` backend why: Supports virtual sub-tries * CoreDb: Account address tracking for `StorageTrie` virtual tries details: Supported with API tracking/logging * CoreDb: Keep account address in payload object why: No need to provide extra address argument for `merge()`, also provides tracking possibility for account debugging. * Ledger: Update new API for `Aristo` specific storage trie handling * CoreDb+Ledger: Update unit tests * Fix copyright headers
2024-02-02 20:23:04 +00:00
doAssert not base.rootHashFn.isNil
doAssert not base.triePrintFn.isNil
Core db update api and fix tracer methods (#1816) * CoreDB: Re-org API details: Legacy API internally uses vertex ID for root node abstraction * Cosmetics: Move some unit test helpers to common sub-directory * Extract constant from `accouns_cache.nim` => `constants.nim` * Fix tracer methods why: Logger dump data were wrongly dumped from the production database. This caused an assert exception when iterating over the persistent database (instead of the memory logger.) This event in turn was enabled after fixing another inconsistency which just set up an empty iterator. Unit tests failed to detect that.
2023-10-11 19:09:11 +00:00
doAssert not base.errorPrintFn.isNil
doAssert not base.legacySetupFn.isNil
Core db+ledger aristo backend update (#2006) * CoreDb: Improve API and API tracking why: Now logs state roots where appropriate * CoreDb: re-implement `CoreDbVidRef` => `CoreDbTrieRef` why: Instead of a root vertex ID wrapper, the purpose of this object type has been upgrades to a sub-trie prototype. caveat: `Aristo` backend not fully functional, yet. * CoreDb: Update `Aristo` backend why: Supports virtual sub-tries * CoreDb: Account address tracking for `StorageTrie` virtual tries details: Supported with API tracking/logging * CoreDb: Keep account address in payload object why: No need to provide extra address argument for `merge()`, also provides tracking possibility for account debugging. * Ledger: Update new API for `Aristo` specific storage trie handling * CoreDb+Ledger: Update unit tests * Fix copyright headers
2024-02-02 20:23:04 +00:00
doAssert not base.getTrieFn.isNil
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
doAssert not base.levelFn.isNil
Core db update api and fix tracer methods (#1816) * CoreDB: Re-org API details: Legacy API internally uses vertex ID for root node abstraction * Cosmetics: Move some unit test helpers to common sub-directory * Extract constant from `accouns_cache.nim` => `constants.nim` * Fix tracer methods why: Logger dump data were wrongly dumped from the production database. This caused an assert exception when iterating over the persistent database (instead of the memory logger.) This event in turn was enabled after fixing another inconsistency which just set up an empty iterator. Unit tests failed to detect that.
2023-10-11 19:09:11 +00:00
doAssert not base.newKvtFn.isNil
doAssert not base.newMptFn.isNil
doAssert not base.newAccFn.isNil
doAssert not base.getIdFn.isNil
doAssert not base.beginFn.isNil
doAssert not base.captureFn.isNil
Core db update providing additional results code interface (#1776) * Split `core_db/base.nim` into several sources * Rename `core_db/legacy.nim` => `core_db/legacy_db.nim` * Update `CoreDb` API, dual methods returning `Result[]` or plain value detail: Plain value methods implemet the legacy API, they defect on error results * Redesign `CoreDB` direct backend access why: Made the `backend` directive integral part of the API * Discontinue providing unused or otherwise available functions details: + setTransactionID() removed, not used and not easily replicable in Aristo + maybeGet() removed, available via direct backend access + newPhk() removed, never used & was experimental anyway * Update/reorg backend API why: + Added error print function `$$()` + General descriptor completion (and optional validation) via `bless()` * Update `Aristo`/`Kvt` exception handling why: Avoid `CatchableError` exceptions, rather pass them as error code where appropriate. * More `CoreDB` compliant `Aristo` and `Kvt` methods details: + Providing functions like `contains()`, `getVtxRc()` (returns `Result[]`). + Additional error code: `NotImplemented` * Rewrite/reorg of Aristo DB constructor why: Previously used global object `DefaultQidLayoutRef` as default initialiser. This object was created at compile time which lead to non-gc safe functions. * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/aristo/aristo_transcode.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> --------- Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-09-26 09:21:13 +00:00
proc validateMethodsDesc(kvt: CoreDbKvtFns) =
doAssert not kvt.backendFn.isNil
doAssert not kvt.getFn.isNil
doAssert not kvt.delFn.isNil
doAssert not kvt.putFn.isNil
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
doAssert not kvt.persistentFn.isNil
doAssert not kvt.forgetFn.isNil
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
doAssert not kvt.hasKeyFn.isNil
Core db update providing additional results code interface (#1776) * Split `core_db/base.nim` into several sources * Rename `core_db/legacy.nim` => `core_db/legacy_db.nim` * Update `CoreDb` API, dual methods returning `Result[]` or plain value detail: Plain value methods implemet the legacy API, they defect on error results * Redesign `CoreDB` direct backend access why: Made the `backend` directive integral part of the API * Discontinue providing unused or otherwise available functions details: + setTransactionID() removed, not used and not easily replicable in Aristo + maybeGet() removed, available via direct backend access + newPhk() removed, never used & was experimental anyway * Update/reorg backend API why: + Added error print function `$$()` + General descriptor completion (and optional validation) via `bless()` * Update `Aristo`/`Kvt` exception handling why: Avoid `CatchableError` exceptions, rather pass them as error code where appropriate. * More `CoreDB` compliant `Aristo` and `Kvt` methods details: + Providing functions like `contains()`, `getVtxRc()` (returns `Result[]`). + Additional error code: `NotImplemented` * Rewrite/reorg of Aristo DB constructor why: Previously used global object `DefaultQidLayoutRef` as default initialiser. This object was created at compile time which lead to non-gc safe functions. * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/aristo/aristo_transcode.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> --------- Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-09-26 09:21:13 +00:00
proc validateMethodsDesc(fns: CoreDbMptFns) =
doAssert not fns.backendFn.isNil
Core db update api and fix tracer methods (#1816) * CoreDB: Re-org API details: Legacy API internally uses vertex ID for root node abstraction * Cosmetics: Move some unit test helpers to common sub-directory * Extract constant from `accouns_cache.nim` => `constants.nim` * Fix tracer methods why: Logger dump data were wrongly dumped from the production database. This caused an assert exception when iterating over the persistent database (instead of the memory logger.) This event in turn was enabled after fixing another inconsistency which just set up an empty iterator. Unit tests failed to detect that.
2023-10-11 19:09:11 +00:00
doAssert not fns.fetchFn.isNil
doAssert not fns.deleteFn.isNil
doAssert not fns.mergeFn.isNil
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
doAssert not fns.hasPathFn.isNil
Core db+ledger aristo backend update (#2006) * CoreDb: Improve API and API tracking why: Now logs state roots where appropriate * CoreDb: re-implement `CoreDbVidRef` => `CoreDbTrieRef` why: Instead of a root vertex ID wrapper, the purpose of this object type has been upgrades to a sub-trie prototype. caveat: `Aristo` backend not fully functional, yet. * CoreDb: Update `Aristo` backend why: Supports virtual sub-tries * CoreDb: Account address tracking for `StorageTrie` virtual tries details: Supported with API tracking/logging * CoreDb: Keep account address in payload object why: No need to provide extra address argument for `merge()`, also provides tracking possibility for account debugging. * Ledger: Update new API for `Aristo` specific storage trie handling * CoreDb+Ledger: Update unit tests * Fix copyright headers
2024-02-02 20:23:04 +00:00
doAssert not fns.getTrieFn.isNil
Core db update providing additional results code interface (#1776) * Split `core_db/base.nim` into several sources * Rename `core_db/legacy.nim` => `core_db/legacy_db.nim` * Update `CoreDb` API, dual methods returning `Result[]` or plain value detail: Plain value methods implemet the legacy API, they defect on error results * Redesign `CoreDB` direct backend access why: Made the `backend` directive integral part of the API * Discontinue providing unused or otherwise available functions details: + setTransactionID() removed, not used and not easily replicable in Aristo + maybeGet() removed, available via direct backend access + newPhk() removed, never used & was experimental anyway * Update/reorg backend API why: + Added error print function `$$()` + General descriptor completion (and optional validation) via `bless()` * Update `Aristo`/`Kvt` exception handling why: Avoid `CatchableError` exceptions, rather pass them as error code where appropriate. * More `CoreDB` compliant `Aristo` and `Kvt` methods details: + Providing functions like `contains()`, `getVtxRc()` (returns `Result[]`). + Additional error code: `NotImplemented` * Rewrite/reorg of Aristo DB constructor why: Previously used global object `DefaultQidLayoutRef` as default initialiser. This object was created at compile time which lead to non-gc safe functions. * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/aristo/aristo_transcode.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> --------- Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-09-26 09:21:13 +00:00
doAssert not fns.isPruningFn.isNil
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
doAssert not fns.persistentFn.isNil
doAssert not fns.forgetFn.isNil
Core db update providing additional results code interface (#1776) * Split `core_db/base.nim` into several sources * Rename `core_db/legacy.nim` => `core_db/legacy_db.nim` * Update `CoreDb` API, dual methods returning `Result[]` or plain value detail: Plain value methods implemet the legacy API, they defect on error results * Redesign `CoreDB` direct backend access why: Made the `backend` directive integral part of the API * Discontinue providing unused or otherwise available functions details: + setTransactionID() removed, not used and not easily replicable in Aristo + maybeGet() removed, available via direct backend access + newPhk() removed, never used & was experimental anyway * Update/reorg backend API why: + Added error print function `$$()` + General descriptor completion (and optional validation) via `bless()` * Update `Aristo`/`Kvt` exception handling why: Avoid `CatchableError` exceptions, rather pass them as error code where appropriate. * More `CoreDB` compliant `Aristo` and `Kvt` methods details: + Providing functions like `contains()`, `getVtxRc()` (returns `Result[]`). + Additional error code: `NotImplemented` * Rewrite/reorg of Aristo DB constructor why: Previously used global object `DefaultQidLayoutRef` as default initialiser. This object was created at compile time which lead to non-gc safe functions. * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/aristo/aristo_transcode.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> --------- Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-09-26 09:21:13 +00:00
Core db update api and fix tracer methods (#1816) * CoreDB: Re-org API details: Legacy API internally uses vertex ID for root node abstraction * Cosmetics: Move some unit test helpers to common sub-directory * Extract constant from `accouns_cache.nim` => `constants.nim` * Fix tracer methods why: Logger dump data were wrongly dumped from the production database. This caused an assert exception when iterating over the persistent database (instead of the memory logger.) This event in turn was enabled after fixing another inconsistency which just set up an empty iterator. Unit tests failed to detect that.
2023-10-11 19:09:11 +00:00
proc validateMethodsDesc(fns: CoreDbAccFns) =
doAssert not fns.backendFn.isNil
Optional accounts cache module for creating genesis (#1897) * Split off `ReadOnlyStateDB` from `AccountStateDB` from `state_db.nim` why: Apart from testing, applications use `ReadOnlyStateDB` as an easy way to access the accounts ledger. This is well supported by the `Aristo` db, but writable mode is only parially supported. The writable AccountStateDB` object for modifying accounts is not used by production code. So, for lecgacy and testing apps, the full support of the previous `AccountStateDB` is now enabled by `import db/state_db/read_write` and the `import db/state_db` provides read-only mode. * Encapsulate `AccountStateDB` as `GenesisLedgerRef` or genesis creation why: `AccountStateDB` has poor support for `Aristo` and is not widely used in favour of `AccountsLedger` (which will be abstracted as `ledger`.) Currently, using other than the `AccountStateDB` ledgers within the `GenesisLedgerRef` wrapper is experimental and test only. Eventually, the wrapper should disappear so that the `Ledger` object (which encapsulates `AccountsCache` and `AccountsLedger`) will prevail. * For the `Ledger`, provide access to raw accounts `MPT` why: This gives to the `CoreDbMptRef` descriptor from the `CoreDb` (which is the legacy version of CoreDxMptRef`.) For the new `ledger` API, the accounts are based on the `CoreDxMAccRef` descriptor which uses a particular sub-system for accounts while legacy applications use the `CoreDbPhkRef` equivalent of the `SecureHexaryTrie`. The only place where this feature will currently be used is the `genesis.nim` source file. * Fix `Aristo` bugs, missing boundary checks, typos, etc. * Verify root vertex in `MPT` and account constructors why: Was missing so far, in particular the accounts constructor must verify `VertexID(1) * Fix include file
2023-11-20 11:51:43 +00:00
doAssert not fns.newMptFn.isNil
Core db update api and fix tracer methods (#1816) * CoreDB: Re-org API details: Legacy API internally uses vertex ID for root node abstraction * Cosmetics: Move some unit test helpers to common sub-directory * Extract constant from `accouns_cache.nim` => `constants.nim` * Fix tracer methods why: Logger dump data were wrongly dumped from the production database. This caused an assert exception when iterating over the persistent database (instead of the memory logger.) This event in turn was enabled after fixing another inconsistency which just set up an empty iterator. Unit tests failed to detect that.
2023-10-11 19:09:11 +00:00
doAssert not fns.fetchFn.isNil
doAssert not fns.deleteFn.isNil
Core db+aristo update storage trie handling (#2023) * CoreDb: Test module with additional sample selector cmd line options * Aristo: Do not automatically remove a storage trie with the account why: This is an unnecessary side effect. Rather than using an automatism, a a storage root must be deleted manually. * Aristo: Can handle stale storage root vertex IDs as empty IDs. why: This is currently needed for the ledger API supporting both, a legacy and the `Aristo` database backend. This feature can be disabled at compile time by re-setting the `LOOSE_STORAGE_TRIE_COUPLING` flag in the `aristo_constants` module. * CoreDb+Aristo: Flush/delete storage trie when deleting account why: On either backend, a deleted account leave a dangling storage trie on the database. For consistency nn the legacy backend, storage tries must not be deleted as they might be shared by several accounts whereas on `Aristo` they are always unique.
2024-02-12 19:37:00 +00:00
doAssert not fns.stoFlushFn.isNil
Core db update api and fix tracer methods (#1816) * CoreDB: Re-org API details: Legacy API internally uses vertex ID for root node abstraction * Cosmetics: Move some unit test helpers to common sub-directory * Extract constant from `accouns_cache.nim` => `constants.nim` * Fix tracer methods why: Logger dump data were wrongly dumped from the production database. This caused an assert exception when iterating over the persistent database (instead of the memory logger.) This event in turn was enabled after fixing another inconsistency which just set up an empty iterator. Unit tests failed to detect that.
2023-10-11 19:09:11 +00:00
doAssert not fns.mergeFn.isNil
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
doAssert not fns.hasPathFn.isNil
Core db+ledger aristo backend update (#2006) * CoreDb: Improve API and API tracking why: Now logs state roots where appropriate * CoreDb: re-implement `CoreDbVidRef` => `CoreDbTrieRef` why: Instead of a root vertex ID wrapper, the purpose of this object type has been upgrades to a sub-trie prototype. caveat: `Aristo` backend not fully functional, yet. * CoreDb: Update `Aristo` backend why: Supports virtual sub-tries * CoreDb: Account address tracking for `StorageTrie` virtual tries details: Supported with API tracking/logging * CoreDb: Keep account address in payload object why: No need to provide extra address argument for `merge()`, also provides tracking possibility for account debugging. * Ledger: Update new API for `Aristo` specific storage trie handling * CoreDb+Ledger: Update unit tests * Fix copyright headers
2024-02-02 20:23:04 +00:00
doAssert not fns.getTrieFn.isNil
Core db update api and fix tracer methods (#1816) * CoreDB: Re-org API details: Legacy API internally uses vertex ID for root node abstraction * Cosmetics: Move some unit test helpers to common sub-directory * Extract constant from `accouns_cache.nim` => `constants.nim` * Fix tracer methods why: Logger dump data were wrongly dumped from the production database. This caused an assert exception when iterating over the persistent database (instead of the memory logger.) This event in turn was enabled after fixing another inconsistency which just set up an empty iterator. Unit tests failed to detect that.
2023-10-11 19:09:11 +00:00
doAssert not fns.isPruningFn.isNil
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
doAssert not fns.persistentFn.isNil
doAssert not fns.forgetFn.isNil
Core db update providing additional results code interface (#1776) * Split `core_db/base.nim` into several sources * Rename `core_db/legacy.nim` => `core_db/legacy_db.nim` * Update `CoreDb` API, dual methods returning `Result[]` or plain value detail: Plain value methods implemet the legacy API, they defect on error results * Redesign `CoreDB` direct backend access why: Made the `backend` directive integral part of the API * Discontinue providing unused or otherwise available functions details: + setTransactionID() removed, not used and not easily replicable in Aristo + maybeGet() removed, available via direct backend access + newPhk() removed, never used & was experimental anyway * Update/reorg backend API why: + Added error print function `$$()` + General descriptor completion (and optional validation) via `bless()` * Update `Aristo`/`Kvt` exception handling why: Avoid `CatchableError` exceptions, rather pass them as error code where appropriate. * More `CoreDB` compliant `Aristo` and `Kvt` methods details: + Providing functions like `contains()`, `getVtxRc()` (returns `Result[]`). + Additional error code: `NotImplemented` * Rewrite/reorg of Aristo DB constructor why: Previously used global object `DefaultQidLayoutRef` as default initialiser. This object was created at compile time which lead to non-gc safe functions. * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/aristo/aristo_transcode.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> --------- Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-09-26 09:21:13 +00:00
# ------------
Core db+ledger aristo backend update (#2006) * CoreDb: Improve API and API tracking why: Now logs state roots where appropriate * CoreDb: re-implement `CoreDbVidRef` => `CoreDbTrieRef` why: Instead of a root vertex ID wrapper, the purpose of this object type has been upgrades to a sub-trie prototype. caveat: `Aristo` backend not fully functional, yet. * CoreDb: Update `Aristo` backend why: Supports virtual sub-tries * CoreDb: Account address tracking for `StorageTrie` virtual tries details: Supported with API tracking/logging * CoreDb: Keep account address in payload object why: No need to provide extra address argument for `merge()`, also provides tracking possibility for account debugging. * Ledger: Update new API for `Aristo` specific storage trie handling * CoreDb+Ledger: Update unit tests * Fix copyright headers
2024-02-02 20:23:04 +00:00
proc validateMethodsDesc(trie: CoreDbTrieRef) =
doAssert not trie.isNil
doAssert not trie.parent.isNil
doAssert trie.ready == true
Core db update api and fix tracer methods (#1816) * CoreDB: Re-org API details: Legacy API internally uses vertex ID for root node abstraction * Cosmetics: Move some unit test helpers to common sub-directory * Extract constant from `accouns_cache.nim` => `constants.nim` * Fix tracer methods why: Logger dump data were wrongly dumped from the production database. This caused an assert exception when iterating over the persistent database (instead of the memory logger.) This event in turn was enabled after fixing another inconsistency which just set up an empty iterator. Unit tests failed to detect that.
2023-10-11 19:09:11 +00:00
Redefine result[] error object as ref (see issue #1796) (#1797) details: This does not affect the legacy interface which is the production one, currently.
2023-10-02 18:05:17 +00:00
proc validateMethodsDesc(e: CoreDbErrorRef) =
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
doAssert e.error != CoreDbErrorCode(0)
Redefine result[] error object as ref (see issue #1796) (#1797) details: This does not affect the legacy interface which is the production one, currently.
2023-10-02 18:05:17 +00:00
doAssert not e.isNil
Core db update providing additional results code interface (#1776) * Split `core_db/base.nim` into several sources * Rename `core_db/legacy.nim` => `core_db/legacy_db.nim` * Update `CoreDb` API, dual methods returning `Result[]` or plain value detail: Plain value methods implemet the legacy API, they defect on error results * Redesign `CoreDB` direct backend access why: Made the `backend` directive integral part of the API * Discontinue providing unused or otherwise available functions details: + setTransactionID() removed, not used and not easily replicable in Aristo + maybeGet() removed, available via direct backend access + newPhk() removed, never used & was experimental anyway * Update/reorg backend API why: + Added error print function `$$()` + General descriptor completion (and optional validation) via `bless()` * Update `Aristo`/`Kvt` exception handling why: Avoid `CatchableError` exceptions, rather pass them as error code where appropriate. * More `CoreDB` compliant `Aristo` and `Kvt` methods details: + Providing functions like `contains()`, `getVtxRc()` (returns `Result[]`). + Additional error code: `NotImplemented` * Rewrite/reorg of Aristo DB constructor why: Previously used global object `DefaultQidLayoutRef` as default initialiser. This object was created at compile time which lead to non-gc safe functions. * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/aristo/aristo_transcode.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> --------- Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-09-26 09:21:13 +00:00
doAssert not e.parent.isNil
proc validateMethodsDesc(eph: EphemMethodsDesc) =
doAssert not eph.isNil
doAssert not eph.parent.isNil
Core db update api and fix tracer methods (#1816) * CoreDB: Re-org API details: Legacy API internally uses vertex ID for root node abstraction * Cosmetics: Move some unit test helpers to common sub-directory * Extract constant from `accouns_cache.nim` => `constants.nim` * Fix tracer methods why: Logger dump data were wrongly dumped from the production database. This caused an assert exception when iterating over the persistent database (instead of the memory logger.) This event in turn was enabled after fixing another inconsistency which just set up an empty iterator. Unit tests failed to detect that.
2023-10-11 19:09:11 +00:00
Core db update providing additional results code interface (#1776) * Split `core_db/base.nim` into several sources * Rename `core_db/legacy.nim` => `core_db/legacy_db.nim` * Update `CoreDb` API, dual methods returning `Result[]` or plain value detail: Plain value methods implemet the legacy API, they defect on error results * Redesign `CoreDB` direct backend access why: Made the `backend` directive integral part of the API * Discontinue providing unused or otherwise available functions details: + setTransactionID() removed, not used and not easily replicable in Aristo + maybeGet() removed, available via direct backend access + newPhk() removed, never used & was experimental anyway * Update/reorg backend API why: + Added error print function `$$()` + General descriptor completion (and optional validation) via `bless()` * Update `Aristo`/`Kvt` exception handling why: Avoid `CatchableError` exceptions, rather pass them as error code where appropriate. * More `CoreDB` compliant `Aristo` and `Kvt` methods details: + Providing functions like `contains()`, `getVtxRc()` (returns `Result[]`). + Additional error code: `NotImplemented` * Rewrite/reorg of Aristo DB constructor why: Previously used global object `DefaultQidLayoutRef` as default initialiser. This object was created at compile time which lead to non-gc safe functions. * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/aristo/aristo_transcode.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> --------- Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-09-26 09:21:13 +00:00
proc validateMethodsDesc(kvt: CoreDxKvtRef) =
doAssert not kvt.isNil
doAssert not kvt.parent.isNil
kvt.methods.validateMethodsDesc
proc validateMethodsDesc(mpt: CoreDxMptRef) =
doAssert not mpt.isNil
doAssert not mpt.parent.isNil
mpt.methods.validateMethodsDesc
Core db update api and fix tracer methods (#1816) * CoreDB: Re-org API details: Legacy API internally uses vertex ID for root node abstraction * Cosmetics: Move some unit test helpers to common sub-directory * Extract constant from `accouns_cache.nim` => `constants.nim` * Fix tracer methods why: Logger dump data were wrongly dumped from the production database. This caused an assert exception when iterating over the persistent database (instead of the memory logger.) This event in turn was enabled after fixing another inconsistency which just set up an empty iterator. Unit tests failed to detect that.
2023-10-11 19:09:11 +00:00
proc validateMethodsDesc(acc: CoreDxAccRef) =
doAssert not acc.isNil
doAssert not acc.parent.isNil
acc.methods.validateMethodsDesc
Core db update providing additional results code interface (#1776) * Split `core_db/base.nim` into several sources * Rename `core_db/legacy.nim` => `core_db/legacy_db.nim` * Update `CoreDb` API, dual methods returning `Result[]` or plain value detail: Plain value methods implemet the legacy API, they defect on error results * Redesign `CoreDB` direct backend access why: Made the `backend` directive integral part of the API * Discontinue providing unused or otherwise available functions details: + setTransactionID() removed, not used and not easily replicable in Aristo + maybeGet() removed, available via direct backend access + newPhk() removed, never used & was experimental anyway * Update/reorg backend API why: + Added error print function `$$()` + General descriptor completion (and optional validation) via `bless()` * Update `Aristo`/`Kvt` exception handling why: Avoid `CatchableError` exceptions, rather pass them as error code where appropriate. * More `CoreDB` compliant `Aristo` and `Kvt` methods details: + Providing functions like `contains()`, `getVtxRc()` (returns `Result[]`). + Additional error code: `NotImplemented` * Rewrite/reorg of Aristo DB constructor why: Previously used global object `DefaultQidLayoutRef` as default initialiser. This object was created at compile time which lead to non-gc safe functions. * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/aristo/aristo_transcode.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> --------- Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-09-26 09:21:13 +00:00
proc validateMethodsDesc(phk: CoreDxPhkRef) =
doAssert not phk.isNil
doAssert not phk.fromMpt.isNil
phk.methods.validateMethodsDesc
proc validateMethodsDesc(cpt: CoreDxCaptRef) =
doAssert not cpt.isNil
doAssert not cpt.parent.isNil
doAssert not cpt.methods.recorderFn.isNil
doAssert not cpt.methods.getFlagsFn.isNil
proc validateMethodsDesc(tx: CoreDxTxRef) =
doAssert not tx.isNil
doAssert not tx.parent.isNil
Core db update api tracking (#1907) * Fix copyright year * Show elapsed times with enabled `CoreDb` API tracking * Show elapsed times with enabled `LedgerRef` API tracking * Reorg `CoreDb` auto destructors for `Aristo` DB why: While `Aristo` supports some parallelism for concurrent database access, this comes with a price of management overhead. With a naive approach, the auto-destructor will slow down execution because the ledger and evm treat the database in a shared mode where a DB descriptor is just created and thrown away shortly after. This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt` where a few `Shared` type descriptors are cached and a shared reference is returned rather than a disposable new object. * For `CoreDb` support transaction level tracking details: This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support this already. Also return an error on the legacy DB backend when `persistent()` is called while there are transactions pending (the `persistent()` call does nothing otherwise on the legacy backend.) * Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
doAssert not tx.methods.levelFn.isNil
Core db update providing additional results code interface (#1776) * Split `core_db/base.nim` into several sources * Rename `core_db/legacy.nim` => `core_db/legacy_db.nim` * Update `CoreDb` API, dual methods returning `Result[]` or plain value detail: Plain value methods implemet the legacy API, they defect on error results * Redesign `CoreDB` direct backend access why: Made the `backend` directive integral part of the API * Discontinue providing unused or otherwise available functions details: + setTransactionID() removed, not used and not easily replicable in Aristo + maybeGet() removed, available via direct backend access + newPhk() removed, never used & was experimental anyway * Update/reorg backend API why: + Added error print function `$$()` + General descriptor completion (and optional validation) via `bless()` * Update `Aristo`/`Kvt` exception handling why: Avoid `CatchableError` exceptions, rather pass them as error code where appropriate. * More `CoreDB` compliant `Aristo` and `Kvt` methods details: + Providing functions like `contains()`, `getVtxRc()` (returns `Result[]`). + Additional error code: `NotImplemented` * Rewrite/reorg of Aristo DB constructor why: Previously used global object `DefaultQidLayoutRef` as default initialiser. This object was created at compile time which lead to non-gc safe functions. * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/aristo/aristo_transcode.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> --------- Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-09-26 09:21:13 +00:00
doAssert not tx.methods.commitFn.isNil
doAssert not tx.methods.rollbackFn.isNil
doAssert not tx.methods.disposeFn.isNil
doAssert not tx.methods.safeDisposeFn.isNil
proc validateMethodsDesc(id: CoreDxTxID) =
doAssert not id.isNil
doAssert not id.parent.isNil
doAssert not id.methods.roWrapperFn.isNil
proc validateMethodsDesc(db: CoreDbRef) =
doAssert not db.isNil
doAssert db.dbType != CoreDbType(0)
db.methods.validateMethodsDesc
# ------------------------------------------------------------------------------
# Public debugging helpers
# ------------------------------------------------------------------------------
Core db update api and fix tracer methods (#1816) * CoreDB: Re-org API details: Legacy API internally uses vertex ID for root node abstraction * Cosmetics: Move some unit test helpers to common sub-directory * Extract constant from `accouns_cache.nim` => `constants.nim` * Fix tracer methods why: Logger dump data were wrongly dumped from the production database. This caused an assert exception when iterating over the persistent database (instead of the memory logger.) This event in turn was enabled after fixing another inconsistency which just set up an empty iterator. Unit tests failed to detect that.
2023-10-11 19:09:11 +00:00
proc validate*(dsc: ValidateDesc) =
dsc.validateMethodsDesc
Core db update providing additional results code interface (#1776) * Split `core_db/base.nim` into several sources * Rename `core_db/legacy.nim` => `core_db/legacy_db.nim` * Update `CoreDb` API, dual methods returning `Result[]` or plain value detail: Plain value methods implemet the legacy API, they defect on error results * Redesign `CoreDB` direct backend access why: Made the `backend` directive integral part of the API * Discontinue providing unused or otherwise available functions details: + setTransactionID() removed, not used and not easily replicable in Aristo + maybeGet() removed, available via direct backend access + newPhk() removed, never used & was experimental anyway * Update/reorg backend API why: + Added error print function `$$()` + General descriptor completion (and optional validation) via `bless()` * Update `Aristo`/`Kvt` exception handling why: Avoid `CatchableError` exceptions, rather pass them as error code where appropriate. * More `CoreDB` compliant `Aristo` and `Kvt` methods details: + Providing functions like `contains()`, `getVtxRc()` (returns `Result[]`). + Additional error code: `NotImplemented` * Rewrite/reorg of Aristo DB constructor why: Previously used global object `DefaultQidLayoutRef` as default initialiser. This object was created at compile time which lead to non-gc safe functions. * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/aristo/aristo_transcode.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> --------- Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-09-26 09:21:13 +00:00
proc validate*(db: CoreDbRef) =
db.validateMethodsDesc
# ------------------------------------------------------------------------------
# End
# ------------------------------------------------------------------------------