status-im
/
nimbus-eth1
mirror of https://github.com/status-im/nimbus-eth1.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth1/nimbus/db/aristo/aristo_constants.nim

59 lines
2.0 KiB
Nim
Raw Normal View History

Multi layer architecture 4 aristo db (#1581) * Cosmetics, renamed fields (eVtx, bVtx) -> (eVid, bVid) * Multilayered delta architecture for Aristo DB details: Any VertexID or data retrieval needs to go down the rabbit hole and fetch/get/manipulate the bottom layer -- even without explicit backend. * Direct reference to backend from top-level layer why: Some services as the vid management needs to be synchronised among all layers. So access is optimised.
2023-05-14 17:43:01 +00:00
# nimbus-eth1
Aristo: Update unit test suite (#2002) * Aristo: Update unit test suite * Aristo/Kvt: Fix iterators why: Generic iterators were not properly updated after backend change * Aristo: Add sub-trie deletion functionality why: For storage tries linked to an account payload vertex ID, a the whole storage trie needs to be deleted with the account. * Aristo: Reserve vertex ID numbers for static custom state roots why: Static custom state roots may be controlled by an application, e.g. for a receipt or a transaction root. The `Aristo` functions are agnostic of what the static state roots are when different from the internal tree vertex ID 1. details; The `merge()` function applied to a non-static state root (assumed to be a storage root) will check the payload of an accounts leaf and mark its Merkle keys to be re-checked. * Aristo: Correct error code symbol * Aristo: Update error code symbols * Aristo: Code cosmetics/comments * Aristo: Fix hashify schedule calculator why: Had a tendency to stop early leaving an incomplete job
2024-02-01 21:27:48 +00:00
# Copyright (c) 2023-2024 Status Research & Development GmbH
Multi layer architecture 4 aristo db (#1581) * Cosmetics, renamed fields (eVtx, bVtx) -> (eVid, bVid) * Multilayered delta architecture for Aristo DB details: Any VertexID or data retrieval needs to go down the rabbit hole and fetch/get/manipulate the bottom layer -- even without explicit backend. * Direct reference to backend from top-level layer why: Some services as the vid management needs to be synchronised among all layers. So access is optimised.
2023-05-14 17:43:01 +00:00
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or
# http://www.apache.org/licenses/LICENSE-2.0)
# * MIT license ([LICENSE-MIT](LICENSE-MIT) or
# http://opensource.org/licenses/MIT)
# at your option. This file may not be copied, modified, or distributed
# except according to those terms.
{.push raises: [].}
import
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
std/sets,
Allocation-free nibbles buffer (#2406) This buffer eleminates a large part of allocations during MPT traversal, reducing overall memory usage and GC pressure. Ideally, we would use it throughout in the API instead of `openArray[byte]` since the built-in length limit appropriately exposes the natural 64-nibble depth constraint that `openArray` fails to capture.
2024-06-22 20:33:37 +00:00
eth/common,
Aristo db implement filter storage scheduler (#1713) * Rename FilterID => QueueID why: The current usage does not identify a particular filter but uses it as storage tag to manage it on the database (to be organised in a set of FIFOs or queues.) * Split `aristo_filter` source into sub-files why: Make space for filter management API * Store filter queue IDs in pairs on the backend why: Any pair will will describe a FIFO accessed by bottom/top IDs * Reorg some source file names why: The "aristo_" prefix for make local/private files is tedious to use, so removed. * Implement filter slot scheduler details: Filters will be stored on the database on cascaded FIFOs. When a FIFO queue is full, some filter items are bundled together and stored on the next FIFO.
2023-08-25 22:53:59 +00:00
./aristo_desc/desc_identifiers
Multi layer architecture 4 aristo db (#1581) * Cosmetics, renamed fields (eVtx, bVtx) -> (eVid, bVid) * Multilayered delta architecture for Aristo DB details: Any VertexID or data retrieval needs to go down the rabbit hole and fetch/get/manipulate the bottom layer -- even without explicit backend. * Direct reference to backend from top-level layer why: Some services as the vid management needs to be synchronised among all layers. So access is optimised.
2023-05-14 17:43:01 +00:00
const
EmptyBlob* = seq[byte].default
## Useful shortcut (borrowed from `sync/snap/constants.nim`)
Aristo db update foreground caching (#1605) * Fix vertex ID generator state handling for rocksdb backend why: * Key error in walk iterator * Needs to be loaded when opening the database * Use non-zero sub-table prefixes for rocksdb why: Handy for debugging * Fix error code for missing key on rocksdb backend why: Previously returned `VOID_HASH_KEY` rather than `GetKeyNotFound` * Explicitly copy vertex data between internal table and function/result argument why: Function argument or return reference may still refer to the same data object. * Updated error symbols why: Error symbol names for the hike module now start with the prefix `Hike`. * Write back modified branch node into local top layer cache why: With the backend available, the source of the branch node references might not be the top layer cache. So any change must be explicitely recorded.
2023-06-22 11:13:24 +00:00
EmptyVidSeq* = seq[VertexID].default
## Useful shortcut
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
EmptyVidSet* = EmptyVidSeq.toHashSet
Aristo db store filters on backend (#1703) * Simplify RocksDB sub-tables iterator * Implement `filter` storage on backend db details: Unit tests working
2023-08-22 18:44:54 +00:00
## Useful shortcut
Extended data Payload specs for the backend. (#1630) why: For the main tree with root vertex ID 1, the leaf nodes hold the account data. These accounts may link to sub trees the storage root node ID of which must be registered here. There is no reverse key lookup on the backend. note: These definitions are experimental. Also, there are some tests missing for validating Payload data conversions.
2023-07-05 20:27:48 +00:00
VOID_CODE_HASH* = EMPTY_CODE_HASH
Aristodb remove obsolete and time consuming admin features (#2048) * Aristo: Reorg `hashify()` using different schedule algorithm why: Directly calculating the search tree top down from the roots turns out to be faster than using the cached structures left over by `merge()` and `delete()`. Time gains is short of 20% * Aristo: Remove `lTab[]` leaf entry object type why: Not used anymore. It was previously needed to build the schedule for `hashify()`. * Aristo: Avoid unnecessary re-org of the vertex ID recycling list why: This list can become quite large so a heuristic is employed whether it makes sense to re-org. Also, re-org check is only done by `delete()` functions. * Aristo: Remove key/reverse lookup table from tx layers why: It is ignored except for handling proof nodes and costs unnecessary run time resources. This feature was originally needed to accommodate the mental transition from the legacy MPT to the `Aristo` trie :). * Fix copyright year
2024-02-22 08:24:58 +00:00
## Equivalent of `nil` for `Account` object code hash field
Aristo db address sup trie items properly (#1600) * Fix include why: Eth67 not default yet so that got missed * Rename `LeafKey` => `LeafTie` why: Name is a pen picture of what this object is for. Also, it avoids the ubiquitous term `key`. * Provided `getOrVoid()` wrapper for `getOrDefault()` also: Provide `isValid()` syntactic sugar for `.isNil.not`, `!= 0` etc. Reorg descriptor source, split into sub-sources * Bundled `NodeKey` objects with root ID and called it `HashLabel` why: `NodeKey` (aka repurposed Hash265) objects are unique only within a particular sub-trie (e.g. storage slots) which are kept separated (i.e non-interleaved) by design. This is not applied to the backend as the map VertexID->NodeKey labelling the nodes needs not be injective. For the in-memory database (transaction) layers, the injective map VertexID->(VertexID,NodeKey) is used where the first field of the image tuple is the root ID of the sub-trie the `NodeKey` object is valid. So identical storage tries for different accounts can be represented.
2023-06-12 13:48:47 +00:00
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
VOID_HASH_KEY* = HashKey()
Aristo db traversal helpers (#1638) * Misc fixes detail: * Fix de-serialisation for account leafs * Update node recovery from unit tests * Remove `LegacyAccount` from `PayloadRef` object why: Legacy accounts use a hash key as storage root which is detrimental to the working of the Aristo database which uses a vertex ID. * Dissolve `hashify_helper` into `aristo_utils` and `aristo_transcode` why: Functions are of general interest so they should live in first level code files. * Added left/right iterators over leaf nodes * Some helper/wrapper functions that might be useful
2023-07-12 23:03:14 +00:00
## Void equivalent for Merkle hash value
Detach from snap/sync declarations & definitions (#1601) why: Tests and some basic components were originally borrowed from the snap/sync implementation. These have fully been re-implemented.
2023-06-12 18:16:03 +00:00
Aristo: Update unit test suite (#2002) * Aristo: Update unit test suite * Aristo/Kvt: Fix iterators why: Generic iterators were not properly updated after backend change * Aristo: Add sub-trie deletion functionality why: For storage tries linked to an account payload vertex ID, a the whole storage trie needs to be deleted with the account. * Aristo: Reserve vertex ID numbers for static custom state roots why: Static custom state roots may be controlled by an application, e.g. for a receipt or a transaction root. The `Aristo` functions are agnostic of what the static state roots are when different from the internal tree vertex ID 1. details; The `merge()` function applied to a non-static state root (assumed to be a storage root) will check the payload of an accounts leaf and mark its Merkle keys to be re-checked. * Aristo: Correct error code symbol * Aristo: Update error code symbols * Aristo: Code cosmetics/comments * Aristo: Fix hashify schedule calculator why: Had a tendency to stop early leaving an incomplete job
2024-02-01 21:27:48 +00:00
VOID_PATH_ID* = PathID()
## Void equivalent for Merkle hash value
LEAST_FREE_VID* = 100
## Vids smaller are used as known state roots and cannot be recycled. Only
## the `VertexID(1)` state root is used by the `Aristo` methods. The other
## numbers smaller than `LEAST_FREE_VID` may be used by application
## functions with fixed assignments of the type of a state root (e.g. for
## a receipt or a transaction root.)
Core db+aristo update storage trie handling (#2023) * CoreDb: Test module with additional sample selector cmd line options * Aristo: Do not automatically remove a storage trie with the account why: This is an unnecessary side effect. Rather than using an automatism, a a storage root must be deleted manually. * Aristo: Can handle stale storage root vertex IDs as empty IDs. why: This is currently needed for the ledger API supporting both, a legacy and the `Aristo` database backend. This feature can be disabled at compile time by re-setting the `LOOSE_STORAGE_TRIE_COUPLING` flag in the `aristo_constants` module. * CoreDb+Aristo: Flush/delete storage trie when deleting account why: On either backend, a deleted account leave a dangling storage trie on the database. For consistency nn the legacy backend, storage tries must not be deleted as they might be shared by several accounts whereas on `Aristo` they are always unique.
2024-02-12 19:37:00 +00:00
LOOSE_STORAGE_TRIE_COUPLING* = true
## Enabling the `LOOSE_STORAGE_TRIE_COUPLING` flag a sub-trie is considered
## empty if the root vertex ID is zero or at least `LEAST_FREE_VID` and
## there is no vertex available. If the vertex ID is not zero and should
## be considered as such will affect calculating the Merkel hash node key
## for an accou.t leaf of payload type `AccountData`.
##
## Setting this flag `true` might be helpful for running an API supporting
## both, a legacy and# the `Aristo` database backend.
##
Aristo: Update unit test suite (#2002) * Aristo: Update unit test suite * Aristo/Kvt: Fix iterators why: Generic iterators were not properly updated after backend change * Aristo: Add sub-trie deletion functionality why: For storage tries linked to an account payload vertex ID, a the whole storage trie needs to be deleted with the account. * Aristo: Reserve vertex ID numbers for static custom state roots why: Static custom state roots may be controlled by an application, e.g. for a receipt or a transaction root. The `Aristo` functions are agnostic of what the static state roots are when different from the internal tree vertex ID 1. details; The `merge()` function applied to a non-static state root (assumed to be a storage root) will check the payload of an accounts leaf and mark its Merkle keys to be re-checked. * Aristo: Correct error code symbol * Aristo: Update error code symbols * Aristo: Code cosmetics/comments * Aristo: Fix hashify schedule calculator why: Had a tendency to stop early leaving an incomplete job
2024-02-01 21:27:48 +00:00
static:
doAssert 1 < LEAST_FREE_VID # must stay away from `VertexID(1)`
Multi layer architecture 4 aristo db (#1581) * Cosmetics, renamed fields (eVtx, bVtx) -> (eVid, bVid) * Multilayered delta architecture for Aristo DB details: Any VertexID or data retrieval needs to go down the rabbit hole and fetch/get/manipulate the bottom layer -- even without explicit backend. * Direct reference to backend from top-level layer why: Some services as the vid management needs to be synchronised among all layers. So access is optimised.
2023-05-14 17:43:01 +00:00
# End