status-im
/
nimbus-eth1
mirror of https://github.com/status-im/nimbus-eth1.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth1/nimbus/db/aristo/aristo_constants.nim

50 lines
1.4 KiB
Nim
Raw Normal View History

Multi layer architecture 4 aristo db (#1581) * Cosmetics, renamed fields (eVtx, bVtx) -> (eVid, bVid) * Multilayered delta architecture for Aristo DB details: Any VertexID or data retrieval needs to go down the rabbit hole and fetch/get/manipulate the bottom layer -- even without explicit backend. * Direct reference to backend from top-level layer why: Some services as the vid management needs to be synchronised among all layers. So access is optimised.
2023-05-14 17:43:01 +00:00
# nimbus-eth1
Aristo and ledger small updates (#1888) * Fix debug noise in `hashify()` for perfectly normal situation why: Was previously considered a fixable error * Fix test sample file names why: The larger test file `goerli68161.txt.gz` is already in the local archive. So there is no need to use the smaller one from the external repo. * Activate `accounts_cache` module from `db/ledger` why: A copy of the original `accounts_cache.nim` source to be integrated into the `Ledger` module wrapper which allows to switch between different `accounts_cache` implementations unser tha same API. details: At a later state, the `db/accounts_cache.nim` wrapper will be removed so that there is only one access to that module via `db/ledger/accounts_cache.nim`. * Fix copyright headers in source code
2023-11-08 16:52:25 +00:00
# Copyright (c) 2023 Status Research & Development GmbH
Multi layer architecture 4 aristo db (#1581) * Cosmetics, renamed fields (eVtx, bVtx) -> (eVid, bVid) * Multilayered delta architecture for Aristo DB details: Any VertexID or data retrieval needs to go down the rabbit hole and fetch/get/manipulate the bottom layer -- even without explicit backend. * Direct reference to backend from top-level layer why: Some services as the vid management needs to be synchronised among all layers. So access is optimised.
2023-05-14 17:43:01 +00:00
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or
# http://www.apache.org/licenses/LICENSE-2.0)
# * MIT license ([LICENSE-MIT](LICENSE-MIT) or
# http://opensource.org/licenses/MIT)
# at your option. This file may not be copied, modified, or distributed
# except according to those terms.
{.push raises: [].}
import
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
std/sets,
Aristo db address sup trie items properly (#1600) * Fix include why: Eth67 not default yet so that got missed * Rename `LeafKey` => `LeafTie` why: Name is a pen picture of what this object is for. Also, it avoids the ubiquitous term `key`. * Provided `getOrVoid()` wrapper for `getOrDefault()` also: Provide `isValid()` syntactic sugar for `.isNil.not`, `!= 0` etc. Reorg descriptor source, split into sub-sources * Bundled `NodeKey` objects with root ID and called it `HashLabel` why: `NodeKey` (aka repurposed Hash265) objects are unique only within a particular sub-trie (e.g. storage slots) which are kept separated (i.e non-interleaved) by design. This is not applied to the backend as the map VertexID->NodeKey labelling the nodes needs not be injective. For the in-memory database (transaction) layers, the injective map VertexID->(VertexID,NodeKey) is used where the first field of the image tuple is the root ID of the sub-trie the `NodeKey` object is valid. So identical storage tries for different accounts can be represented.
2023-06-12 13:48:47 +00:00
eth/[common, trie/nibbles],
Aristo db implement filter storage scheduler (#1713) * Rename FilterID => QueueID why: The current usage does not identify a particular filter but uses it as storage tag to manage it on the database (to be organised in a set of FIFOs or queues.) * Split `aristo_filter` source into sub-files why: Make space for filter management API * Store filter queue IDs in pairs on the backend why: Any pair will will describe a FIFO accessed by bottom/top IDs * Reorg some source file names why: The "aristo_" prefix for make local/private files is tedious to use, so removed. * Implement filter slot scheduler details: Filters will be stored on the database on cascaded FIFOs. When a FIFO queue is full, some filter items are bundled together and stored on the next FIFO.
2023-08-25 22:53:59 +00:00
./aristo_desc/desc_identifiers
Multi layer architecture 4 aristo db (#1581) * Cosmetics, renamed fields (eVtx, bVtx) -> (eVid, bVid) * Multilayered delta architecture for Aristo DB details: Any VertexID or data retrieval needs to go down the rabbit hole and fetch/get/manipulate the bottom layer -- even without explicit backend. * Direct reference to backend from top-level layer why: Some services as the vid management needs to be synchronised among all layers. So access is optimised.
2023-05-14 17:43:01 +00:00
const
EmptyBlob* = seq[byte].default
## Useful shortcut (borrowed from `sync/snap/constants.nim`)
EmptyNibbleSeq* = EmptyBlob.initNibbleRange
## Useful shortcut (borrowed from `sync/snap/constants.nim`)
Aristo db update foreground caching (#1605) * Fix vertex ID generator state handling for rocksdb backend why: * Key error in walk iterator * Needs to be loaded when opening the database * Use non-zero sub-table prefixes for rocksdb why: Handy for debugging * Fix error code for missing key on rocksdb backend why: Previously returned `VOID_HASH_KEY` rather than `GetKeyNotFound` * Explicitly copy vertex data between internal table and function/result argument why: Function argument or return reference may still refer to the same data object. * Updated error symbols why: Error symbol names for the hike module now start with the prefix `Hike`. * Write back modified branch node into local top layer cache why: With the backend available, the source of the branch node references might not be the top layer cache. So any change must be explicitely recorded.
2023-06-22 11:13:24 +00:00
EmptyVidSeq* = seq[VertexID].default
## Useful shortcut
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
EmptyVidSet* = EmptyVidSeq.toHashSet
Aristo db store filters on backend (#1703) * Simplify RocksDB sub-tables iterator * Implement `filter` storage on backend db details: Unit tests working
2023-08-22 18:44:54 +00:00
## Useful shortcut
Extended data Payload specs for the backend. (#1630) why: For the main tree with root vertex ID 1, the leaf nodes hold the account data. These accounts may link to sub trees the storage root node ID of which must be registered here. There is no reverse key lookup on the backend. note: These definitions are experimental. Also, there are some tests missing for validating Payload data conversions.
2023-07-05 20:27:48 +00:00
VOID_CODE_HASH* = EMPTY_CODE_HASH
Aristo db address sup trie items properly (#1600) * Fix include why: Eth67 not default yet so that got missed * Rename `LeafKey` => `LeafTie` why: Name is a pen picture of what this object is for. Also, it avoids the ubiquitous term `key`. * Provided `getOrVoid()` wrapper for `getOrDefault()` also: Provide `isValid()` syntactic sugar for `.isNil.not`, `!= 0` etc. Reorg descriptor source, split into sub-sources * Bundled `NodeKey` objects with root ID and called it `HashLabel` why: `NodeKey` (aka repurposed Hash265) objects are unique only within a particular sub-trie (e.g. storage slots) which are kept separated (i.e non-interleaved) by design. This is not applied to the backend as the map VertexID->NodeKey labelling the nodes needs not be injective. For the in-memory database (transaction) layers, the injective map VertexID->(VertexID,NodeKey) is used where the first field of the image tuple is the root ID of the sub-trie the `NodeKey` object is valid. So identical storage tries for different accounts can be represented.
2023-06-12 13:48:47 +00:00
## Equivalent of `nil` for `Account` object code hash
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
VOID_HASH_KEY* = HashKey()
Aristo db traversal helpers (#1638) * Misc fixes detail: * Fix de-serialisation for account leafs * Update node recovery from unit tests * Remove `LegacyAccount` from `PayloadRef` object why: Legacy accounts use a hash key as storage root which is detrimental to the working of the Aristo database which uses a vertex ID. * Dissolve `hashify_helper` into `aristo_utils` and `aristo_transcode` why: Functions are of general interest so they should live in first level code files. * Added left/right iterators over leaf nodes * Some helper/wrapper functions that might be useful
2023-07-12 23:03:14 +00:00
## Void equivalent for Merkle hash value
Detach from snap/sync declarations & definitions (#1601) why: Tests and some basic components were originally borrowed from the snap/sync implementation. These have fully been re-implemented.
2023-06-12 18:16:03 +00:00
Aristo db update merkle hasher (#1925) * Register paths for added leafs because of trie re-balancing why: While the payload would not change, the prefix in the leaf vertex would. So it needs to be flagged for hash recompilation for the `hashify()` module. also: Make sure that `Hike` paths which might have vertex links into the backend filter are replaced by vertex copies before manipulating. Otherwise the vertices on the immutable filter might be involuntarily changed. * Also check for paths where the leaf vertex is on the backend, already why: A a path can have dome vertices on the top layer cache with the `Leaf` vertex on the backend. * Re-define a void `HashLabel` type. why: A `HashLabel` type is a pair `(root-vertex-ID, Keccak-hash)`. Previously, a valid `HashLabel` consisted of a non-empty hash and a non-zero vertex ID. This definition leads to a non-unique representation of a void `HashLabel` with either root-ID or has void. This has been changed to the unique void `HashLabel` exactly if the hash entry is void. * Update consistency checkers * Re-org `hashify()` procedure why: Syncing against block chain showed serious deficiencies which produced wrong hashes or simply bailed out with error. So all fringe cases (mainly due to deleted entries) could be integrated into the labelling schedule rather than handling separate fringe cases.
2023-12-04 20:39:26 +00:00
VOID_HASH_LABEL* = HashLabel(key: VOID_HASH_KEY)
Aristo db traversal helpers (#1638) * Misc fixes detail: * Fix de-serialisation for account leafs * Update node recovery from unit tests * Remove `LegacyAccount` from `PayloadRef` object why: Legacy accounts use a hash key as storage root which is detrimental to the working of the Aristo database which uses a vertex ID. * Dissolve `hashify_helper` into `aristo_utils` and `aristo_transcode` why: Functions are of general interest so they should live in first level code files. * Added left/right iterators over leaf nodes * Some helper/wrapper functions that might be useful
2023-07-12 23:03:14 +00:00
## Void equivalent for Merkle hash value
Multi layer architecture 4 aristo db (#1581) * Cosmetics, renamed fields (eVtx, bVtx) -> (eVid, bVid) * Multilayered delta architecture for Aristo DB details: Any VertexID or data retrieval needs to go down the rabbit hole and fetch/get/manipulate the bottom layer -- even without explicit backend. * Direct reference to backend from top-level layer why: Some services as the vid management needs to be synchronised among all layers. So access is optimised.
2023-05-14 17:43:01 +00:00
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
EmptyQidPairSeq* = seq[(QueueID,QueueID)].default
## Useful shortcut
Aristo db implement filter storage scheduler (#1713) * Rename FilterID => QueueID why: The current usage does not identify a particular filter but uses it as storage tag to manage it on the database (to be organised in a set of FIFOs or queues.) * Split `aristo_filter` source into sub-files why: Make space for filter management API * Store filter queue IDs in pairs on the backend why: Any pair will will describe a FIFO accessed by bottom/top IDs * Reorg some source file names why: The "aristo_" prefix for make local/private files is tedious to use, so removed. * Implement filter slot scheduler details: Filters will be stored on the database on cascaded FIFOs. When a FIFO queue is full, some filter items are bundled together and stored on the next FIFO.
2023-08-25 22:53:59 +00:00
DEFAULT_QID_QUEUES* = [
(128, 0), ## Consecutive list of 128 filter slots
( 64, 63), ## Overflow list, 64 filters, skipping 63 filters in-between
( 64, 127), ## ..
( 64, 255)]
Multi layer architecture 4 aristo db (#1581) * Cosmetics, renamed fields (eVtx, bVtx) -> (eVid, bVid) * Multilayered delta architecture for Aristo DB details: Any VertexID or data retrieval needs to go down the rabbit hole and fetch/get/manipulate the bottom layer -- even without explicit backend. * Direct reference to backend from top-level layer why: Some services as the vid management needs to be synchronised among all layers. So access is optimised.
2023-05-14 17:43:01 +00:00
# End