status-im
/
nimbus-eth1
mirror of https://github.com/status-im/nimbus-eth1.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth1/nimbus/db/aristo/aristo_desc/desc_error.nim

270 lines
6.4 KiB
Nim
Raw Normal View History

Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 14:25:29 +00:00
# nimbus-eth1
Aristo and ledger small updates (#1888) * Fix debug noise in `hashify()` for perfectly normal situation why: Was previously considered a fixable error * Fix test sample file names why: The larger test file `goerli68161.txt.gz` is already in the local archive. So there is no need to use the smaller one from the external repo. * Activate `accounts_cache` module from `db/ledger` why: A copy of the original `accounts_cache.nim` source to be integrated into the `Ledger` module wrapper which allows to switch between different `accounts_cache` implementations unser tha same API. details: At a later state, the `db/accounts_cache.nim` wrapper will be removed so that there is only one access to that module via `db/ledger/accounts_cache.nim`. * Fix copyright headers in source code
2023-11-08 16:52:25 +00:00
# Copyright (c) 2023 Status Research & Development GmbH
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 14:25:29 +00:00
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or
# http://www.apache.org/licenses/LICENSE-2.0)
# * MIT license ([LICENSE-MIT](LICENSE-MIT) or
# http://opensource.org/licenses/MIT)
# at your option. This file may not be copied, modified, or distributed
# except according to those terms.
type
AristoError* = enum
NothingSerious = 0
Aristo db merkle hashify functionality added (#1593) * Keep vertex ID generator state with each db-layer why: The vertex ID generator state is part of the difference to the below layer * Move otherwise unused source to test directory * Add Merkle hash generator also: * Verification facility for debugging * Empty Merkle key hashes encoded as `EMPTY_ROOT_HASH`
2023-05-30 21:21:15 +00:00
GenericError
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 14:25:29 +00:00
Aristo db supporting forest and layered tx architecture (#1598) * Exclude some storage tests why: These test running on external dumps slipped through. The particular dumps were reported earlier as somehow dodgy. This was changed in `#1457` but having a second look, the change on hexary_interpolate.nim(350) might be incorrect. * Redesign `Aristo DB` descriptor for transaction based layers why: Previous descriptor layout made it cumbersome to push/pop database delta layers. The new architecture keeps each layer with the full delta set relative to the database backend. * Keep root ID as part of the `Patricia Trie` leaf path why; That way, forests are supported
2023-06-09 11:17:37 +00:00
# Rlp decoder, `read()`
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 14:25:29 +00:00
Rlp2Or17ListEntries
RlpBlobExpected
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
RlpBranchHashKeyExpected
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 14:25:29 +00:00
RlpEmptyBlobExpected
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
RlpExtHashKeyExpected
RlpHashKeyExpected
RlpNonEmptyBlobExpected
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 14:25:29 +00:00
RlpOtherException
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
RlpRlpException
# Serialise decoder
SerCantResolveStorageRoot
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 14:25:29 +00:00
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
# Data record transcoders, `deblobify()` and `blobify()`
Aristo db fixes n updates needed for filter fifo (#1728) * Set scheduler state as part of the backend descriptor details: Moved type definitions `QidLayoutRef` and `QidSchedRef` to `desc_structural.nim` so that it shares the same folder as `desc_backend.nim` * Automatic filter queue table initialisation in backend details: Scheduler can be tweaked or completely disabled * Updated backend unit tests details: + some code clean up/beautification, reads better now + disabled persistent filters so that there is no automated filter management which will be implemented next * Prettify/update unit tests source code details: Mostly replacing the `check()` paradigm by `xCheck()` * Somewhat simplified backend type management why: Backend objects are labelled with a `BackendType` symbol where the `BackendVoid` label is implicitly assumed for a `nil` backend object reference. To make it easier, a `kind()` function is used now applicable to `nil` references as well. * Fix DB storage layout for filter objects why: Need to store the filter ID with the object * Implement reverse [] index on fifo why: An integer index argument on `[]` retrieves the QueueID (label) of the fifo item while a QueueID argument on `[]` retrieves the index (so it is inverse to the former variant). * Provide iterator over filters as fifo why: This iterator goes along the cascased fifo structure (i.e. in historical order)
2023-09-05 13:57:20 +00:00
BlobifyNilFilter
BlobifyNilVertex
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
BlobifyBranchMissingRefs
BlobifyExtMissingRefs
BlobifyExtPathOverflow
BlobifyLeafPathOverflow
Aristo db implement filter serialisation for storage (#1695) * Remove concept of empty/blind filters why: Not needed. A non-existent filter is is coded as a nil reference. * Slightly generalised backend iterators why: * VertexID as key for the ID generator state makes no sense * there will be more tables addressed by non-VertexID keys * Store serialised/blobified vertices on memory backend why: This is more in line with the RocksDB backend so more appropriate for testing when comparing behaviour. For a speedy memory database, a backend-less variant should be used. * Drop the `Aristo` prefix from names `AristoLayerRef`, etc. * Suppress compiler warning why: duplicate imports * Add filter serialisation transcoder why: Will be used as storage format
2023-08-18 19:46:55 +00:00
BlobifyFilterRecordOverflow
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
DeblobNilArgument
DeblobUnknown
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
DeblobVtxTooShort
DeblobHashKeyExpected
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
DeblobBranchTooShort
DeblobBranchSizeGarbled
DeblobBranchInxOutOfRange
DeblobExtTooShort
DeblobExtSizeGarbled
DeblobExtGotLeafPrefix
DeblobLeafSizeGarbled
DeblobLeafGotExtPrefix
DeblobSizeGarbled
DeblobWrongType
Extended data Payload specs for the backend. (#1630) why: For the main tree with root vertex ID 1, the leaf nodes hold the account data. These accounts may link to sub trees the storage root node ID of which must be registered here. There is no reverse key lookup on the backend. note: These definitions are experimental. Also, there are some tests missing for validating Payload data conversions.
2023-07-05 20:27:48 +00:00
DeblobPayloadTooShortInt64
DeblobPayloadTooShortInt256
DeblobNonceLenUnsupported
DeblobBalanceLenUnsupported
DeblobStorageLenUnsupported
DeblobCodeLenUnsupported
Aristo db implement filter serialisation for storage (#1695) * Remove concept of empty/blind filters why: Not needed. A non-existent filter is is coded as a nil reference. * Slightly generalised backend iterators why: * VertexID as key for the ID generator state makes no sense * there will be more tables addressed by non-VertexID keys * Store serialised/blobified vertices on memory backend why: This is more in line with the RocksDB backend so more appropriate for testing when comparing behaviour. For a speedy memory database, a backend-less variant should be used. * Drop the `Aristo` prefix from names `AristoLayerRef`, etc. * Suppress compiler warning why: duplicate imports * Add filter serialisation transcoder why: Will be used as storage format
2023-08-18 19:46:55 +00:00
DeblobFilterTooShort
DeblobFilterGenTooShort
DeblobFilterTrpTooShort
DeblobFilterTrpVtxSizeGarbled
DeblobFilterSizeGarbled
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 14:25:29 +00:00
Aristo db supporting forest and layered tx architecture (#1598) * Exclude some storage tests why: These test running on external dumps slipped through. The particular dumps were reported earlier as somehow dodgy. This was changed in `#1457` but having a second look, the change on hexary_interpolate.nim(350) might be incorrect. * Redesign `Aristo DB` descriptor for transaction based layers why: Previous descriptor layout made it cumbersome to push/pop database delta layers. The new architecture keeps each layer with the full delta set relative to the database backend. * Keep root ID as part of the `Patricia Trie` leaf path why; That way, forests are supported
2023-06-09 11:17:37 +00:00
# Converter `asNode()`, currenly for unit tests only
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 14:25:29 +00:00
CacheMissingNodekeys
Aristo db supporting forest and layered tx architecture (#1598) * Exclude some storage tests why: These test running on external dumps slipped through. The particular dumps were reported earlier as somehow dodgy. This was changed in `#1457` but having a second look, the change on hexary_interpolate.nim(350) might be incorrect. * Redesign `Aristo DB` descriptor for transaction based layers why: Previous descriptor layout made it cumbersome to push/pop database delta layers. The new architecture keeps each layer with the full delta set relative to the database backend. * Keep root ID as part of the `Patricia Trie` leaf path why; That way, forests are supported
2023-06-09 11:17:37 +00:00
# Path function `hikeUp()`
Aristo db update foreground caching (#1605) * Fix vertex ID generator state handling for rocksdb backend why: * Key error in walk iterator * Needs to be loaded when opening the database * Use non-zero sub-table prefixes for rocksdb why: Handy for debugging * Fix error code for missing key on rocksdb backend why: Previously returned `VOID_HASH_KEY` rather than `GetKeyNotFound` * Explicitly copy vertex data between internal table and function/result argument why: Function argument or return reference may still refer to the same data object. * Updated error symbols why: Error symbol names for the hike module now start with the prefix `Hike`. * Write back modified branch node into local top layer cache why: With the backend available, the source of the branch node references might not be the top layer cache. So any change must be explicitely recorded.
2023-06-22 11:13:24 +00:00
HikeRootMissing
Aristo db allow shorter than 64 nibbles path keys (#1864) * Aristo: Single `FetchPathNotFound` error in `fetchXxx()` and `hasPath()` why: Missing path hike returns too many detailed reasons why it failed which becomes cumbersome to handle. also: Renamed `contains()` => `hasPath()` which disables the `in` operator on non-boolean `contains()` functions * Kvt: Renamed `contains()` => `hasKey()` why: which disables the `in` operator on non-boolean `contains()` functions * Aristo: Generalising `HashID` by variable length `PathID` why: There are cases when the `Aristo` database is to be used with shorter than 64 nibbles keys when handling transactions indexes with sequence IDs. caveat: This patch only works reliable for full length `PathID` values. Tests for shorter `PathID` values are currently missing.
2023-10-27 21:36:51 +00:00
HikeEmptyPath
Aristo db update foreground caching (#1605) * Fix vertex ID generator state handling for rocksdb backend why: * Key error in walk iterator * Needs to be loaded when opening the database * Use non-zero sub-table prefixes for rocksdb why: Handy for debugging * Fix error code for missing key on rocksdb backend why: Previously returned `VOID_HASH_KEY` rather than `GetKeyNotFound` * Explicitly copy vertex data between internal table and function/result argument why: Function argument or return reference may still refer to the same data object. * Updated error symbols why: Error symbol names for the hike module now start with the prefix `Hike`. * Write back modified branch node into local top layer cache why: With the backend available, the source of the branch node references might not be the top layer cache. So any change must be explicitely recorded.
2023-06-22 11:13:24 +00:00
HikeLeafTooEarly
HikeBranchTailEmpty
HikeBranchBlindEdge
HikeExtTailEmpty
HikeExtTailMismatch
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 11:47:47 +00:00
# Path/nibble/key conversions in `aisto_path.nim`
PathExpected64Nibbles
Aristo db allow shorter than 64 nibbles path keys (#1864) * Aristo: Single `FetchPathNotFound` error in `fetchXxx()` and `hasPath()` why: Missing path hike returns too many detailed reasons why it failed which becomes cumbersome to handle. also: Renamed `contains()` => `hasPath()` which disables the `in` operator on non-boolean `contains()` functions * Kvt: Renamed `contains()` => `hasKey()` why: which disables the `in` operator on non-boolean `contains()` functions * Aristo: Generalising `HashID` by variable length `PathID` why: There are cases when the `Aristo` database is to be used with shorter than 64 nibbles keys when handling transactions indexes with sequence IDs. caveat: This patch only works reliable for full length `PathID` values. Tests for shorter `PathID` values are currently missing.
2023-10-27 21:36:51 +00:00
PathAtMost64Nibbles
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 11:47:47 +00:00
PathExpectedLeaf
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 14:25:29 +00:00
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 11:47:47 +00:00
# Merge leaf `merge()`
Aristo db update merkle hasher (#1925) * Register paths for added leafs because of trie re-balancing why: While the payload would not change, the prefix in the leaf vertex would. So it needs to be flagged for hash recompilation for the `hashify()` module. also: Make sure that `Hike` paths which might have vertex links into the backend filter are replaced by vertex copies before manipulating. Otherwise the vertices on the immutable filter might be involuntarily changed. * Also check for paths where the leaf vertex is on the backend, already why: A a path can have dome vertices on the top layer cache with the `Leaf` vertex on the backend. * Re-define a void `HashLabel` type. why: A `HashLabel` type is a pair `(root-vertex-ID, Keccak-hash)`. Previously, a valid `HashLabel` consisted of a non-empty hash and a non-zero vertex ID. This definition leads to a non-unique representation of a void `HashLabel` with either root-ID or has void. This has been changed to the unique void `HashLabel` exactly if the hash entry is void. * Update consistency checkers * Re-org `hashify()` procedure why: Syncing against block chain showed serious deficiencies which produced wrong hashes or simply bailed out with error. So all fringe cases (mainly due to deleted entries) could be integrated into the labelling schedule rather than handling separate fringe cases.
2023-12-04 20:39:26 +00:00
MergeBranchLinkLeafGarbled
MergeBranchLinkVtxPfxTooShort
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 11:47:47 +00:00
MergeBranchGarbledNibble
MergeBranchGarbledTail
Aristo db merkle hashify functionality added (#1593) * Keep vertex ID generator state with each db-layer why: The vertex ID generator state is part of the difference to the below layer * Move otherwise unused source to test directory * Add Merkle hash generator also: * Verification facility for debugging * Empty Merkle key hashes encoded as `EMPTY_ROOT_HASH`
2023-05-30 21:21:15 +00:00
MergeBranchLinkLockedKey
MergeBranchLinkProofModeLock
MergeBranchProofModeLock
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 11:47:47 +00:00
MergeBranchRootExpected
MergeLeafGarbledHike
Aristo db merkle hashify functionality added (#1593) * Keep vertex ID generator state with each db-layer why: The vertex ID generator state is part of the difference to the below layer * Move otherwise unused source to test directory * Add Merkle hash generator also: * Verification facility for debugging * Empty Merkle key hashes encoded as `EMPTY_ROOT_HASH`
2023-05-30 21:21:15 +00:00
MergeLeafPathCachedAlready
Aristo db update merkle hasher (#1925) * Register paths for added leafs because of trie re-balancing why: While the payload would not change, the prefix in the leaf vertex would. So it needs to be flagged for hash recompilation for the `hashify()` module. also: Make sure that `Hike` paths which might have vertex links into the backend filter are replaced by vertex copies before manipulating. Otherwise the vertices on the immutable filter might be involuntarily changed. * Also check for paths where the leaf vertex is on the backend, already why: A a path can have dome vertices on the top layer cache with the `Leaf` vertex on the backend. * Re-define a void `HashLabel` type. why: A `HashLabel` type is a pair `(root-vertex-ID, Keccak-hash)`. Previously, a valid `HashLabel` consisted of a non-empty hash and a non-zero vertex ID. This definition leads to a non-unique representation of a void `HashLabel` with either root-ID or has void. This has been changed to the unique void `HashLabel` exactly if the hash entry is void. * Update consistency checkers * Re-org `hashify()` procedure why: Syncing against block chain showed serious deficiencies which produced wrong hashes or simply bailed out with error. So all fringe cases (mainly due to deleted entries) could be integrated into the labelling schedule rather than handling separate fringe cases.
2023-12-04 20:39:26 +00:00
MergeLeafPathOnBackendAlready
Aristo db merkle hashify functionality added (#1593) * Keep vertex ID generator state with each db-layer why: The vertex ID generator state is part of the difference to the below layer * Move otherwise unused source to test directory * Add Merkle hash generator also: * Verification facility for debugging * Empty Merkle key hashes encoded as `EMPTY_ROOT_HASH`
2023-05-30 21:21:15 +00:00
MergeNonBranchProofModeLock
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 11:47:47 +00:00
MergeRootBranchLinkBusy
Aristo db update foreground caching (#1605) * Fix vertex ID generator state handling for rocksdb backend why: * Key error in walk iterator * Needs to be loaded when opening the database * Use non-zero sub-table prefixes for rocksdb why: Handy for debugging * Fix error code for missing key on rocksdb backend why: Previously returned `VOID_HASH_KEY` rather than `GetKeyNotFound` * Explicitly copy vertex data between internal table and function/result argument why: Function argument or return reference may still refer to the same data object. * Updated error symbols why: Error symbol names for the hike module now start with the prefix `Hike`. * Write back modified branch node into local top layer cache why: With the backend available, the source of the branch node references might not be the top layer cache. So any change must be explicitely recorded.
2023-06-22 11:13:24 +00:00
MergeAssemblyFailed # Ooops, internal error
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 11:47:47 +00:00
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
MergeHashKeyInvalid
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
MergeHashKeyCachedAlready
MergeHashKeyDiffersFromCached
MergeHashKeyRevLookUpGarbled
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
MergeRootVidInvalid
MergeRootKeyInvalid
MergeRevVidMustHaveBeenCached
MergeNodeVtxDiffersFromExisting
Aristo db supporting forest and layered tx architecture (#1598) * Exclude some storage tests why: These test running on external dumps slipped through. The particular dumps were reported earlier as somehow dodgy. This was changed in `#1457` but having a second look, the change on hexary_interpolate.nim(350) might be incorrect. * Redesign `Aristo DB` descriptor for transaction based layers why: Previous descriptor layout made it cumbersome to push/pop database delta layers. The new architecture keeps each layer with the full delta set relative to the database backend. * Keep root ID as part of the `Patricia Trie` leaf path why; That way, forests are supported
2023-06-09 11:17:37 +00:00
MergeRootKeyDiffersForVid
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
MergeNodeVtxDuplicates
Aristo db merkle hashify functionality added (#1593) * Keep vertex ID generator state with each db-layer why: The vertex ID generator state is part of the difference to the below layer * Move otherwise unused source to test directory * Add Merkle hash generator also: * Verification facility for debugging * Empty Merkle key hashes encoded as `EMPTY_ROOT_HASH`
2023-05-30 21:21:15 +00:00
# Update `Merkle` hashes `hashify()`
Aristo db update merkle hasher (#1925) * Register paths for added leafs because of trie re-balancing why: While the payload would not change, the prefix in the leaf vertex would. So it needs to be flagged for hash recompilation for the `hashify()` module. also: Make sure that `Hike` paths which might have vertex links into the backend filter are replaced by vertex copies before manipulating. Otherwise the vertices on the immutable filter might be involuntarily changed. * Also check for paths where the leaf vertex is on the backend, already why: A a path can have dome vertices on the top layer cache with the `Leaf` vertex on the backend. * Re-define a void `HashLabel` type. why: A `HashLabel` type is a pair `(root-vertex-ID, Keccak-hash)`. Previously, a valid `HashLabel` consisted of a non-empty hash and a non-zero vertex ID. This definition leads to a non-unique representation of a void `HashLabel` with either root-ID or has void. This has been changed to the unique void `HashLabel` exactly if the hash entry is void. * Update consistency checkers * Re-org `hashify()` procedure why: Syncing against block chain showed serious deficiencies which produced wrong hashes or simply bailed out with error. So all fringe cases (mainly due to deleted entries) could be integrated into the labelling schedule rather than handling separate fringe cases.
2023-12-04 20:39:26 +00:00
HashifyEmptyHike
Aristo db merkle hashify functionality added (#1593) * Keep vertex ID generator state with each db-layer why: The vertex ID generator state is part of the difference to the below layer * Move otherwise unused source to test directory * Add Merkle hash generator also: * Verification facility for debugging * Empty Merkle key hashes encoded as `EMPTY_ROOT_HASH`
2023-05-30 21:21:15 +00:00
HashifyExistingHashMismatch
Aristo db update merkle hasher (#1925) * Register paths for added leafs because of trie re-balancing why: While the payload would not change, the prefix in the leaf vertex would. So it needs to be flagged for hash recompilation for the `hashify()` module. also: Make sure that `Hike` paths which might have vertex links into the backend filter are replaced by vertex copies before manipulating. Otherwise the vertices on the immutable filter might be involuntarily changed. * Also check for paths where the leaf vertex is on the backend, already why: A a path can have dome vertices on the top layer cache with the `Leaf` vertex on the backend. * Re-define a void `HashLabel` type. why: A `HashLabel` type is a pair `(root-vertex-ID, Keccak-hash)`. Previously, a valid `HashLabel` consisted of a non-empty hash and a non-zero vertex ID. This definition leads to a non-unique representation of a void `HashLabel` with either root-ID or has void. This has been changed to the unique void `HashLabel` exactly if the hash entry is void. * Update consistency checkers * Re-org `hashify()` procedure why: Syncing against block chain showed serious deficiencies which produced wrong hashes or simply bailed out with error. So all fringe cases (mainly due to deleted entries) could be integrated into the labelling schedule rather than handling separate fringe cases.
2023-12-04 20:39:26 +00:00
HashifyNodeUnresolved
Aristo db merkle hashify functionality added (#1593) * Keep vertex ID generator state with each db-layer why: The vertex ID generator state is part of the difference to the below layer * Move otherwise unused source to test directory * Add Merkle hash generator also: * Verification facility for debugging * Empty Merkle key hashes encoded as `EMPTY_ROOT_HASH`
2023-05-30 21:21:15 +00:00
HashifyRootHashMismatch
Aristo db update merkle hasher (#1925) * Register paths for added leafs because of trie re-balancing why: While the payload would not change, the prefix in the leaf vertex would. So it needs to be flagged for hash recompilation for the `hashify()` module. also: Make sure that `Hike` paths which might have vertex links into the backend filter are replaced by vertex copies before manipulating. Otherwise the vertices on the immutable filter might be involuntarily changed. * Also check for paths where the leaf vertex is on the backend, already why: A a path can have dome vertices on the top layer cache with the `Leaf` vertex on the backend. * Re-define a void `HashLabel` type. why: A `HashLabel` type is a pair `(root-vertex-ID, Keccak-hash)`. Previously, a valid `HashLabel` consisted of a non-empty hash and a non-zero vertex ID. This definition leads to a non-unique representation of a void `HashLabel` with either root-ID or has void. This has been changed to the unique void `HashLabel` exactly if the hash entry is void. * Update consistency checkers * Re-org `hashify()` procedure why: Syncing against block chain showed serious deficiencies which produced wrong hashes or simply bailed out with error. So all fringe cases (mainly due to deleted entries) could be integrated into the labelling schedule rather than handling separate fringe cases.
2023-12-04 20:39:26 +00:00
HashifyRootNodeUnresolved
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
# Cache checker `checkCache()`
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
CheckStkKeyStrayZeroEntry
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
CheckStkRevKeyMismatch
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
CheckStkRevKeyMissing
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
CheckStkVtxCountMismatch
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
CheckStkVtxIncomplete
CheckStkVtxKeyMismatch
CheckStkVtxKeyMissing
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
CheckRlxVidVtxMismatch
CheckRlxVtxIncomplete
CheckRlxVtxKeyMissing
CheckRlxVtxKeyMismatch
CheckRlxRevKeyMissing
CheckRlxRevKeyMismatch
Aristo db tidy up a bit (#1625) * Slightly tighten some self-check conditions * Redefined the database descriptor object as reference (to the object) why: The upcoming transaction wrapper will work with a database reference rather than the object itself * Append state before `save()` to the Aristo descriptor why: This stae was previously returned by the function. Appending it to a field of the Aristo descriptor seems easier to handle.
2023-07-04 18:24:03 +00:00
CheckAnyVtxEmptyKeyMissing
CheckAnyVtxEmptyKeyExpected
CheckAnyVtxEmptyKeyMismatch
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
CheckAnyVtxBranchLinksMissing
CheckAnyVtxExtPfxMissing
CheckAnyVtxLockWithoutKey
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
CheckAnyRevVtxMissing
CheckAnyRevVtxDup
CheckAnyRevCountMismatch
# Backend structural check `checkBE()`
CheckBeVtxInvalid
CheckBeVtxMissing
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
CheckBeVtxBranchLinksMissing
CheckBeVtxExtPfxMissing
CheckBeKeyInvalid
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
CheckBeKeyMissing
CheckBeKeyCantCompile
CheckBeKeyMismatch
CheckBeGarbledVGen
Aristo db implement distributed backend access (#1688) * Fix hashing algorithm why: Particular case where a sub-tree is on the backend, linked by an Extension vertex to the top level. * Update backend verification to report `dirty` top layer * Implement distributed merge of backend filters * Implement distributed backend access management details: Implemented and tested as described in chapter 5 of the `README.md` file.
2023-08-17 13:42:01 +00:00
CheckBeCacheIsDirty
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
CheckBeCacheKeyMissing
CheckBeCacheKeyNonEmpty
CheckBeCacheVidUnsynced
CheckBeCacheKeyDangling
CheckBeCacheVtxDangling
CheckBeCacheKeyCantCompile
CheckBeCacheKeyMismatch
CheckBeCacheGarbledVGen
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 11:47:47 +00:00
Aristo db state root repos and reorg (#1744) * Reorg of distributed backend access details: Now handled via API provided in `aristo_desc`. * Rename `checkCache()` => `checkTop()` why: Better naming for top layer cache checker also: Provide cascaded fifos checker * Provide `eq` directive for finding filter by exact filter ID (think block number) * Some code beautification (for better code reading) * State root reposition and reorg details: Repositioning is supported by forking a new descriptor. Reorg is then accomplished by writing this forked state on the backend database.
2023-09-11 20:38:49 +00:00
CheckBeFifoSrcTrgMismatch
CheckBeFifoTrgNotStateRoot
Aristo db fixes after storage slots dump tests added (#1595) * Fix missing Merkle key removal in `merge()` * Accept optional root hash argument in `hashify()` why: For importing a full database, there will be no proof data except the root key. So this can be used to check and set the root key in the database descriptor. also: Associate vertex ID to `hashify()` error return code * Added Aristo Trie traversal function why: * step along leaf vertices in sorted order * tree/trie consistency checks when debugging * Enabled storage slots test data for Aristo DB
2023-06-02 10:04:29 +00:00
# Neighbour vertex, tree traversal `nearbyRight()` and `nearbyLeft()`
NearbyBeyondRange
NearbyBranchError
NearbyDanglingLink
NearbyEmptyHike
NearbyExtensionError
NearbyFailed
NearbyBranchExpected
NearbyLeafExpected
NearbyNestingTooDeep
NearbyPathTailUnexpected
NearbyUnexpectedVtx
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
NearbyVidInvalid
Aristo db fixes after storage slots dump tests added (#1595) * Fix missing Merkle key removal in `merge()` * Accept optional root hash argument in `hashify()` why: For importing a full database, there will be no proof data except the root key. So this can be used to check and set the root key in the database descriptor. also: Associate vertex ID to `hashify()` error return code * Added Aristo Trie traversal function why: * step along leaf vertices in sorted order * tree/trie consistency checks when debugging * Enabled storage slots test data for Aristo DB
2023-06-02 10:04:29 +00:00
Added delete fuctionality (#1596)
2023-06-02 19:21:46 +00:00
# Deletion of vertices, `delete()`
DelPathTagError
DelLeafExpexted
DelLeafLocked
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
DelLeafUnexpected
Added delete fuctionality (#1596)
2023-06-02 19:21:46 +00:00
DelBranchExpexted
DelBranchLocked
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
DelBranchWithoutRefs
Added delete fuctionality (#1596)
2023-06-02 19:21:46 +00:00
DelExtLocked
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
DelVidStaleVtx
Added delete fuctionality (#1596)
2023-06-02 19:21:46 +00:00
Aristo db use filter betw backend and tx cache (#1678) * Provide deep copy for each transaction layer why: Localising changes. Selective deep copy was just overlooked. * Generalise vertex ID generator state reorg function `vidReorg()` why: makes it somewhat easier to handle when saving layers. * Provide dummy back end descriptor `NoneBackendRef` * Optional read-only filter between backend and transaction cache why: Some staging area for accumulating changes to the backend DB. This will eventually be an access layer for emulating a backend with multiple/historic state roots. * Re-factor `persistent()` with filter between backend/tx-cache => `stow()` why: The filter provides an abstraction from the physically stored data on disk. So, there can be several MPT instances using the same disk data with different state roots. Of course, all the MPT instances should not differ too much for practical reasons :). TODO: Filter administration tools need to be provided.
2023-08-10 20:01:28 +00:00
# Functions from `aristo_filter.nim`
Implement backend filter mechanics (#1730) details: * Tested features + Successively store filters with increasing filter ID (think block number) + Cascading through fifos, deeper fifos merge groups of filters + Fetch squash merged N top fifos + Delete N top fifos, push back merged fifo, continue storing + Fifo chain is verified by hashes and filter ID * Not tested yet + Real live scenario (using data dumps) + Real filter data (only shallow filters used so far)
2023-09-05 18:00:40 +00:00
FilBackendMissing
FilBackendRoMode
Aristo db state root repos and reorg (#1744) * Reorg of distributed backend access details: Now handled via API provided in `aristo_desc`. * Rename `checkCache()` => `checkTop()` why: Better naming for top layer cache checker also: Provide cascaded fifos checker * Provide `eq` directive for finding filter by exact filter ID (think block number) * Some code beautification (for better code reading) * State root reposition and reorg details: Repositioning is supported by forking a new descriptor. Reorg is then accomplished by writing this forked state on the backend database.
2023-09-11 20:38:49 +00:00
FilBackStepsExpected
Aristo db implement distributed backend access (#1688) * Fix hashing algorithm why: Particular case where a sub-tree is on the backend, linked by an Extension vertex to the top level. * Update backend verification to report `dirty` top layer * Implement distributed merge of backend filters * Implement distributed backend access management details: Implemented and tested as described in chapter 5 of the `README.md` file.
2023-08-17 13:42:01 +00:00
FilDudeFilterUpdateError
Implement backend filter mechanics (#1730) details: * Tested features + Successively store filters with increasing filter ID (think block number) + Cascading through fifos, deeper fifos merge groups of filters + Fetch squash merged N top fifos + Delete N top fifos, push back merged fifo, continue storing + Fifo chain is verified by hashes and filter ID * Not tested yet + Real live scenario (using data dumps) + Real filter data (only shallow filters used so far)
2023-09-05 18:00:40 +00:00
FilExecDublicateSave
FilExecHoldExpected
FilExecOops
FilExecSaveMissing
FilExecStackUnderflow
Aristo db state root repos and reorg (#1744) * Reorg of distributed backend access details: Now handled via API provided in `aristo_desc`. * Rename `checkCache()` => `checkTop()` why: Better naming for top layer cache checker also: Provide cascaded fifos checker * Provide `eq` directive for finding filter by exact filter ID (think block number) * Some code beautification (for better code reading) * State root reposition and reorg details: Repositioning is supported by forking a new descriptor. Reorg is then accomplished by writing this forked state on the backend database.
2023-09-11 20:38:49 +00:00
FilFilterInvalid
FilFilterNotFound
FilInxByQidFailed
FilNegativeEpisode
Implement backend filter mechanics (#1730) details: * Tested features + Successively store filters with increasing filter ID (think block number) + Cascading through fifos, deeper fifos merge groups of filters + Fetch squash merged N top fifos + Delete N top fifos, push back merged fifo, continue storing + Fifo chain is verified by hashes and filter ID * Not tested yet + Real live scenario (using data dumps) + Real filter data (only shallow filters used so far)
2023-09-05 18:00:40 +00:00
FilNilFilterRejected
Aristo db state root repos and reorg (#1744) * Reorg of distributed backend access details: Now handled via API provided in `aristo_desc`. * Rename `checkCache()` => `checkTop()` why: Better naming for top layer cache checker also: Provide cascaded fifos checker * Provide `eq` directive for finding filter by exact filter ID (think block number) * Some code beautification (for better code reading) * State root reposition and reorg details: Repositioning is supported by forking a new descriptor. Reorg is then accomplished by writing this forked state on the backend database.
2023-09-11 20:38:49 +00:00
FilNoMatchOnFifo
Implement backend filter mechanics (#1730) details: * Tested features + Successively store filters with increasing filter ID (think block number) + Cascading through fifos, deeper fifos merge groups of filters + Fetch squash merged N top fifos + Delete N top fifos, push back merged fifo, continue storing + Fifo chain is verified by hashes and filter ID * Not tested yet + Real live scenario (using data dumps) + Real filter data (only shallow filters used so far)
2023-09-05 18:00:40 +00:00
FilPrettyPointlessLayer
FilQidByLeFidFailed
Aristo db fixes n updates needed for filter fifo (#1728) * Set scheduler state as part of the backend descriptor details: Moved type definitions `QidLayoutRef` and `QidSchedRef` to `desc_structural.nim` so that it shares the same folder as `desc_backend.nim` * Automatic filter queue table initialisation in backend details: Scheduler can be tweaked or completely disabled * Updated backend unit tests details: + some code clean up/beautification, reads better now + disabled persistent filters so that there is no automated filter management which will be implemented next * Prettify/update unit tests source code details: Mostly replacing the `check()` paradigm by `xCheck()` * Somewhat simplified backend type management why: Backend objects are labelled with a `BackendType` symbol where the `BackendVoid` label is implicitly assumed for a `nil` backend object reference. To make it easier, a `kind()` function is used now applicable to `nil` references as well. * Fix DB storage layout for filter objects why: Need to store the filter ID with the object * Implement reverse [] index on fifo why: An integer index argument on `[]` retrieves the QueueID (label) of the fifo item while a QueueID argument on `[]` retrieves the index (so it is inverse to the former variant). * Provide iterator over filters as fifo why: This iterator goes along the cascased fifo structure (i.e. in historical order)
2023-09-05 13:57:20 +00:00
FilQuSchedDisabled
Implement backend filter mechanics (#1730) details: * Tested features + Successively store filters with increasing filter ID (think block number) + Cascading through fifos, deeper fifos merge groups of filters + Fetch squash merged N top fifos + Delete N top fifos, push back merged fifo, continue storing + Fifo chain is verified by hashes and filter ID * Not tested yet + Real live scenario (using data dumps) + Real filter data (only shallow filters used so far)
2023-09-05 18:00:40 +00:00
FilStateRootMismatch
FilStateRootMissing
FilTrgSrcMismatch
FilTrgTopSrcMismatch
Aristo db state root repos and reorg (#1744) * Reorg of distributed backend access details: Now handled via API provided in `aristo_desc`. * Rename `checkCache()` => `checkTop()` why: Better naming for top layer cache checker also: Provide cascaded fifos checker * Provide `eq` directive for finding filter by exact filter ID (think block number) * Some code beautification (for better code reading) * State root reposition and reorg details: Repositioning is supported by forking a new descriptor. Reorg is then accomplished by writing this forked state on the backend database.
2023-09-11 20:38:49 +00:00
FilSiblingsCommitUnfinshed
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
Aristo db allow shorter than 64 nibbles path keys (#1864) * Aristo: Single `FetchPathNotFound` error in `fetchXxx()` and `hasPath()` why: Missing path hike returns too many detailed reasons why it failed which becomes cumbersome to handle. also: Renamed `contains()` => `hasPath()` which disables the `in` operator on non-boolean `contains()` functions * Kvt: Renamed `contains()` => `hasKey()` why: which disables the `in` operator on non-boolean `contains()` functions * Aristo: Generalising `HashID` by variable length `PathID` why: There are cases when the `Aristo` database is to be used with shorter than 64 nibbles keys when handling transactions indexes with sequence IDs. caveat: This patch only works reliable for full length `PathID` values. Tests for shorter `PathID` values are currently missing.
2023-10-27 21:36:51 +00:00
# Get functions from `aristo_get.nim`
Core db use differential tx layers for aristo and kvt (#1949) * Fix kvt headers * Provide differential layers for KVT transaction stack why: Significant performance improvement * Provide abstraction layer for database top cache layer why: This will eventually implemented as a differential database layers or transaction layers. The latter is needed to improve performance. behavioural changes: Zero vertex and keys (i.e. delete requests) are not optimised out until the last layer is written to the database. * Provide differential layers for Aristo transaction stack why: Significant performance improvement
2023-12-19 12:39:23 +00:00
GetLeafMissing
GetKeyUpdateNeeded
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
GetLeafNotFound
GetVtxNotFound
GetKeyNotFound
Aristo db update serialisation (#1700) * Remove unused unit test sources * Redefine and document serialised data records for Aristo backend why: Unique record types determined by marker byte, i.e. the last byte of a serialisation record. This just needed some tweaking after adding new record types.
2023-08-21 18:18:06 +00:00
GetFilNotFound
Aristo db use filter betw backend and tx cache (#1678) * Provide deep copy for each transaction layer why: Localising changes. Selective deep copy was just overlooked. * Generalise vertex ID generator state reorg function `vidReorg()` why: makes it somewhat easier to handle when saving layers. * Provide dummy back end descriptor `NoneBackendRef` * Optional read-only filter between backend and transaction cache why: Some staging area for accumulating changes to the backend DB. This will eventually be an access layer for emulating a backend with multiple/historic state roots. * Re-factor `persistent()` with filter between backend/tx-cache => `stow()` why: The filter provides an abstraction from the physically stored data on disk. So, there can be several MPT instances using the same disk data with different state roots. Of course, all the MPT instances should not differ too much for practical reasons :). TODO: Filter administration tools need to be provided.
2023-08-10 20:01:28 +00:00
GetIdgNotFound
Aristo db implement filter storage scheduler (#1713) * Rename FilterID => QueueID why: The current usage does not identify a particular filter but uses it as storage tag to manage it on the database (to be organised in a set of FIFOs or queues.) * Split `aristo_filter` source into sub-files why: Make space for filter management API * Store filter queue IDs in pairs on the backend why: Any pair will will describe a FIFO accessed by bottom/top IDs * Reorg some source file names why: The "aristo_" prefix for make local/private files is tedious to use, so removed. * Implement filter slot scheduler details: Filters will be stored on the database on cascaded FIFOs. When a FIFO queue is full, some filter items are bundled together and stored on the next FIFO.
2023-08-25 22:53:59 +00:00
GetFqsNotFound
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
Aristo db allow shorter than 64 nibbles path keys (#1864) * Aristo: Single `FetchPathNotFound` error in `fetchXxx()` and `hasPath()` why: Missing path hike returns too many detailed reasons why it failed which becomes cumbersome to handle. also: Renamed `contains()` => `hasPath()` which disables the `in` operator on non-boolean `contains()` functions * Kvt: Renamed `contains()` => `hasKey()` why: which disables the `in` operator on non-boolean `contains()` functions * Aristo: Generalising `HashID` by variable length `PathID` why: There are cases when the `Aristo` database is to be used with shorter than 64 nibbles keys when handling transactions indexes with sequence IDs. caveat: This patch only works reliable for full length `PathID` values. Tests for shorter `PathID` values are currently missing.
2023-10-27 21:36:51 +00:00
# Fetch functions from `aristo_fetch.nim`
FetchPathNotFound
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
# RocksDB backend
RdbBeCantCreateDataDir
RdbBeCantCreateBackupDir
RdbBeCantCreateTmpDir
RdbBeDriverInitError
RdbBeDriverGetError
RdbBeDriverDelError
RdbBeCreateSstWriter
RdbBeOpenSstWriter
RdbBeAddSstWriter
RdbBeFinishSstWriter
RdbBeIngestSstWriter
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
RdbHashKeyExpected
Aristo db supporting forest and layered tx architecture (#1598) * Exclude some storage tests why: These test running on external dumps slipped through. The particular dumps were reported earlier as somehow dodgy. This was changed in `#1457` but having a second look, the change on hexary_interpolate.nim(350) might be incorrect. * Redesign `Aristo DB` descriptor for transaction based layers why: Previous descriptor layout made it cumbersome to push/pop database delta layers. The new architecture keeps each layer with the full delta set relative to the database backend. * Keep root ID as part of the `Patricia Trie` leaf path why; That way, forests are supported
2023-06-09 11:17:37 +00:00
Aristo db transaction based interface (#1628) * Provide transaction based interface for standard operations * Provide unit tests for new Aristo interface using transactions details: These new tests combine and replace several single-purpose tests. The now unused test sources will be kept for a while to be eventually removed.
2023-07-05 13:50:11 +00:00
# Transaction wrappers
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
TxArgStaleTx
Aristo db code massage n cosmetics (#1745) * Rewrite remaining `AristoError` return code into `Result[void,AristoError]` why: Better code maintenance * Update import sections * Update Aristo DB paths why: More systematic so directory can be shared with other DB types * More cosmetcs * Update unit tests runners why: Proper handling of persistent and mem-only DB. The latter can be consistently triggered by an empty DB path.
2023-09-12 18:45:12 +00:00
TxBackendNotWritable
Aristo db integrate hashify into tx (#1679) * Renamed type `NoneBackendRef` => `VoidBackendRef` * Clarify names: `BE=filter+backend` and `UBE=backend (unfiltered)` why: Most functions used full names as `getVtxUnfilteredBackend()` or `getKeyBackend()`. After defining abbreviations (and its meaning) it seems easier to use `getVtxUBE()` and `getKeyBE()`. * Integrate `hashify()` process into transaction logic why: Is now transparent unless explicitly controlled. details: Cache changes imply setting a `dirty` flag which in turn triggers `hashify()` processing in transaction and `pack()` directives. * Removed `aristo_tx.exec()` directive why: Inconsistent implementation, functionality will be provided with a different paradigm.
2023-08-11 17:23:57 +00:00
TxNoPendingTx
Aristo db implement distributed backend access (#1688) * Fix hashing algorithm why: Particular case where a sub-tree is on the backend, linked by an Extension vertex to the top level. * Update backend verification to report `dirty` top layer * Implement distributed merge of backend filters * Implement distributed backend access management details: Implemented and tested as described in chapter 5 of the `README.md` file.
2023-08-17 13:42:01 +00:00
TxPendingTx
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
TxNotTopTx
Aristo db integrate hashify into tx (#1679) * Renamed type `NoneBackendRef` => `VoidBackendRef` * Clarify names: `BE=filter+backend` and `UBE=backend (unfiltered)` why: Most functions used full names as `getVtxUnfilteredBackend()` or `getKeyBackend()`. After defining abbreviations (and its meaning) it seems easier to use `getVtxUBE()` and `getKeyBE()`. * Integrate `hashify()` process into transaction logic why: Is now transparent unless explicitly controlled. details: Cache changes imply setting a `dirty` flag which in turn triggers `hashify()` processing in transaction and `pack()` directives. * Removed `aristo_tx.exec()` directive why: Inconsistent implementation, functionality will be provided with a different paradigm.
2023-08-11 17:23:57 +00:00
TxStackGarbled
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
TxStackUnderflow
Aristo db api extensions for use as core db backend (#1754) * Update docu * Update Aristo/Kvt constructor prototype why: Previous version used an `enum` value to indicate what backend is to be used. This was replaced by using the backend object type. * Rewrite `hikeUp()` return code into `Result[Hike,(Hike,AristoError)]` why: Better code maintenance. Previously, the `Hike` object was returned. It had an internal error field so partial success was also available on a failure. This error field has been removed. * Use `openArray[byte]` rather than `Blob` in functions prototypes * Provide synchronised multi instance transactions why: The `CoreDB` object was geared towards the legacy DB which used a single transaction for the key-value backend DB. Different state roots are provided by the backend database, so all instances work directly on the same backend. Aristo db instances have different in-memory mappings (aka different state roots) and the transactions are on top of there mappings. So each instance might run different transactions. Multi instance transactions are a compromise to converge towards the legacy behaviour. The synchronised transactions span over all instances available at the time when base transaction was opened. Instances created later are unaffected. * Provide key-value pair database iterator why: Needed in `CoreDB` for `replicate()` emulation also: Some update of internal code * Extend API (i.e. prototype variants) why: Needed for `CoreDB` geared towards the legacy backend which has a more basic API than Aristo.
2023-09-15 15:23:53 +00:00
TxGarbledSpan
Aristo db transaction based interface (#1628) * Provide transaction based interface for standard operations * Provide unit tests for new Aristo interface using transactions details: These new tests combine and replace several single-purpose tests. The now unused test sources will be kept for a while to be eventually removed.
2023-07-05 13:50:11 +00:00
Aristo db state root repos and reorg (#1744) * Reorg of distributed backend access details: Now handled via API provided in `aristo_desc`. * Rename `checkCache()` => `checkTop()` why: Better naming for top layer cache checker also: Provide cascaded fifos checker * Provide `eq` directive for finding filter by exact filter ID (think block number) * Some code beautification (for better code reading) * State root reposition and reorg details: Repositioning is supported by forking a new descriptor. Reorg is then accomplished by writing this forked state on the backend database.
2023-09-11 20:38:49 +00:00
# Functions from `aristo_desc`
MustBeOnCentre
NotAllowedOnCentre
Aristo db traversal helpers (#1638) * Misc fixes detail: * Fix de-serialisation for account leafs * Update node recovery from unit tests * Remove `LegacyAccount` from `PayloadRef` object why: Legacy accounts use a hash key as storage root which is detrimental to the working of the Aristo database which uses a vertex ID. * Dissolve `hashify_helper` into `aristo_utils` and `aristo_transcode` why: Functions are of general interest so they should live in first level code files. * Added left/right iterators over leaf nodes * Some helper/wrapper functions that might be useful
2023-07-12 23:03:14 +00:00
# Miscelaneous handy helpers
PayloadTypeUnsupported
Core db update providing additional results code interface (#1776) * Split `core_db/base.nim` into several sources * Rename `core_db/legacy.nim` => `core_db/legacy_db.nim` * Update `CoreDb` API, dual methods returning `Result[]` or plain value detail: Plain value methods implemet the legacy API, they defect on error results * Redesign `CoreDB` direct backend access why: Made the `backend` directive integral part of the API * Discontinue providing unused or otherwise available functions details: + setTransactionID() removed, not used and not easily replicable in Aristo + maybeGet() removed, available via direct backend access + newPhk() removed, never used & was experimental anyway * Update/reorg backend API why: + Added error print function `$$()` + General descriptor completion (and optional validation) via `bless()` * Update `Aristo`/`Kvt` exception handling why: Avoid `CatchableError` exceptions, rather pass them as error code where appropriate. * More `CoreDB` compliant `Aristo` and `Kvt` methods details: + Providing functions like `contains()`, `getVtxRc()` (returns `Result[]`). + Additional error code: `NotImplemented` * Rewrite/reorg of Aristo DB constructor why: Previously used global object `DefaultQidLayoutRef` as default initialiser. This object was created at compile time which lead to non-gc safe functions. * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/aristo/aristo_transcode.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> --------- Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-09-26 09:21:13 +00:00
LeafKeyInvalid
Optional accounts cache module for creating genesis (#1897) * Split off `ReadOnlyStateDB` from `AccountStateDB` from `state_db.nim` why: Apart from testing, applications use `ReadOnlyStateDB` as an easy way to access the accounts ledger. This is well supported by the `Aristo` db, but writable mode is only parially supported. The writable AccountStateDB` object for modifying accounts is not used by production code. So, for lecgacy and testing apps, the full support of the previous `AccountStateDB` is now enabled by `import db/state_db/read_write` and the `import db/state_db` provides read-only mode. * Encapsulate `AccountStateDB` as `GenesisLedgerRef` or genesis creation why: `AccountStateDB` has poor support for `Aristo` and is not widely used in favour of `AccountsLedger` (which will be abstracted as `ledger`.) Currently, using other than the `AccountStateDB` ledgers within the `GenesisLedgerRef` wrapper is experimental and test only. Eventually, the wrapper should disappear so that the `Ledger` object (which encapsulates `AccountsCache` and `AccountsLedger`) will prevail. * For the `Ledger`, provide access to raw accounts `MPT` why: This gives to the `CoreDbMptRef` descriptor from the `CoreDb` (which is the legacy version of CoreDxMptRef`.) For the new `ledger` API, the accounts are based on the `CoreDxMAccRef` descriptor which uses a particular sub-system for accounts while legacy applications use the `CoreDbPhkRef` equivalent of the `SecureHexaryTrie`. The only place where this feature will currently be used is the `genesis.nim` source file. * Fix `Aristo` bugs, missing boundary checks, typos, etc. * Verify root vertex in `MPT` and account constructors why: Was missing so far, in particular the accounts constructor must verify `VertexID(1) * Fix include file
2023-11-20 11:51:43 +00:00
AccountRootUnacceptable
AccountRootCannotCreate
Aristo db traversal helpers (#1638) * Misc fixes detail: * Fix de-serialisation for account leafs * Update node recovery from unit tests * Remove `LegacyAccount` from `PayloadRef` object why: Legacy accounts use a hash key as storage root which is detrimental to the working of the Aristo database which uses a vertex ID. * Dissolve `hashify_helper` into `aristo_utils` and `aristo_transcode` why: Functions are of general interest so they should live in first level code files. * Added left/right iterators over leaf nodes * Some helper/wrapper functions that might be useful
2023-07-12 23:03:14 +00:00
AccountRlpDecodingError
AccountStorageKeyMissing
AccountVtxUnsupported
AccountNodeUnsupported
Core db update providing additional results code interface (#1776) * Split `core_db/base.nim` into several sources * Rename `core_db/legacy.nim` => `core_db/legacy_db.nim` * Update `CoreDb` API, dual methods returning `Result[]` or plain value detail: Plain value methods implemet the legacy API, they defect on error results * Redesign `CoreDB` direct backend access why: Made the `backend` directive integral part of the API * Discontinue providing unused or otherwise available functions details: + setTransactionID() removed, not used and not easily replicable in Aristo + maybeGet() removed, available via direct backend access + newPhk() removed, never used & was experimental anyway * Update/reorg backend API why: + Added error print function `$$()` + General descriptor completion (and optional validation) via `bless()` * Update `Aristo`/`Kvt` exception handling why: Avoid `CatchableError` exceptions, rather pass them as error code where appropriate. * More `CoreDB` compliant `Aristo` and `Kvt` methods details: + Providing functions like `contains()`, `getVtxRc()` (returns `Result[]`). + Additional error code: `NotImplemented` * Rewrite/reorg of Aristo DB constructor why: Previously used global object `DefaultQidLayoutRef` as default initialiser. This object was created at compile time which lead to non-gc safe functions. * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/aristo/aristo_transcode.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> --------- Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-09-26 09:21:13 +00:00
NotImplemented
Aristo db traversal helpers (#1638) * Misc fixes detail: * Fix de-serialisation for account leafs * Update node recovery from unit tests * Remove `LegacyAccount` from `PayloadRef` object why: Legacy accounts use a hash key as storage root which is detrimental to the working of the Aristo database which uses a vertex ID. * Dissolve `hashify_helper` into `aristo_utils` and `aristo_transcode` why: Functions are of general interest so they should live in first level code files. * Added left/right iterators over leaf nodes * Some helper/wrapper functions that might be useful
2023-07-12 23:03:14 +00:00
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 11:47:47 +00:00
# End