status-im
/
nimbus-eth1
mirror of https://github.com/status-im/nimbus-eth1.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth1/nimbus/db/aristo/aristo_desc/desc_error.nim

291 lines
6.8 KiB
Nim
Raw Normal View History

Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 14:25:29 +00:00
# nimbus-eth1
Core db update storage root management for sub tries (#1964) * Aristo: Re-phrase `LayerDelta` and `LayerFinal` as object references why: Avoids copying in some cases * Fix copyright header * Aristo: Verify `leafTie.root` function argument for `merge()` proc why: Zero root will lead to inconsistent DB entry * Aristo: Update failure condition for hash labels compiler `hashify()` why: Node need not be rejected as long as links are on the schedule. In that case, `redo[]` is to become `wff.base[]` at a later stage. This amends an earlier fix, part of #1952 by also testing against the target nodes of the `wff.base[]` sets. * Aristo: Add storage root glue record to `hashify()` schedule why: An account leaf node might refer to a non-resolvable storage root ID. Storage root node chains will end up at the storage root. So the link `storage-root->account-leaf` needs an extra item in the schedule. * Aristo: fix error code returned by `fetchPayload()` details: Final error code is implied by the error code form the `hikeUp()` function. * CoreDb: Discard `createOk` argument in API `getRoot()` function why: Not needed for the legacy DB. For the `Arsto` DB, a lazy approach is implemented where a stprage root node is created on-the-fly. * CoreDb: Prevent `$$` logging in some cases why: Logging the function `$$` is not useful when it is used for internal use, i.e. retrieving an an error text for logging. * CoreDb: Add `tryHashFn()` to API for pretty printing why: Pretty printing must not change the hashification status for the `Aristo` DB. So there is an independent API wrapper for getting the node hash which never updated the hashes. * CoreDb: Discard `update` argument in API `hash()` function why: When calling the API function `hash()`, the latest state is always wanted. For a version that uses the current state as-is without checking, the function `tryHash()` was added to the backend. * CoreDb: Update opaque vertex ID objects for the `Aristo` backend why: For `Aristo`, vID objects encapsulate a numeric `VertexID` referencing a vertex (rather than a node hash as used on the legacy backend.) For storage sub-tries, there might be no initial vertex known when the descriptor is created. So opaque vertex ID objects are supported without a valid `VertexID` which will be initalised on-the-fly when the first item is merged. * CoreDb: Add pretty printer for opaque vertex ID objects * Cosmetics, printing profiling data * CoreDb: Fix segfault in `Aristo` backend when creating MPT descriptor why: Missing initialisation error * CoreDb: Allow MPT to inherit shared context on `Aristo` backend why: Creates descriptors with different storage roots for the same shared `Aristo` DB descriptor. * Cosmetics, update diagnostic message items for `Aristo` backend * Fix Copyright year
2024-01-11 19:11:38 +00:00
# Copyright (c) 2023-2024 Status Research & Development GmbH
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 14:25:29 +00:00
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or
# http://www.apache.org/licenses/LICENSE-2.0)
# * MIT license ([LICENSE-MIT](LICENSE-MIT) or
# http://opensource.org/licenses/MIT)
# at your option. This file may not be copied, modified, or distributed
# except according to those terms.
type
AristoError* = enum
NothingSerious = 0
Aristo db merkle hashify functionality added (#1593) * Keep vertex ID generator state with each db-layer why: The vertex ID generator state is part of the difference to the below layer * Move otherwise unused source to test directory * Add Merkle hash generator also: * Verification facility for debugging * Empty Merkle key hashes encoded as `EMPTY_ROOT_HASH`
2023-05-30 21:21:15 +00:00
GenericError
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 14:25:29 +00:00
Aristo db supporting forest and layered tx architecture (#1598) * Exclude some storage tests why: These test running on external dumps slipped through. The particular dumps were reported earlier as somehow dodgy. This was changed in `#1457` but having a second look, the change on hexary_interpolate.nim(350) might be incorrect. * Redesign `Aristo DB` descriptor for transaction based layers why: Previous descriptor layout made it cumbersome to push/pop database delta layers. The new architecture keeps each layer with the full delta set relative to the database backend. * Keep root ID as part of the `Patricia Trie` leaf path why; That way, forests are supported
2023-06-09 11:17:37 +00:00
# Rlp decoder, `read()`
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 14:25:29 +00:00
Rlp2Or17ListEntries
RlpBlobExpected
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
RlpBranchHashKeyExpected
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 14:25:29 +00:00
RlpEmptyBlobExpected
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
RlpExtHashKeyExpected
RlpHashKeyExpected
RlpNonEmptyBlobExpected
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 14:25:29 +00:00
RlpOtherException
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
RlpRlpException
# Serialise decoder
SerCantResolveStorageRoot
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 14:25:29 +00:00
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
# Data record transcoders, `deblobify()` and `blobify()`
Aristo db fixes n updates needed for filter fifo (#1728) * Set scheduler state as part of the backend descriptor details: Moved type definitions `QidLayoutRef` and `QidSchedRef` to `desc_structural.nim` so that it shares the same folder as `desc_backend.nim` * Automatic filter queue table initialisation in backend details: Scheduler can be tweaked or completely disabled * Updated backend unit tests details: + some code clean up/beautification, reads better now + disabled persistent filters so that there is no automated filter management which will be implemented next * Prettify/update unit tests source code details: Mostly replacing the `check()` paradigm by `xCheck()` * Somewhat simplified backend type management why: Backend objects are labelled with a `BackendType` symbol where the `BackendVoid` label is implicitly assumed for a `nil` backend object reference. To make it easier, a `kind()` function is used now applicable to `nil` references as well. * Fix DB storage layout for filter objects why: Need to store the filter ID with the object * Implement reverse [] index on fifo why: An integer index argument on `[]` retrieves the QueueID (label) of the fifo item while a QueueID argument on `[]` retrieves the index (so it is inverse to the former variant). * Provide iterator over filters as fifo why: This iterator goes along the cascased fifo structure (i.e. in historical order)
2023-09-05 13:57:20 +00:00
BlobifyNilFilter
BlobifyNilVertex
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
BlobifyBranchMissingRefs
BlobifyExtMissingRefs
BlobifyExtPathOverflow
BlobifyLeafPathOverflow
Aristo db implement filter serialisation for storage (#1695) * Remove concept of empty/blind filters why: Not needed. A non-existent filter is is coded as a nil reference. * Slightly generalised backend iterators why: * VertexID as key for the ID generator state makes no sense * there will be more tables addressed by non-VertexID keys * Store serialised/blobified vertices on memory backend why: This is more in line with the RocksDB backend so more appropriate for testing when comparing behaviour. For a speedy memory database, a backend-less variant should be used. * Drop the `Aristo` prefix from names `AristoLayerRef`, etc. * Suppress compiler warning why: duplicate imports * Add filter serialisation transcoder why: Will be used as storage format
2023-08-18 19:46:55 +00:00
BlobifyFilterRecordOverflow
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
DeblobNilArgument
DeblobUnknown
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
DeblobVtxTooShort
DeblobHashKeyExpected
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
DeblobBranchTooShort
DeblobBranchSizeGarbled
DeblobBranchInxOutOfRange
DeblobExtTooShort
DeblobExtSizeGarbled
DeblobExtGotLeafPrefix
DeblobLeafSizeGarbled
DeblobLeafGotExtPrefix
DeblobSizeGarbled
DeblobWrongType
Run coredb without journal (#2266) * Add persistent last state stamp feature why: This allows to run `CoreDb` without journal * Start `CoreDb` without journal * Remove journal related functions from `CoredDb`
2024-05-31 17:32:22 +00:00
DeblobWrongSize
Extended data Payload specs for the backend. (#1630) why: For the main tree with root vertex ID 1, the leaf nodes hold the account data. These accounts may link to sub trees the storage root node ID of which must be registered here. There is no reverse key lookup on the backend. note: These definitions are experimental. Also, there are some tests missing for validating Payload data conversions.
2023-07-05 20:27:48 +00:00
DeblobPayloadTooShortInt64
DeblobPayloadTooShortInt256
DeblobNonceLenUnsupported
DeblobBalanceLenUnsupported
DeblobStorageLenUnsupported
DeblobCodeLenUnsupported
Aristo db implement filter serialisation for storage (#1695) * Remove concept of empty/blind filters why: Not needed. A non-existent filter is is coded as a nil reference. * Slightly generalised backend iterators why: * VertexID as key for the ID generator state makes no sense * there will be more tables addressed by non-VertexID keys * Store serialised/blobified vertices on memory backend why: This is more in line with the RocksDB backend so more appropriate for testing when comparing behaviour. For a speedy memory database, a backend-less variant should be used. * Drop the `Aristo` prefix from names `AristoLayerRef`, etc. * Suppress compiler warning why: duplicate imports * Add filter serialisation transcoder why: Will be used as storage format
2023-08-18 19:46:55 +00:00
DeblobFilterTooShort
DeblobFilterGenTooShort
DeblobFilterTrpTooShort
DeblobFilterTrpVtxSizeGarbled
DeblobFilterSizeGarbled
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 14:25:29 +00:00
Aristo db supporting forest and layered tx architecture (#1598) * Exclude some storage tests why: These test running on external dumps slipped through. The particular dumps were reported earlier as somehow dodgy. This was changed in `#1457` but having a second look, the change on hexary_interpolate.nim(350) might be incorrect. * Redesign `Aristo DB` descriptor for transaction based layers why: Previous descriptor layout made it cumbersome to push/pop database delta layers. The new architecture keeps each layer with the full delta set relative to the database backend. * Keep root ID as part of the `Patricia Trie` leaf path why; That way, forests are supported
2023-06-09 11:17:37 +00:00
# Converter `asNode()`, currenly for unit tests only
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 14:25:29 +00:00
CacheMissingNodekeys
Aristo db supporting forest and layered tx architecture (#1598) * Exclude some storage tests why: These test running on external dumps slipped through. The particular dumps were reported earlier as somehow dodgy. This was changed in `#1457` but having a second look, the change on hexary_interpolate.nim(350) might be incorrect. * Redesign `Aristo DB` descriptor for transaction based layers why: Previous descriptor layout made it cumbersome to push/pop database delta layers. The new architecture keeps each layer with the full delta set relative to the database backend. * Keep root ID as part of the `Patricia Trie` leaf path why; That way, forests are supported
2023-06-09 11:17:37 +00:00
# Path function `hikeUp()`
Aristo: Update unit test suite (#2002) * Aristo: Update unit test suite * Aristo/Kvt: Fix iterators why: Generic iterators were not properly updated after backend change * Aristo: Add sub-trie deletion functionality why: For storage tries linked to an account payload vertex ID, a the whole storage trie needs to be deleted with the account. * Aristo: Reserve vertex ID numbers for static custom state roots why: Static custom state roots may be controlled by an application, e.g. for a receipt or a transaction root. The `Aristo` functions are agnostic of what the static state roots are when different from the internal tree vertex ID 1. details; The `merge()` function applied to a non-static state root (assumed to be a storage root) will check the payload of an accounts leaf and mark its Merkle keys to be re-checked. * Aristo: Correct error code symbol * Aristo: Update error code symbols * Aristo: Code cosmetics/comments * Aristo: Fix hashify schedule calculator why: Had a tendency to stop early leaving an incomplete job
2024-02-01 21:27:48 +00:00
HikeBranchMissingEdge
Aristo db update foreground caching (#1605) * Fix vertex ID generator state handling for rocksdb backend why: * Key error in walk iterator * Needs to be loaded when opening the database * Use non-zero sub-table prefixes for rocksdb why: Handy for debugging * Fix error code for missing key on rocksdb backend why: Previously returned `VOID_HASH_KEY` rather than `GetKeyNotFound` * Explicitly copy vertex data between internal table and function/result argument why: Function argument or return reference may still refer to the same data object. * Updated error symbols why: Error symbol names for the hike module now start with the prefix `Hike`. * Write back modified branch node into local top layer cache why: With the backend available, the source of the branch node references might not be the top layer cache. So any change must be explicitely recorded.
2023-06-22 11:13:24 +00:00
HikeBranchTailEmpty
Core db+aristo fixes and tx handling updates (#2164) * Aristo: Rename journal related sources and functions why: Previously, the naming was hinged on the phrases `fifo`, `filter` etc. which reflect the inner workings of cascaded filters. This was unfortunate for reading/understanding the source code for actions where the focus is the journal as a whole. * Aristo: Fix buffer overflow (path length truncating error) * Aristo: Tighten `hikeUp()` stop check, update error code why: Detect dangling vertex links. These are legit with `snap` sync processing but not with regular processing. * Aristo: Raise assert in regular mode `merge()` at a dangling link/edge why: With `snap` sync processing, partial trees are ok and can be amended. Not so in regular mode. Previously there was only a debug message when a non-legit dangling edge was encountered. * Aristo: Make sure that vertices are copied before modification why: Otherwise vertices from lower layers might also be modified * Aristo: Fix relaxed mode for validity checker `check()` * Remove cruft * Aristo: Update API for transaction handling details: + Split `aristo_tx.nim` into sub-modules + Split `forkWith()` into `findTx()` + `forkTx()` + Removed `forkTop()`, `forkBase()` (now superseded by new `forkTx()`) * CoreDb+Aristo: Fix initialiser (missing methods)
2024-05-03 17:38:17 +00:00
HikeDanglingEdge
Aristo: Update unit test suite (#2002) * Aristo: Update unit test suite * Aristo/Kvt: Fix iterators why: Generic iterators were not properly updated after backend change * Aristo: Add sub-trie deletion functionality why: For storage tries linked to an account payload vertex ID, a the whole storage trie needs to be deleted with the account. * Aristo: Reserve vertex ID numbers for static custom state roots why: Static custom state roots may be controlled by an application, e.g. for a receipt or a transaction root. The `Aristo` functions are agnostic of what the static state roots are when different from the internal tree vertex ID 1. details; The `merge()` function applied to a non-static state root (assumed to be a storage root) will check the payload of an accounts leaf and mark its Merkle keys to be re-checked. * Aristo: Correct error code symbol * Aristo: Update error code symbols * Aristo: Code cosmetics/comments * Aristo: Fix hashify schedule calculator why: Had a tendency to stop early leaving an incomplete job
2024-02-01 21:27:48 +00:00
HikeEmptyPath
Core db+aristo fixes and tx handling updates (#2164) * Aristo: Rename journal related sources and functions why: Previously, the naming was hinged on the phrases `fifo`, `filter` etc. which reflect the inner workings of cascaded filters. This was unfortunate for reading/understanding the source code for actions where the focus is the journal as a whole. * Aristo: Fix buffer overflow (path length truncating error) * Aristo: Tighten `hikeUp()` stop check, update error code why: Detect dangling vertex links. These are legit with `snap` sync processing but not with regular processing. * Aristo: Raise assert in regular mode `merge()` at a dangling link/edge why: With `snap` sync processing, partial trees are ok and can be amended. Not so in regular mode. Previously there was only a debug message when a non-legit dangling edge was encountered. * Aristo: Make sure that vertices are copied before modification why: Otherwise vertices from lower layers might also be modified * Aristo: Fix relaxed mode for validity checker `check()` * Remove cruft * Aristo: Update API for transaction handling details: + Split `aristo_tx.nim` into sub-modules + Split `forkWith()` into `findTx()` + `forkTx()` + Removed `forkTop()`, `forkBase()` (now superseded by new `forkTx()`) * CoreDb+Aristo: Fix initialiser (missing methods)
2024-05-03 17:38:17 +00:00
HikeExtMissingEdge
Aristo db update foreground caching (#1605) * Fix vertex ID generator state handling for rocksdb backend why: * Key error in walk iterator * Needs to be loaded when opening the database * Use non-zero sub-table prefixes for rocksdb why: Handy for debugging * Fix error code for missing key on rocksdb backend why: Previously returned `VOID_HASH_KEY` rather than `GetKeyNotFound` * Explicitly copy vertex data between internal table and function/result argument why: Function argument or return reference may still refer to the same data object. * Updated error symbols why: Error symbol names for the hike module now start with the prefix `Hike`. * Write back modified branch node into local top layer cache why: With the backend available, the source of the branch node references might not be the top layer cache. So any change must be explicitely recorded.
2023-06-22 11:13:24 +00:00
HikeExtTailEmpty
HikeExtTailMismatch
Core db update storage root management for sub tries (#1964) * Aristo: Re-phrase `LayerDelta` and `LayerFinal` as object references why: Avoids copying in some cases * Fix copyright header * Aristo: Verify `leafTie.root` function argument for `merge()` proc why: Zero root will lead to inconsistent DB entry * Aristo: Update failure condition for hash labels compiler `hashify()` why: Node need not be rejected as long as links are on the schedule. In that case, `redo[]` is to become `wff.base[]` at a later stage. This amends an earlier fix, part of #1952 by also testing against the target nodes of the `wff.base[]` sets. * Aristo: Add storage root glue record to `hashify()` schedule why: An account leaf node might refer to a non-resolvable storage root ID. Storage root node chains will end up at the storage root. So the link `storage-root->account-leaf` needs an extra item in the schedule. * Aristo: fix error code returned by `fetchPayload()` details: Final error code is implied by the error code form the `hikeUp()` function. * CoreDb: Discard `createOk` argument in API `getRoot()` function why: Not needed for the legacy DB. For the `Arsto` DB, a lazy approach is implemented where a stprage root node is created on-the-fly. * CoreDb: Prevent `$$` logging in some cases why: Logging the function `$$` is not useful when it is used for internal use, i.e. retrieving an an error text for logging. * CoreDb: Add `tryHashFn()` to API for pretty printing why: Pretty printing must not change the hashification status for the `Aristo` DB. So there is an independent API wrapper for getting the node hash which never updated the hashes. * CoreDb: Discard `update` argument in API `hash()` function why: When calling the API function `hash()`, the latest state is always wanted. For a version that uses the current state as-is without checking, the function `tryHash()` was added to the backend. * CoreDb: Update opaque vertex ID objects for the `Aristo` backend why: For `Aristo`, vID objects encapsulate a numeric `VertexID` referencing a vertex (rather than a node hash as used on the legacy backend.) For storage sub-tries, there might be no initial vertex known when the descriptor is created. So opaque vertex ID objects are supported without a valid `VertexID` which will be initalised on-the-fly when the first item is merged. * CoreDb: Add pretty printer for opaque vertex ID objects * Cosmetics, printing profiling data * CoreDb: Fix segfault in `Aristo` backend when creating MPT descriptor why: Missing initialisation error * CoreDb: Allow MPT to inherit shared context on `Aristo` backend why: Creates descriptors with different storage roots for the same shared `Aristo` DB descriptor. * Cosmetics, update diagnostic message items for `Aristo` backend * Fix Copyright year
2024-01-11 19:11:38 +00:00
HikeLeafUnexpected
Aristo: Update unit test suite (#2002) * Aristo: Update unit test suite * Aristo/Kvt: Fix iterators why: Generic iterators were not properly updated after backend change * Aristo: Add sub-trie deletion functionality why: For storage tries linked to an account payload vertex ID, a the whole storage trie needs to be deleted with the account. * Aristo: Reserve vertex ID numbers for static custom state roots why: Static custom state roots may be controlled by an application, e.g. for a receipt or a transaction root. The `Aristo` functions are agnostic of what the static state roots are when different from the internal tree vertex ID 1. details; The `merge()` function applied to a non-static state root (assumed to be a storage root) will check the payload of an accounts leaf and mark its Merkle keys to be re-checked. * Aristo: Correct error code symbol * Aristo: Update error code symbols * Aristo: Code cosmetics/comments * Aristo: Fix hashify schedule calculator why: Had a tendency to stop early leaving an incomplete job
2024-02-01 21:27:48 +00:00
HikeNoLegs
HikeRootMissing
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 11:47:47 +00:00
# Path/nibble/key conversions in `aisto_path.nim`
PathExpected64Nibbles
Aristo db allow shorter than 64 nibbles path keys (#1864) * Aristo: Single `FetchPathNotFound` error in `fetchXxx()` and `hasPath()` why: Missing path hike returns too many detailed reasons why it failed which becomes cumbersome to handle. also: Renamed `contains()` => `hasPath()` which disables the `in` operator on non-boolean `contains()` functions * Kvt: Renamed `contains()` => `hasKey()` why: which disables the `in` operator on non-boolean `contains()` functions * Aristo: Generalising `HashID` by variable length `PathID` why: There are cases when the `Aristo` database is to be used with shorter than 64 nibbles keys when handling transactions indexes with sequence IDs. caveat: This patch only works reliable for full length `PathID` values. Tests for shorter `PathID` values are currently missing.
2023-10-27 21:36:51 +00:00
PathAtMost64Nibbles
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 11:47:47 +00:00
PathExpectedLeaf
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 14:25:29 +00:00
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 11:47:47 +00:00
# Merge leaf `merge()`
Aristo avoid storage trie update race conditions (#2251) * Update TDD suite logger output format choices why: New format is not practical for TDD as it just dumps data across a wide range (considerably larder than 80 columns.) So the new format can be turned on by function argument. * Update unit tests samples configuration why: Slightly changed the way to find the `era1` directory * Remove compiler warnings (fix deprecated expressions and phrases) * Update `Aristo` debugging tools * Always update the `storageID` field of account leaf vertices why: Storage tries are weekly linked to an account leaf object in that the `storageID` field is updated by the application. Previously, `Aristo` verified that leaf objects make sense when passed to the database. As a consequence * the database was inconsistent for a short while * the burden for correctness was all on the application which led to delayed error handling which is hard to debug. So `Aristo` will internally update the account leaf objects so that there are no race conditions due to the storage trie handling * Aristo: Let `stow()`/`persist()` bail out unless there is a `VertexID(1)` why: The journal and filter logic depends on the hash of the `VertexID(1)` which is commonly known as the state root. This implies that all changes to the database are somehow related to that. * Make sure that a `Ledger` account does not overwrite the storage trie reference why: Due to the abstraction of a sub-trie (now referred to as column with a hash describing its state) there was a weakness in the `Aristo` handler where an account leaf could be overwritten though changing the validity of the database. This has been changed and the database will now reject such changes. This patch fixes the behaviour on the application layer. In particular, the column handle returned by the `CoreDb` needs to be updated by the `Aristo` database state. This mitigates the problem that a storage trie might have vanished or re-apperaed with a different vertex ID. * Fix sub-trie deletion test why: Was originally hinged on `VertexID(1)` which cannot be wholesale deleted anymore after the last Aristo update. Also, running with `VertexID(2)` needs an artificial `VertexID(1)` for making `stow()` or `persist()` work. * Cosmetics * Activate `test_generalstate_json` * Temporarily `deactivate test_tracer_json` * Fix copyright header --------- Co-authored-by: jordan <jordan@dry.pudding> Co-authored-by: Jacek Sieka <jacek@status.im>
2024-05-30 17:48:38 +00:00
MergeAssemblyFailed # Ooops, internal error
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 11:47:47 +00:00
MergeBranchGarbledNibble
MergeBranchGarbledTail
Aristo avoid storage trie update race conditions (#2251) * Update TDD suite logger output format choices why: New format is not practical for TDD as it just dumps data across a wide range (considerably larder than 80 columns.) So the new format can be turned on by function argument. * Update unit tests samples configuration why: Slightly changed the way to find the `era1` directory * Remove compiler warnings (fix deprecated expressions and phrases) * Update `Aristo` debugging tools * Always update the `storageID` field of account leaf vertices why: Storage tries are weekly linked to an account leaf object in that the `storageID` field is updated by the application. Previously, `Aristo` verified that leaf objects make sense when passed to the database. As a consequence * the database was inconsistent for a short while * the burden for correctness was all on the application which led to delayed error handling which is hard to debug. So `Aristo` will internally update the account leaf objects so that there are no race conditions due to the storage trie handling * Aristo: Let `stow()`/`persist()` bail out unless there is a `VertexID(1)` why: The journal and filter logic depends on the hash of the `VertexID(1)` which is commonly known as the state root. This implies that all changes to the database are somehow related to that. * Make sure that a `Ledger` account does not overwrite the storage trie reference why: Due to the abstraction of a sub-trie (now referred to as column with a hash describing its state) there was a weakness in the `Aristo` handler where an account leaf could be overwritten though changing the validity of the database. This has been changed and the database will now reject such changes. This patch fixes the behaviour on the application layer. In particular, the column handle returned by the `CoreDb` needs to be updated by the `Aristo` database state. This mitigates the problem that a storage trie might have vanished or re-apperaed with a different vertex ID. * Fix sub-trie deletion test why: Was originally hinged on `VertexID(1)` which cannot be wholesale deleted anymore after the last Aristo update. Also, running with `VertexID(2)` needs an artificial `VertexID(1)` for making `stow()` or `persist()` work. * Cosmetics * Activate `test_generalstate_json` * Temporarily `deactivate test_tracer_json` * Fix copyright header --------- Co-authored-by: jordan <jordan@dry.pudding> Co-authored-by: Jacek Sieka <jacek@status.im>
2024-05-30 17:48:38 +00:00
MergeBranchLinkLeafGarbled
Aristo db merkle hashify functionality added (#1593) * Keep vertex ID generator state with each db-layer why: The vertex ID generator state is part of the difference to the below layer * Move otherwise unused source to test directory * Add Merkle hash generator also: * Verification facility for debugging * Empty Merkle key hashes encoded as `EMPTY_ROOT_HASH`
2023-05-30 21:21:15 +00:00
MergeBranchLinkLockedKey
MergeBranchLinkProofModeLock
Aristo avoid storage trie update race conditions (#2251) * Update TDD suite logger output format choices why: New format is not practical for TDD as it just dumps data across a wide range (considerably larder than 80 columns.) So the new format can be turned on by function argument. * Update unit tests samples configuration why: Slightly changed the way to find the `era1` directory * Remove compiler warnings (fix deprecated expressions and phrases) * Update `Aristo` debugging tools * Always update the `storageID` field of account leaf vertices why: Storage tries are weekly linked to an account leaf object in that the `storageID` field is updated by the application. Previously, `Aristo` verified that leaf objects make sense when passed to the database. As a consequence * the database was inconsistent for a short while * the burden for correctness was all on the application which led to delayed error handling which is hard to debug. So `Aristo` will internally update the account leaf objects so that there are no race conditions due to the storage trie handling * Aristo: Let `stow()`/`persist()` bail out unless there is a `VertexID(1)` why: The journal and filter logic depends on the hash of the `VertexID(1)` which is commonly known as the state root. This implies that all changes to the database are somehow related to that. * Make sure that a `Ledger` account does not overwrite the storage trie reference why: Due to the abstraction of a sub-trie (now referred to as column with a hash describing its state) there was a weakness in the `Aristo` handler where an account leaf could be overwritten though changing the validity of the database. This has been changed and the database will now reject such changes. This patch fixes the behaviour on the application layer. In particular, the column handle returned by the `CoreDb` needs to be updated by the `Aristo` database state. This mitigates the problem that a storage trie might have vanished or re-apperaed with a different vertex ID. * Fix sub-trie deletion test why: Was originally hinged on `VertexID(1)` which cannot be wholesale deleted anymore after the last Aristo update. Also, running with `VertexID(2)` needs an artificial `VertexID(1)` for making `stow()` or `persist()` work. * Cosmetics * Activate `test_generalstate_json` * Temporarily `deactivate test_tracer_json` * Fix copyright header --------- Co-authored-by: jordan <jordan@dry.pudding> Co-authored-by: Jacek Sieka <jacek@status.im>
2024-05-30 17:48:38 +00:00
MergeBranchLinkVtxPfxTooShort
Aristo db merkle hashify functionality added (#1593) * Keep vertex ID generator state with each db-layer why: The vertex ID generator state is part of the difference to the below layer * Move otherwise unused source to test directory * Add Merkle hash generator also: * Verification facility for debugging * Empty Merkle key hashes encoded as `EMPTY_ROOT_HASH`
2023-05-30 21:21:15 +00:00
MergeBranchProofModeLock
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 11:47:47 +00:00
MergeBranchRootExpected
Aristo avoid storage trie update race conditions (#2251) * Update TDD suite logger output format choices why: New format is not practical for TDD as it just dumps data across a wide range (considerably larder than 80 columns.) So the new format can be turned on by function argument. * Update unit tests samples configuration why: Slightly changed the way to find the `era1` directory * Remove compiler warnings (fix deprecated expressions and phrases) * Update `Aristo` debugging tools * Always update the `storageID` field of account leaf vertices why: Storage tries are weekly linked to an account leaf object in that the `storageID` field is updated by the application. Previously, `Aristo` verified that leaf objects make sense when passed to the database. As a consequence * the database was inconsistent for a short while * the burden for correctness was all on the application which led to delayed error handling which is hard to debug. So `Aristo` will internally update the account leaf objects so that there are no race conditions due to the storage trie handling * Aristo: Let `stow()`/`persist()` bail out unless there is a `VertexID(1)` why: The journal and filter logic depends on the hash of the `VertexID(1)` which is commonly known as the state root. This implies that all changes to the database are somehow related to that. * Make sure that a `Ledger` account does not overwrite the storage trie reference why: Due to the abstraction of a sub-trie (now referred to as column with a hash describing its state) there was a weakness in the `Aristo` handler where an account leaf could be overwritten though changing the validity of the database. This has been changed and the database will now reject such changes. This patch fixes the behaviour on the application layer. In particular, the column handle returned by the `CoreDb` needs to be updated by the `Aristo` database state. This mitigates the problem that a storage trie might have vanished or re-apperaed with a different vertex ID. * Fix sub-trie deletion test why: Was originally hinged on `VertexID(1)` which cannot be wholesale deleted anymore after the last Aristo update. Also, running with `VertexID(2)` needs an artificial `VertexID(1)` for making `stow()` or `persist()` work. * Cosmetics * Activate `test_generalstate_json` * Temporarily `deactivate test_tracer_json` * Fix copyright header --------- Co-authored-by: jordan <jordan@dry.pudding> Co-authored-by: Jacek Sieka <jacek@status.im>
2024-05-30 17:48:38 +00:00
MergeLeafCantChangePayloadType
MergeLeafCantChangeStorageID
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 11:47:47 +00:00
MergeLeafGarbledHike
Aristo db merkle hashify functionality added (#1593) * Keep vertex ID generator state with each db-layer why: The vertex ID generator state is part of the difference to the below layer * Move otherwise unused source to test directory * Add Merkle hash generator also: * Verification facility for debugging * Empty Merkle key hashes encoded as `EMPTY_ROOT_HASH`
2023-05-30 21:21:15 +00:00
MergeLeafPathCachedAlready
Aristo db update merkle hasher (#1925) * Register paths for added leafs because of trie re-balancing why: While the payload would not change, the prefix in the leaf vertex would. So it needs to be flagged for hash recompilation for the `hashify()` module. also: Make sure that `Hike` paths which might have vertex links into the backend filter are replaced by vertex copies before manipulating. Otherwise the vertices on the immutable filter might be involuntarily changed. * Also check for paths where the leaf vertex is on the backend, already why: A a path can have dome vertices on the top layer cache with the `Leaf` vertex on the backend. * Re-define a void `HashLabel` type. why: A `HashLabel` type is a pair `(root-vertex-ID, Keccak-hash)`. Previously, a valid `HashLabel` consisted of a non-empty hash and a non-zero vertex ID. This definition leads to a non-unique representation of a void `HashLabel` with either root-ID or has void. This has been changed to the unique void `HashLabel` exactly if the hash entry is void. * Update consistency checkers * Re-org `hashify()` procedure why: Syncing against block chain showed serious deficiencies which produced wrong hashes or simply bailed out with error. So all fringe cases (mainly due to deleted entries) could be integrated into the labelling schedule rather than handling separate fringe cases.
2023-12-04 20:39:26 +00:00
MergeLeafPathOnBackendAlready
Core db+aristo re org tracer (#2123) * Kvt: Update API hooks * Aristo: Generalised merging snap proofs, now for multiple state roots why: This accommodates pre-loading partial tries for unit tests * Aristo: Update some unit tests * CoreDb+Aristo: Re-factor tracer why: Was bonkers anyway. The main change is that the trace journal is now kept in a way similar to a transaction layer so that it can predictably interact with DB transactions. * Ledger: Debugging helper * Update tracer unit test applicable for `Aristo` * Fix copyright year * Disable `dump()` function as compile time default why: This needs to pull in the `rocks_db` library at compile time.
2024-04-03 15:48:35 +00:00
MergeLeafProofModeLock
Aristo db merkle hashify functionality added (#1593) * Keep vertex ID generator state with each db-layer why: The vertex ID generator state is part of the difference to the below layer * Move otherwise unused source to test directory * Add Merkle hash generator also: * Verification facility for debugging * Empty Merkle key hashes encoded as `EMPTY_ROOT_HASH`
2023-05-30 21:21:15 +00:00
MergeNonBranchProofModeLock
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 11:47:47 +00:00
MergeRootBranchLinkBusy
Core db update storage root management for sub tries (#1964) * Aristo: Re-phrase `LayerDelta` and `LayerFinal` as object references why: Avoids copying in some cases * Fix copyright header * Aristo: Verify `leafTie.root` function argument for `merge()` proc why: Zero root will lead to inconsistent DB entry * Aristo: Update failure condition for hash labels compiler `hashify()` why: Node need not be rejected as long as links are on the schedule. In that case, `redo[]` is to become `wff.base[]` at a later stage. This amends an earlier fix, part of #1952 by also testing against the target nodes of the `wff.base[]` sets. * Aristo: Add storage root glue record to `hashify()` schedule why: An account leaf node might refer to a non-resolvable storage root ID. Storage root node chains will end up at the storage root. So the link `storage-root->account-leaf` needs an extra item in the schedule. * Aristo: fix error code returned by `fetchPayload()` details: Final error code is implied by the error code form the `hikeUp()` function. * CoreDb: Discard `createOk` argument in API `getRoot()` function why: Not needed for the legacy DB. For the `Arsto` DB, a lazy approach is implemented where a stprage root node is created on-the-fly. * CoreDb: Prevent `$$` logging in some cases why: Logging the function `$$` is not useful when it is used for internal use, i.e. retrieving an an error text for logging. * CoreDb: Add `tryHashFn()` to API for pretty printing why: Pretty printing must not change the hashification status for the `Aristo` DB. So there is an independent API wrapper for getting the node hash which never updated the hashes. * CoreDb: Discard `update` argument in API `hash()` function why: When calling the API function `hash()`, the latest state is always wanted. For a version that uses the current state as-is without checking, the function `tryHash()` was added to the backend. * CoreDb: Update opaque vertex ID objects for the `Aristo` backend why: For `Aristo`, vID objects encapsulate a numeric `VertexID` referencing a vertex (rather than a node hash as used on the legacy backend.) For storage sub-tries, there might be no initial vertex known when the descriptor is created. So opaque vertex ID objects are supported without a valid `VertexID` which will be initalised on-the-fly when the first item is merged. * CoreDb: Add pretty printer for opaque vertex ID objects * Cosmetics, printing profiling data * CoreDb: Fix segfault in `Aristo` backend when creating MPT descriptor why: Missing initialisation error * CoreDb: Allow MPT to inherit shared context on `Aristo` backend why: Creates descriptors with different storage roots for the same shared `Aristo` DB descriptor. * Cosmetics, update diagnostic message items for `Aristo` backend * Fix Copyright year
2024-01-11 19:11:38 +00:00
MergeRootMissing
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 11:47:47 +00:00
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
MergeHashKeyInvalid
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
MergeHashKeyDiffersFromCached
MergeHashKeyRevLookUpGarbled
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
MergeRootKeyInvalid
Core db+aristo re org tracer (#2123) * Kvt: Update API hooks * Aristo: Generalised merging snap proofs, now for multiple state roots why: This accommodates pre-loading partial tries for unit tests * Aristo: Update some unit tests * CoreDb+Aristo: Re-factor tracer why: Was bonkers anyway. The main change is that the trace journal is now kept in a way similar to a transaction layer so that it can predictably interact with DB transactions. * Ledger: Debugging helper * Update tracer unit test applicable for `Aristo` * Fix copyright year * Disable `dump()` function as compile time default why: This needs to pull in the `rocks_db` library at compile time.
2024-04-03 15:48:35 +00:00
MergeRootKeyNotInProof
MergeRootKeysMissing
MergeRootKeysOverflow
Aristodb remove obsolete and time consuming admin features (#2048) * Aristo: Reorg `hashify()` using different schedule algorithm why: Directly calculating the search tree top down from the roots turns out to be faster than using the cached structures left over by `merge()` and `delete()`. Time gains is short of 20% * Aristo: Remove `lTab[]` leaf entry object type why: Not used anymore. It was previously needed to build the schedule for `hashify()`. * Aristo: Avoid unnecessary re-org of the vertex ID recycling list why: This list can become quite large so a heuristic is employed whether it makes sense to re-org. Also, re-org check is only done by `delete()` functions. * Aristo: Remove key/reverse lookup table from tx layers why: It is ignored except for handling proof nodes and costs unnecessary run time resources. This feature was originally needed to accommodate the mental transition from the legacy MPT to the `Aristo` trie :). * Fix copyright year
2024-02-22 08:24:58 +00:00
MergeProofInitMissing
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
MergeRevVidMustHaveBeenCached
MergeNodeVtxDiffersFromExisting
Core db+aristo re org tracer (#2123) * Kvt: Update API hooks * Aristo: Generalised merging snap proofs, now for multiple state roots why: This accommodates pre-loading partial tries for unit tests * Aristo: Update some unit tests * CoreDb+Aristo: Re-factor tracer why: Was bonkers anyway. The main change is that the trace journal is now kept in a way similar to a transaction layer so that it can predictably interact with DB transactions. * Ledger: Debugging helper * Update tracer unit test applicable for `Aristo` * Fix copyright year * Disable `dump()` function as compile time default why: This needs to pull in the `rocks_db` library at compile time.
2024-04-03 15:48:35 +00:00
MergeNodeVidMissing
MergeNodeAccountPayloadError
Aristo db supporting forest and layered tx architecture (#1598) * Exclude some storage tests why: These test running on external dumps slipped through. The particular dumps were reported earlier as somehow dodgy. This was changed in `#1457` but having a second look, the change on hexary_interpolate.nim(350) might be incorrect. * Redesign `Aristo DB` descriptor for transaction based layers why: Previous descriptor layout made it cumbersome to push/pop database delta layers. The new architecture keeps each layer with the full delta set relative to the database backend. * Keep root ID as part of the `Patricia Trie` leaf path why; That way, forests are supported
2023-06-09 11:17:37 +00:00
MergeRootKeyDiffersForVid
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
MergeNodeVtxDuplicates
Aristodb remove obsolete and time consuming admin features (#2048) * Aristo: Reorg `hashify()` using different schedule algorithm why: Directly calculating the search tree top down from the roots turns out to be faster than using the cached structures left over by `merge()` and `delete()`. Time gains is short of 20% * Aristo: Remove `lTab[]` leaf entry object type why: Not used anymore. It was previously needed to build the schedule for `hashify()`. * Aristo: Avoid unnecessary re-org of the vertex ID recycling list why: This list can become quite large so a heuristic is employed whether it makes sense to re-org. Also, re-org check is only done by `delete()` functions. * Aristo: Remove key/reverse lookup table from tx layers why: It is ignored except for handling proof nodes and costs unnecessary run time resources. This feature was originally needed to accommodate the mental transition from the legacy MPT to the `Aristo` trie :). * Fix copyright year
2024-02-22 08:24:58 +00:00
MergeRootKeyMissing
MergeRootArgsIncomplete
Aristo db merkle hashify functionality added (#1593) * Keep vertex ID generator state with each db-layer why: The vertex ID generator state is part of the difference to the below layer * Move otherwise unused source to test directory * Add Merkle hash generator also: * Verification facility for debugging * Empty Merkle key hashes encoded as `EMPTY_ROOT_HASH`
2023-05-30 21:21:15 +00:00
Core db and aristo maintenance update (#2014) * Aristo: Update error return code why: Failing of `Aristo` function `delete()` might fail because there is no such data item on the db. This must return a single error code as is done with `fetch()`. * Ledger: Better error handling why: The `expect()` clauses have been replaced by raising asserts indicating the error from the database backend. Also, `delete()` failures are legitimate if the item to delete does not exist. * Aristo: Delete function must always leave a label on DB for `hashify()` why: The `hashify()` uses the labels left bu `merge()` and `delete()` to compile (and optimise) a scheduler for subsequent hashing. Originally, the labels were not used for deleted entries and `delete()` still had some edge case where the deletion label was not properly handled. * Aristo: Update `hashify()` scheduler, remove buggy optimisation why: Was left over from version without virtual state roots which did not know about account payload leaf vertices referring to storage roots. * Aristo: Label storage trie account in `delete()` similar to `merge()` details; The `delete()` function applied to a non-static state root (assumed to be a storage root) will check the payload of an accounts leaf and mark its Merkle keys to be re-checked when runninh `hashify()` * Aristo: Clean up and re-org recycled vertex IDs in `hashify()` why: Re-organising the recycled vertex IDs list intends to reduce the size of the list. This list is organised as a LIFO (or stack.) By reorganising it in a way so that the least vertex ID numbers are on top, the list will be kept smaller as observed on some examples (less than 30%.) * CoreDb: Accept storage trie deletion requests in non-initialised state why: Due to lazy initialisation, the root vertex ID might not yet exist. So the `Aristo` database handlers would reject this call with an error and this condition needs to be handled by the API (which realises the lazy feature.) * Cosmetics & code massage, prettify logging * fix missing import
2024-02-08 16:32:16 +00:00
# Utils
UtilsAccPathMissing
UtilsAccPathWithoutLeaf
UtilsAccUnaccessible
UtilsAccWrongStorageRoot
UtilsStoRootMissing
Aristo db merkle hashify functionality added (#1593) * Keep vertex ID generator state with each db-layer why: The vertex ID generator state is part of the difference to the below layer * Move otherwise unused source to test directory * Add Merkle hash generator also: * Verification facility for debugging * Empty Merkle key hashes encoded as `EMPTY_ROOT_HASH`
2023-05-30 21:21:15 +00:00
# Update `Merkle` hashes `hashify()`
Aristodb remove obsolete and time consuming admin features (#2048) * Aristo: Reorg `hashify()` using different schedule algorithm why: Directly calculating the search tree top down from the roots turns out to be faster than using the cached structures left over by `merge()` and `delete()`. Time gains is short of 20% * Aristo: Remove `lTab[]` leaf entry object type why: Not used anymore. It was previously needed to build the schedule for `hashify()`. * Aristo: Avoid unnecessary re-org of the vertex ID recycling list why: This list can become quite large so a heuristic is employed whether it makes sense to re-org. Also, re-org check is only done by `delete()` functions. * Aristo: Remove key/reverse lookup table from tx layers why: It is ignored except for handling proof nodes and costs unnecessary run time resources. This feature was originally needed to accommodate the mental transition from the legacy MPT to the `Aristo` trie :). * Fix copyright year
2024-02-22 08:24:58 +00:00
HashifyVtxUnresolved
HashifyRootVtxUnresolved
HashifyProofHashMismatch
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
# Cache checker `checkCache()`
Core db aristo hasher profiling and timing improvement (#1938) * Explicitly use shared `Kvt` table on `Ledger` and `Clique` lookup. why: Speeds up lookup time with `Aristo` backend. For writing `Clique` data, the `Companion` model allows to write `Clique` data past the database locked by evm transactions. * Implement `CoreDb` profiling with API tracking why: Chasing time spent per APT procs ... * Implement `Ledger` profiling with API tracking why: Chasing time spent per APT procs ... * Always hashify when commiting or storing why: A dirty cache makes no sense when committing * Make sure that a zero key is created when adding/updating vertices why: This is an error fix mainly for edge cases. A typical error was that the root key got deleted when there were only a few vertices left on the DB. * Need all created and changed vertices zero-keyed on the cache why: A zero key (i.e. empty Merkle hash) indicates that a vertex key needs to be updated. This would not be needed immediately after a merge as there is an actual leaf path on the cache layer. But after subsequent merge and delete operations this information might get blurred. * Re-org hashing algorithm why: Apart from errors, the previous implementation was too slow for two reasons: + some control hashes were calculated for debugging (now all verification is done in `aristo_check` module) + the leaf paths stored on the cache are used to build the labelling (aka hashing) schedule; there paths were accumulated over successive hash sessions although it is clear that all keys were generated, already
2023-12-12 17:47:41 +00:00
CheckStkKeyStrayZeroEntry
CheckStkVtxIncomplete
CheckStkVtxKeyMismatch
CheckStkVtxKeyMissing
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
CheckRlxVidVtxMismatch
CheckRlxVtxIncomplete
CheckRlxVtxKeyMissing
CheckRlxVtxKeyMismatch
Aristo avoid storage trie update race conditions (#2251) * Update TDD suite logger output format choices why: New format is not practical for TDD as it just dumps data across a wide range (considerably larder than 80 columns.) So the new format can be turned on by function argument. * Update unit tests samples configuration why: Slightly changed the way to find the `era1` directory * Remove compiler warnings (fix deprecated expressions and phrases) * Update `Aristo` debugging tools * Always update the `storageID` field of account leaf vertices why: Storage tries are weekly linked to an account leaf object in that the `storageID` field is updated by the application. Previously, `Aristo` verified that leaf objects make sense when passed to the database. As a consequence * the database was inconsistent for a short while * the burden for correctness was all on the application which led to delayed error handling which is hard to debug. So `Aristo` will internally update the account leaf objects so that there are no race conditions due to the storage trie handling * Aristo: Let `stow()`/`persist()` bail out unless there is a `VertexID(1)` why: The journal and filter logic depends on the hash of the `VertexID(1)` which is commonly known as the state root. This implies that all changes to the database are somehow related to that. * Make sure that a `Ledger` account does not overwrite the storage trie reference why: Due to the abstraction of a sub-trie (now referred to as column with a hash describing its state) there was a weakness in the `Aristo` handler where an account leaf could be overwritten though changing the validity of the database. This has been changed and the database will now reject such changes. This patch fixes the behaviour on the application layer. In particular, the column handle returned by the `CoreDb` needs to be updated by the `Aristo` database state. This mitigates the problem that a storage trie might have vanished or re-apperaed with a different vertex ID. * Fix sub-trie deletion test why: Was originally hinged on `VertexID(1)` which cannot be wholesale deleted anymore after the last Aristo update. Also, running with `VertexID(2)` needs an artificial `VertexID(1)` for making `stow()` or `persist()` work. * Cosmetics * Activate `test_generalstate_json` * Temporarily `deactivate test_tracer_json` * Fix copyright header --------- Co-authored-by: jordan <jordan@dry.pudding> Co-authored-by: Jacek Sieka <jacek@status.im>
2024-05-30 17:48:38 +00:00
CheckAnyVidDeadStorageRoot
CheckAnyVidSharedStorageRoot
Aristo db tidy up a bit (#1625) * Slightly tighten some self-check conditions * Redefined the database descriptor object as reference (to the object) why: The upcoming transaction wrapper will work with a database reference rather than the object itself * Append state before `save()` to the Aristo descriptor why: This stae was previously returned by the function. Appending it to a field of the Aristo descriptor seems easier to handle.
2023-07-04 18:24:03 +00:00
CheckAnyVtxEmptyKeyMissing
CheckAnyVtxEmptyKeyExpected
CheckAnyVtxEmptyKeyMismatch
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
CheckAnyVtxBranchLinksMissing
CheckAnyVtxExtPfxMissing
CheckAnyVtxLockWithoutKey
Remove vid recycling feature (#2294)
2024-06-04 15:05:13 +00:00
CheckAnyVTopUnset
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
# Backend structural check `checkBE()`
CheckBeVtxInvalid
CheckBeVtxMissing
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
CheckBeVtxBranchLinksMissing
CheckBeVtxExtPfxMissing
CheckBeKeyInvalid
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
CheckBeKeyMissing
CheckBeKeyCantCompile
CheckBeKeyMismatch
Remove vid recycling feature (#2294)
2024-06-04 15:05:13 +00:00
CheckBeGarbledVTop
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
Aristo db implement distributed backend access (#1688) * Fix hashing algorithm why: Particular case where a sub-tree is on the backend, linked by an Extension vertex to the top level. * Update backend verification to report `dirty` top layer * Implement distributed merge of backend filters * Implement distributed backend access management details: Implemented and tested as described in chapter 5 of the `README.md` file.
2023-08-17 13:42:01 +00:00
CheckBeCacheIsDirty
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
CheckBeCacheKeyMissing
CheckBeCacheKeyNonEmpty
CheckBeCacheVidUnsynced
CheckBeCacheKeyDangling
CheckBeCacheVtxDangling
CheckBeCacheKeyCantCompile
CheckBeCacheKeyMismatch
Remove vid recycling feature (#2294)
2024-06-04 15:05:13 +00:00
CheckBeCacheGarbledVTop
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 11:47:47 +00:00
Aristo db state root repos and reorg (#1744) * Reorg of distributed backend access details: Now handled via API provided in `aristo_desc`. * Rename `checkCache()` => `checkTop()` why: Better naming for top layer cache checker also: Provide cascaded fifos checker * Provide `eq` directive for finding filter by exact filter ID (think block number) * Some code beautification (for better code reading) * State root reposition and reorg details: Repositioning is supported by forking a new descriptor. Reorg is then accomplished by writing this forked state on the backend database.
2023-09-11 20:38:49 +00:00
CheckBeFifoSrcTrgMismatch
CheckBeFifoTrgNotStateRoot
Update aristo journal functionality (#2155) * Aristo: Code cosmetics, e.g. update some CamelCase names * CoreDb+Aristo: Provide oldest known state root implied details: The Aristo journal allows to recover earlier but not all state roots. * Aristo: Fix journal backward index operator, e.g. `[^1]` * Aristo: Fix journal updater why: The `fifosStore()` store function slightly misinterpreted the update instructions when translation is to database `put()` functions. The effect was that the journal was ever growing due to stale entries which were never deleted. * CoreDb+Aristo: Provide utils for purging stale data from the KVT details: See earlier patch, not all state roots are available. This patch provides a mapping from some state root to a block number and allows to remove all KVT data related to a particular block number * Aristo+Kvt: Implement a clean up schedule for expired data in KVT why: For a single state ledger like `Aristo`, there is only a limited backlog of states. So KVT data (i.e. headers etc.) are cleaned up regularly * Fix copyright year
2024-04-26 13:43:52 +00:00
# Jornal check `checkJournal()`
CheckJrnCachedQidOverlap
CheckJrnSavedQidMissing
CheckJrnSavedQidStale
CheckJrnLinkingGap
Aristo db fixes after storage slots dump tests added (#1595) * Fix missing Merkle key removal in `merge()` * Accept optional root hash argument in `hashify()` why: For importing a full database, there will be no proof data except the root key. So this can be used to check and set the root key in the database descriptor. also: Associate vertex ID to `hashify()` error return code * Added Aristo Trie traversal function why: * step along leaf vertices in sorted order * tree/trie consistency checks when debugging * Enabled storage slots test data for Aristo DB
2023-06-02 10:04:29 +00:00
# Neighbour vertex, tree traversal `nearbyRight()` and `nearbyLeft()`
NearbyBeyondRange
NearbyBranchError
NearbyDanglingLink
NearbyEmptyHike
NearbyExtensionError
NearbyFailed
NearbyBranchExpected
NearbyLeafExpected
NearbyNestingTooDeep
NearbyPathTailUnexpected
NearbyUnexpectedVtx
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
NearbyVidInvalid
Aristo db fixes after storage slots dump tests added (#1595) * Fix missing Merkle key removal in `merge()` * Accept optional root hash argument in `hashify()` why: For importing a full database, there will be no proof data except the root key. So this can be used to check and set the root key in the database descriptor. also: Associate vertex ID to `hashify()` error return code * Added Aristo Trie traversal function why: * step along leaf vertices in sorted order * tree/trie consistency checks when debugging * Enabled storage slots test data for Aristo DB
2023-06-02 10:04:29 +00:00
Added delete fuctionality (#1596)
2023-06-02 19:21:46 +00:00
# Deletion of vertices, `delete()`
DelBranchExpexted
DelBranchLocked
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 22:22:33 +00:00
DelBranchWithoutRefs
Core db+aristo update storage trie handling (#2023) * CoreDb: Test module with additional sample selector cmd line options * Aristo: Do not automatically remove a storage trie with the account why: This is an unnecessary side effect. Rather than using an automatism, a a storage root must be deleted manually. * Aristo: Can handle stale storage root vertex IDs as empty IDs. why: This is currently needed for the ledger API supporting both, a legacy and the `Aristo` database backend. This feature can be disabled at compile time by re-setting the `LOOSE_STORAGE_TRIE_COUPLING` flag in the `aristo_constants` module. * CoreDb+Aristo: Flush/delete storage trie when deleting account why: On either backend, a deleted account leave a dangling storage trie on the database. For consistency nn the legacy backend, storage tries must not be deleted as they might be shared by several accounts whereas on `Aristo` they are always unique.
2024-02-12 19:37:00 +00:00
DelDanglingStoTrie
Added delete fuctionality (#1596)
2023-06-02 19:21:46 +00:00
DelExtLocked
Core db+aristo update storage trie handling (#2023) * CoreDb: Test module with additional sample selector cmd line options * Aristo: Do not automatically remove a storage trie with the account why: This is an unnecessary side effect. Rather than using an automatism, a a storage root must be deleted manually. * Aristo: Can handle stale storage root vertex IDs as empty IDs. why: This is currently needed for the ledger API supporting both, a legacy and the `Aristo` database backend. This feature can be disabled at compile time by re-setting the `LOOSE_STORAGE_TRIE_COUPLING` flag in the `aristo_constants` module. * CoreDb+Aristo: Flush/delete storage trie when deleting account why: On either backend, a deleted account leave a dangling storage trie on the database. For consistency nn the legacy backend, storage tries must not be deleted as they might be shared by several accounts whereas on `Aristo` they are always unique.
2024-02-12 19:37:00 +00:00
DelLeafExpexted
DelLeafLocked
DelLeafUnexpected
DelPathNotFound
DelPathTagError
Aristo avoid storage trie update race conditions (#2251) * Update TDD suite logger output format choices why: New format is not practical for TDD as it just dumps data across a wide range (considerably larder than 80 columns.) So the new format can be turned on by function argument. * Update unit tests samples configuration why: Slightly changed the way to find the `era1` directory * Remove compiler warnings (fix deprecated expressions and phrases) * Update `Aristo` debugging tools * Always update the `storageID` field of account leaf vertices why: Storage tries are weekly linked to an account leaf object in that the `storageID` field is updated by the application. Previously, `Aristo` verified that leaf objects make sense when passed to the database. As a consequence * the database was inconsistent for a short while * the burden for correctness was all on the application which led to delayed error handling which is hard to debug. So `Aristo` will internally update the account leaf objects so that there are no race conditions due to the storage trie handling * Aristo: Let `stow()`/`persist()` bail out unless there is a `VertexID(1)` why: The journal and filter logic depends on the hash of the `VertexID(1)` which is commonly known as the state root. This implies that all changes to the database are somehow related to that. * Make sure that a `Ledger` account does not overwrite the storage trie reference why: Due to the abstraction of a sub-trie (now referred to as column with a hash describing its state) there was a weakness in the `Aristo` handler where an account leaf could be overwritten though changing the validity of the database. This has been changed and the database will now reject such changes. This patch fixes the behaviour on the application layer. In particular, the column handle returned by the `CoreDb` needs to be updated by the `Aristo` database state. This mitigates the problem that a storage trie might have vanished or re-apperaed with a different vertex ID. * Fix sub-trie deletion test why: Was originally hinged on `VertexID(1)` which cannot be wholesale deleted anymore after the last Aristo update. Also, running with `VertexID(2)` needs an artificial `VertexID(1)` for making `stow()` or `persist()` work. * Cosmetics * Activate `test_generalstate_json` * Temporarily `deactivate test_tracer_json` * Fix copyright header --------- Co-authored-by: jordan <jordan@dry.pudding> Co-authored-by: Jacek Sieka <jacek@status.im>
2024-05-30 17:48:38 +00:00
DelSubTreeAccRoot
Aristo: Update unit test suite (#2002) * Aristo: Update unit test suite * Aristo/Kvt: Fix iterators why: Generic iterators were not properly updated after backend change * Aristo: Add sub-trie deletion functionality why: For storage tries linked to an account payload vertex ID, a the whole storage trie needs to be deleted with the account. * Aristo: Reserve vertex ID numbers for static custom state roots why: Static custom state roots may be controlled by an application, e.g. for a receipt or a transaction root. The `Aristo` functions are agnostic of what the static state roots are when different from the internal tree vertex ID 1. details; The `merge()` function applied to a non-static state root (assumed to be a storage root) will check the payload of an accounts leaf and mark its Merkle keys to be re-checked. * Aristo: Correct error code symbol * Aristo: Update error code symbols * Aristo: Code cosmetics/comments * Aristo: Fix hashify schedule calculator why: Had a tendency to stop early leaving an incomplete job
2024-02-01 21:27:48 +00:00
DelSubTreeTooBig
DelSubTreeVoidRoot
Core db+aristo update storage trie handling (#2023) * CoreDb: Test module with additional sample selector cmd line options * Aristo: Do not automatically remove a storage trie with the account why: This is an unnecessary side effect. Rather than using an automatism, a a storage root must be deleted manually. * Aristo: Can handle stale storage root vertex IDs as empty IDs. why: This is currently needed for the ledger API supporting both, a legacy and the `Aristo` database backend. This feature can be disabled at compile time by re-setting the `LOOSE_STORAGE_TRIE_COUPLING` flag in the `aristo_constants` module. * CoreDb+Aristo: Flush/delete storage trie when deleting account why: On either backend, a deleted account leave a dangling storage trie on the database. For consistency nn the legacy backend, storage tries must not be deleted as they might be shared by several accounts whereas on `Aristo` they are always unique.
2024-02-12 19:37:00 +00:00
DelVidStaleVtx
Added delete fuctionality (#1596)
2023-06-02 19:21:46 +00:00
Aristo db use filter betw backend and tx cache (#1678) * Provide deep copy for each transaction layer why: Localising changes. Selective deep copy was just overlooked. * Generalise vertex ID generator state reorg function `vidReorg()` why: makes it somewhat easier to handle when saving layers. * Provide dummy back end descriptor `NoneBackendRef` * Optional read-only filter between backend and transaction cache why: Some staging area for accumulating changes to the backend DB. This will eventually be an access layer for emulating a backend with multiple/historic state roots. * Re-factor `persistent()` with filter between backend/tx-cache => `stow()` why: The filter provides an abstraction from the physically stored data on disk. So, there can be several MPT instances using the same disk data with different state roots. Of course, all the MPT instances should not differ too much for practical reasons :). TODO: Filter administration tools need to be provided.
2023-08-10 20:01:28 +00:00
# Functions from `aristo_filter.nim`
Implement backend filter mechanics (#1730) details: * Tested features + Successively store filters with increasing filter ID (think block number) + Cascading through fifos, deeper fifos merge groups of filters + Fetch squash merged N top fifos + Delete N top fifos, push back merged fifo, continue storing + Fifo chain is verified by hashes and filter ID * Not tested yet + Real live scenario (using data dumps) + Real filter data (only shallow filters used so far)
2023-09-05 18:00:40 +00:00
FilBackendMissing
FilBackendRoMode
FilNilFilterRejected
Core db+aristo update recovery journal management (#2156) * Aristo: Allow to define/set `FilterID` for journal filter records why: After some changes, the `FilterID` is isomorphic to the `BlockNumber` scalar (well, the first 2^64 entries of a `BlockNumber`.) The needed change for `FilterID` is that the `FilterID(0)` value is valid part of the `FilterID` scalar. A non-valid `FilterID` entry is represented by `none(FilterID)`. * Aristo: Split off function `persist()` as persistent version of `stow()` why: In production, `stow(persistent=false,..)` is currently unused. So, using `persist()` rather than `stow(persistent=true,..)` improves readability and is better to maintain. * CoreDb+Aristo: Store block numbers in journal records why: This makes journal records searchable by block numbers * Aristo: Rename some journal related functions why: The name *journal* is more appropriate to api functions than something with *fifo* or *filter*. * CoreDb+Aristo: Update last/oldest journal state retrieval * CoreDb+Aristo: Register block number with state root in journal why: No need anymore for extra lookup table `stRootToBlockNum` which maps a storage root -> block number. * Aristo: Remove unused function `getFilUbe()` from api * CoreDb: Remove now unused virtual table `stRootToBlockNum` why: Was used to map a state root to a block number. This functionality is now embedded into the recovery journal backend. * Turn of API tracking (will fail on `fluffy`)
2024-04-29 20:17:17 +00:00
FilSiblingsCommitUnfinshed
FilSrcTrgInconsistent
Implement backend filter mechanics (#1730) details: * Tested features + Successively store filters with increasing filter ID (think block number) + Cascading through fifos, deeper fifos merge groups of filters + Fetch squash merged N top fifos + Delete N top fifos, push back merged fifo, continue storing + Fifo chain is verified by hashes and filter ID * Not tested yet + Real live scenario (using data dumps) + Real filter data (only shallow filters used so far)
2023-09-05 18:00:40 +00:00
FilStateRootMismatch
FilTrgSrcMismatch
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
Aristo db allow shorter than 64 nibbles path keys (#1864) * Aristo: Single `FetchPathNotFound` error in `fetchXxx()` and `hasPath()` why: Missing path hike returns too many detailed reasons why it failed which becomes cumbersome to handle. also: Renamed `contains()` => `hasPath()` which disables the `in` operator on non-boolean `contains()` functions * Kvt: Renamed `contains()` => `hasKey()` why: which disables the `in` operator on non-boolean `contains()` functions * Aristo: Generalising `HashID` by variable length `PathID` why: There are cases when the `Aristo` database is to be used with shorter than 64 nibbles keys when handling transactions indexes with sequence IDs. caveat: This patch only works reliable for full length `PathID` values. Tests for shorter `PathID` values are currently missing.
2023-10-27 21:36:51 +00:00
# Get functions from `aristo_get.nim`
Core db use differential tx layers for aristo and kvt (#1949) * Fix kvt headers * Provide differential layers for KVT transaction stack why: Significant performance improvement * Provide abstraction layer for database top cache layer why: This will eventually implemented as a differential database layers or transaction layers. The latter is needed to improve performance. behavioural changes: Zero vertex and keys (i.e. delete requests) are not optimised out until the last layer is written to the database. * Provide differential layers for Aristo transaction stack why: Significant performance improvement
2023-12-19 12:39:23 +00:00
GetLeafMissing
GetKeyUpdateNeeded
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
GetLeafNotFound
GetVtxNotFound
GetKeyNotFound
Aristo db update serialisation (#1700) * Remove unused unit test sources * Redefine and document serialised data records for Aristo backend why: Unique record types determined by marker byte, i.e. the last byte of a serialisation record. This just needed some tweaking after adding new record types.
2023-08-21 18:18:06 +00:00
GetFilNotFound
Remove vid recycling feature (#2294)
2024-06-04 15:05:13 +00:00
GetTuvNotFound
Run coredb without journal (#2266) * Add persistent last state stamp feature why: This allows to run `CoreDb` without journal * Start `CoreDb` without journal * Remove journal related functions from `CoredDb`
2024-05-31 17:32:22 +00:00
GetLstNotFound
Aristo db implement filter storage scheduler (#1713) * Rename FilterID => QueueID why: The current usage does not identify a particular filter but uses it as storage tag to manage it on the database (to be organised in a set of FIFOs or queues.) * Split `aristo_filter` source into sub-files why: Make space for filter management API * Store filter queue IDs in pairs on the backend why: Any pair will will describe a FIFO accessed by bottom/top IDs * Reorg some source file names why: The "aristo_" prefix for make local/private files is tedious to use, so removed. * Implement filter slot scheduler details: Filters will be stored on the database on cascaded FIFOs. When a FIFO queue is full, some filter items are bundled together and stored on the next FIFO.
2023-08-25 22:53:59 +00:00
GetFqsNotFound
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
Aristo db allow shorter than 64 nibbles path keys (#1864) * Aristo: Single `FetchPathNotFound` error in `fetchXxx()` and `hasPath()` why: Missing path hike returns too many detailed reasons why it failed which becomes cumbersome to handle. also: Renamed `contains()` => `hasPath()` which disables the `in` operator on non-boolean `contains()` functions * Kvt: Renamed `contains()` => `hasKey()` why: which disables the `in` operator on non-boolean `contains()` functions * Aristo: Generalising `HashID` by variable length `PathID` why: There are cases when the `Aristo` database is to be used with shorter than 64 nibbles keys when handling transactions indexes with sequence IDs. caveat: This patch only works reliable for full length `PathID` values. Tests for shorter `PathID` values are currently missing.
2023-10-27 21:36:51 +00:00
# Fetch functions from `aristo_fetch.nim`
FetchPathNotFound
Aristo: Update unit test suite (#2002) * Aristo: Update unit test suite * Aristo/Kvt: Fix iterators why: Generic iterators were not properly updated after backend change * Aristo: Add sub-trie deletion functionality why: For storage tries linked to an account payload vertex ID, a the whole storage trie needs to be deleted with the account. * Aristo: Reserve vertex ID numbers for static custom state roots why: Static custom state roots may be controlled by an application, e.g. for a receipt or a transaction root. The `Aristo` functions are agnostic of what the static state roots are when different from the internal tree vertex ID 1. details; The `merge()` function applied to a non-static state root (assumed to be a storage root) will check the payload of an accounts leaf and mark its Merkle keys to be re-checked. * Aristo: Correct error code symbol * Aristo: Update error code symbols * Aristo: Code cosmetics/comments * Aristo: Fix hashify schedule calculator why: Had a tendency to stop early leaving an incomplete job
2024-02-01 21:27:48 +00:00
LeafKeyInvalid
Aristo db allow shorter than 64 nibbles path keys (#1864) * Aristo: Single `FetchPathNotFound` error in `fetchXxx()` and `hasPath()` why: Missing path hike returns too many detailed reasons why it failed which becomes cumbersome to handle. also: Renamed `contains()` => `hasPath()` which disables the `in` operator on non-boolean `contains()` functions * Kvt: Renamed `contains()` => `hasKey()` why: which disables the `in` operator on non-boolean `contains()` functions * Aristo: Generalising `HashID` by variable length `PathID` why: There are cases when the `Aristo` database is to be used with shorter than 64 nibbles keys when handling transactions indexes with sequence IDs. caveat: This patch only works reliable for full length `PathID` values. Tests for shorter `PathID` values are currently missing.
2023-10-27 21:36:51 +00:00
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
# RocksDB backend
RdbBeCantCreateDataDir
RdbBeCantCreateTmpDir
RdbBeDriverDelError
Aristo update rocksdb backend drivers (#2135) * Aristo+RocksDB: Update backend drivers why: RocksDB update allows use some of the newly provided methods which were previously implemented by using the very C backend (for the lack of NIM methods.) * Aristo+RocksDB: Simplify drivers wrapper * Kvt: Update backend drivers and wrappers similar to `Aristo` * Aristo+Kvm: Use column families for RocksDB * Aristo+MemoryDB: Code cosmetics * Aristo: Provide guest column family for export why: So `Kvt` can piggyback on `Aristo` so there avoiding to run a second DBMS system in parallel. * Kvt: Provide import mechanism for RoksDB guest column family why: So `Kvt` can piggyback on `Aristo` so there avoiding to run a second DBMS system in parallel. * CoreDb+Aristo: Run persistent `Kvt` DB piggybacked on `Aristo` why: Avoiding to run two DBMS systems in parallel. * Fix copyright year * Ditto
2024-04-16 20:39:11 +00:00
RdbBeDriverGetError
RdbBeDriverGuestError
RdbBeDriverInitError
RdbBeDriverPutError
RdbBeDriverWriteError
Kvt remodel tx and forked descriptors (#2168) * Aristo: Generalise alien/guest interface for piggiback on database * Aristo: Code cosmetics * CoreDb+Kvt: Update transaction API why: Use single addressable function `forkTx(backLevel: int)` as used in `Aristo`. So `Kvt` can be synced simultaneously to `Aristo`. also: Refactored `kvt_tx.nim` in a similar fashion to `Aristo`. * Kvt: Replace `LayerDelta` object by reference why: Will be needed when introducing filters * Kvt: Remodel backend filter facility similar to `Aristo` why: This allows to operate on several KVT instances simultaneously. * CoreDb+Kvt: Fix on-disk storage why: Overlooked name change: `stow()` => `persist()` for permanent storage * Fix copyright headers
2024-05-07 19:59:27 +00:00
RdbGuestInstanceUnsupported
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
RdbHashKeyExpected
Aristo db supporting forest and layered tx architecture (#1598) * Exclude some storage tests why: These test running on external dumps slipped through. The particular dumps were reported earlier as somehow dodgy. This was changed in `#1457` but having a second look, the change on hexary_interpolate.nim(350) might be incorrect. * Redesign `Aristo DB` descriptor for transaction based layers why: Previous descriptor layout made it cumbersome to push/pop database delta layers. The new architecture keeps each layer with the full delta set relative to the database backend. * Keep root ID as part of the `Patricia Trie` leaf path why; That way, forests are supported
2023-06-09 11:17:37 +00:00
Aristo db transaction based interface (#1628) * Provide transaction based interface for standard operations * Provide unit tests for new Aristo interface using transactions details: These new tests combine and replace several single-purpose tests. The now unused test sources will be kept for a while to be eventually removed.
2023-07-05 13:50:11 +00:00
# Transaction wrappers
Aristo avoid storage trie update race conditions (#2251) * Update TDD suite logger output format choices why: New format is not practical for TDD as it just dumps data across a wide range (considerably larder than 80 columns.) So the new format can be turned on by function argument. * Update unit tests samples configuration why: Slightly changed the way to find the `era1` directory * Remove compiler warnings (fix deprecated expressions and phrases) * Update `Aristo` debugging tools * Always update the `storageID` field of account leaf vertices why: Storage tries are weekly linked to an account leaf object in that the `storageID` field is updated by the application. Previously, `Aristo` verified that leaf objects make sense when passed to the database. As a consequence * the database was inconsistent for a short while * the burden for correctness was all on the application which led to delayed error handling which is hard to debug. So `Aristo` will internally update the account leaf objects so that there are no race conditions due to the storage trie handling * Aristo: Let `stow()`/`persist()` bail out unless there is a `VertexID(1)` why: The journal and filter logic depends on the hash of the `VertexID(1)` which is commonly known as the state root. This implies that all changes to the database are somehow related to that. * Make sure that a `Ledger` account does not overwrite the storage trie reference why: Due to the abstraction of a sub-trie (now referred to as column with a hash describing its state) there was a weakness in the `Aristo` handler where an account leaf could be overwritten though changing the validity of the database. This has been changed and the database will now reject such changes. This patch fixes the behaviour on the application layer. In particular, the column handle returned by the `CoreDb` needs to be updated by the `Aristo` database state. This mitigates the problem that a storage trie might have vanished or re-apperaed with a different vertex ID. * Fix sub-trie deletion test why: Was originally hinged on `VertexID(1)` which cannot be wholesale deleted anymore after the last Aristo update. Also, running with `VertexID(2)` needs an artificial `VertexID(1)` for making `stow()` or `persist()` work. * Cosmetics * Activate `test_generalstate_json` * Temporarily `deactivate test_tracer_json` * Fix copyright header --------- Co-authored-by: jordan <jordan@dry.pudding> Co-authored-by: Jacek Sieka <jacek@status.im>
2024-05-30 17:48:38 +00:00
TxAccRootMissing
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
TxArgStaleTx
Core db+aristo provides tracer funtionality (#2089) * Aristo: Provide descriptor fork based on search in transaction stack details: Try to find the tx that has a particular pair `(vertex-id,hash-key)`, and by extension try filter and backend if the former fails. * Cleanup & docu * CoreDb+Aristo: Implement context re-position to earlier in-memory state why: It is a easy way to explore how there can be concurrent access to the same backend storage DB with different view states. This one can access an earlier state from the transaction stack. * CoreDb+Aristo: Populate tracer stubs with real functionality * Update `tracer.nim` to new API why: Legacy API does not sufficiently support `Aristo` * Fix logging problems in tracer details: Debug logging turned off by default * Fix function prototypes * Add Copyright header * Add tables import why: For older compiler versions on CI
2024-03-21 10:45:57 +00:00
TxArgsUseless
Aristo db code massage n cosmetics (#1745) * Rewrite remaining `AristoError` return code into `Result[void,AristoError]` why: Better code maintenance * Update import sections * Update Aristo DB paths why: More systematic so directory can be shared with other DB types * More cosmetcs * Update unit tests runners why: Proper handling of persistent and mem-only DB. The latter can be consistently triggered by an empty DB path.
2023-09-12 18:45:12 +00:00
TxBackendNotWritable
Core db+aristo fixes and tx handling updates (#2164) * Aristo: Rename journal related sources and functions why: Previously, the naming was hinged on the phrases `fifo`, `filter` etc. which reflect the inner workings of cascaded filters. This was unfortunate for reading/understanding the source code for actions where the focus is the journal as a whole. * Aristo: Fix buffer overflow (path length truncating error) * Aristo: Tighten `hikeUp()` stop check, update error code why: Detect dangling vertex links. These are legit with `snap` sync processing but not with regular processing. * Aristo: Raise assert in regular mode `merge()` at a dangling link/edge why: With `snap` sync processing, partial trees are ok and can be amended. Not so in regular mode. Previously there was only a debug message when a non-legit dangling edge was encountered. * Aristo: Make sure that vertices are copied before modification why: Otherwise vertices from lower layers might also be modified * Aristo: Fix relaxed mode for validity checker `check()` * Remove cruft * Aristo: Update API for transaction handling details: + Split `aristo_tx.nim` into sub-modules + Split `forkWith()` into `findTx()` + `forkTx()` + Removed `forkTop()`, `forkBase()` (now superseded by new `forkTx()`) * CoreDb+Aristo: Fix initialiser (missing methods)
2024-05-03 17:38:17 +00:00
TxLevelTooDeep
TxLevelUseless
Aristo db integrate hashify into tx (#1679) * Renamed type `NoneBackendRef` => `VoidBackendRef` * Clarify names: `BE=filter+backend` and `UBE=backend (unfiltered)` why: Most functions used full names as `getVtxUnfilteredBackend()` or `getKeyBackend()`. After defining abbreviations (and its meaning) it seems easier to use `getVtxUBE()` and `getKeyBE()`. * Integrate `hashify()` process into transaction logic why: Is now transparent unless explicitly controlled. details: Cache changes imply setting a `dirty` flag which in turn triggers `hashify()` processing in transaction and `pack()` directives. * Removed `aristo_tx.exec()` directive why: Inconsistent implementation, functionality will be provided with a different paradigm.
2023-08-11 17:23:57 +00:00
TxNoPendingTx
Core db+aristo provides tracer funtionality (#2089) * Aristo: Provide descriptor fork based on search in transaction stack details: Try to find the tx that has a particular pair `(vertex-id,hash-key)`, and by extension try filter and backend if the former fails. * Cleanup & docu * CoreDb+Aristo: Implement context re-position to earlier in-memory state why: It is a easy way to explore how there can be concurrent access to the same backend storage DB with different view states. This one can access an earlier state from the transaction stack. * CoreDb+Aristo: Populate tracer stubs with real functionality * Update `tracer.nim` to new API why: Legacy API does not sufficiently support `Aristo` * Fix logging problems in tracer details: Debug logging turned off by default * Fix function prototypes * Add Copyright header * Add tables import why: For older compiler versions on CI
2024-03-21 10:45:57 +00:00
TxNotFound
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
TxNotTopTx
Core db+aristo provides tracer funtionality (#2089) * Aristo: Provide descriptor fork based on search in transaction stack details: Try to find the tx that has a particular pair `(vertex-id,hash-key)`, and by extension try filter and backend if the former fails. * Cleanup & docu * CoreDb+Aristo: Implement context re-position to earlier in-memory state why: It is a easy way to explore how there can be concurrent access to the same backend storage DB with different view states. This one can access an earlier state from the transaction stack. * CoreDb+Aristo: Populate tracer stubs with real functionality * Update `tracer.nim` to new API why: Legacy API does not sufficiently support `Aristo` * Fix logging problems in tracer details: Debug logging turned off by default * Fix function prototypes * Add Copyright header * Add tables import why: For older compiler versions on CI
2024-03-21 10:45:57 +00:00
TxPendingTx
Aristo db integrate hashify into tx (#1679) * Renamed type `NoneBackendRef` => `VoidBackendRef` * Clarify names: `BE=filter+backend` and `UBE=backend (unfiltered)` why: Most functions used full names as `getVtxUnfilteredBackend()` or `getKeyBackend()`. After defining abbreviations (and its meaning) it seems easier to use `getVtxUBE()` and `getKeyBE()`. * Integrate `hashify()` process into transaction logic why: Is now transparent unless explicitly controlled. details: Cache changes imply setting a `dirty` flag which in turn triggers `hashify()` processing in transaction and `pack()` directives. * Removed `aristo_tx.exec()` directive why: Inconsistent implementation, functionality will be provided with a different paradigm.
2023-08-11 17:23:57 +00:00
TxStackGarbled
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
TxStackUnderflow
Aristo db transaction based interface (#1628) * Provide transaction based interface for standard operations * Provide unit tests for new Aristo interface using transactions details: These new tests combine and replace several single-purpose tests. The now unused test sources will be kept for a while to be eventually removed.
2023-07-05 13:50:11 +00:00
Aristo cull journal related stuff (#2288) * Remove all journal related stuff * Refactor function names journal*() => delta*(), filter*() => delta*() * remove `trg` fileld from `FilterRef` why: Same as `kMap[$1]` * Re-type FilterRef.src as `HashKey` why: So it is directly comparable to `kMap[$1]` * Moved `vGen[]` field from `LayerFinalRef` to `LayerDeltaRef` why: Then a separate `FilterRef` type is not needed, anymore * Rename `roFilter` field in `AristoDbRef` => `balancer` why: New name more appropriate. * Replace `FilterRef` by `LayerDeltaRef` type why: This allows to avoid copying into the `balancer` (see next patch set) most of the time. Typically, only one instance is running on the backend and the `balancer` is only used as a stage before saving data. * Refactor way how to store data persistently why: Avoid useless copy when staging `top` layer for persistently saving to backend. * Fix copyright header?
2024-06-03 20:10:35 +00:00
TxPrettyPointlessLayer
TxStateRootMismatch
Aristo: Update unit test suite (#2002) * Aristo: Update unit test suite * Aristo/Kvt: Fix iterators why: Generic iterators were not properly updated after backend change * Aristo: Add sub-trie deletion functionality why: For storage tries linked to an account payload vertex ID, a the whole storage trie needs to be deleted with the account. * Aristo: Reserve vertex ID numbers for static custom state roots why: Static custom state roots may be controlled by an application, e.g. for a receipt or a transaction root. The `Aristo` functions are agnostic of what the static state roots are when different from the internal tree vertex ID 1. details; The `merge()` function applied to a non-static state root (assumed to be a storage root) will check the payload of an accounts leaf and mark its Merkle keys to be re-checked. * Aristo: Correct error code symbol * Aristo: Update error code symbols * Aristo: Code cosmetics/comments * Aristo: Fix hashify schedule calculator why: Had a tendency to stop early leaving an incomplete job
2024-02-01 21:27:48 +00:00
# Functions from `aristo_desc.nim`
Aristo db state root repos and reorg (#1744) * Reorg of distributed backend access details: Now handled via API provided in `aristo_desc`. * Rename `checkCache()` => `checkTop()` why: Better naming for top layer cache checker also: Provide cascaded fifos checker * Provide `eq` directive for finding filter by exact filter ID (think block number) * Some code beautification (for better code reading) * State root reposition and reorg details: Repositioning is supported by forking a new descriptor. Reorg is then accomplished by writing this forked state on the backend database.
2023-09-11 20:38:49 +00:00
MustBeOnCentre
NotAllowedOnCentre
Core db simplify new api storage modes (#2075) * Aristo+Kvt: Fix backend `dup()` function in api setup why: Backend object is subject to an inheritance cascade which was not taken care of, before. Only the base object was duplicated. * Kvt: Simplify DB clone/peers management * Aristo: Simplify DB clone/peers management * Aristo: Adjust unit test for working with memory DB only why: This currently causes some memory corruption persumably in the `libc` background layer. * CoredDb+Kvt: Simplify API for KVT why: Simplified storage models (was over engineered) for better performance and code maintenance. * CoredDb+Aristo: Simplify API for `Aristo` why: Only single database state needed here. Accessing a similar state will be implemented from outside this module using a context layer. This gives better performance and improves code maintenance. * Fix Copyright headers * CoreDb: Turn off API tracking why: CI would ot go through. Was accidentally turned on.
2024-03-14 22:17:43 +00:00
StaleDescriptor
Aristo db state root repos and reorg (#1744) * Reorg of distributed backend access details: Now handled via API provided in `aristo_desc`. * Rename `checkCache()` => `checkTop()` why: Better naming for top layer cache checker also: Provide cascaded fifos checker * Provide `eq` directive for finding filter by exact filter ID (think block number) * Some code beautification (for better code reading) * State root reposition and reorg details: Repositioning is supported by forking a new descriptor. Reorg is then accomplished by writing this forked state on the backend database.
2023-09-11 20:38:49 +00:00
Aristo: Update unit test suite (#2002) * Aristo: Update unit test suite * Aristo/Kvt: Fix iterators why: Generic iterators were not properly updated after backend change * Aristo: Add sub-trie deletion functionality why: For storage tries linked to an account payload vertex ID, a the whole storage trie needs to be deleted with the account. * Aristo: Reserve vertex ID numbers for static custom state roots why: Static custom state roots may be controlled by an application, e.g. for a receipt or a transaction root. The `Aristo` functions are agnostic of what the static state roots are when different from the internal tree vertex ID 1. details; The `merge()` function applied to a non-static state root (assumed to be a storage root) will check the payload of an accounts leaf and mark its Merkle keys to be re-checked. * Aristo: Correct error code symbol * Aristo: Update error code symbols * Aristo: Code cosmetics/comments * Aristo: Fix hashify schedule calculator why: Had a tendency to stop early leaving an incomplete job
2024-02-01 21:27:48 +00:00
# Functions from `aristo_utils.nim`
AccRlpDecodingError
AccStorageKeyMissing
AccVtxUnsupported
AccNodeUnsupported
Aristo db traversal helpers (#1638) * Misc fixes detail: * Fix de-serialisation for account leafs * Update node recovery from unit tests * Remove `LegacyAccount` from `PayloadRef` object why: Legacy accounts use a hash key as storage root which is detrimental to the working of the Aristo database which uses a vertex ID. * Dissolve `hashify_helper` into `aristo_utils` and `aristo_transcode` why: Functions are of general interest so they should live in first level code files. * Added left/right iterators over leaf nodes * Some helper/wrapper functions that might be useful
2023-07-12 23:03:14 +00:00
PayloadTypeUnsupported
Added support for namespaces to RocksDb kvstore. (#2066) * Add new RocksNamespaceRef type and remove backups and readonly support from RocksDb KvStore. * Bump nim-rocksdb to fc2ba4a836b6b47ae1b17d1c45801c7e06585e19 * Fix tests. * Fix copyright notice.
2024-03-12 03:04:46 +00:00
Aristo: Update unit test suite (#2002) * Aristo: Update unit test suite * Aristo/Kvt: Fix iterators why: Generic iterators were not properly updated after backend change * Aristo: Add sub-trie deletion functionality why: For storage tries linked to an account payload vertex ID, a the whole storage trie needs to be deleted with the account. * Aristo: Reserve vertex ID numbers for static custom state roots why: Static custom state roots may be controlled by an application, e.g. for a receipt or a transaction root. The `Aristo` functions are agnostic of what the static state roots are when different from the internal tree vertex ID 1. details; The `merge()` function applied to a non-static state root (assumed to be a storage root) will check the payload of an accounts leaf and mark its Merkle keys to be re-checked. * Aristo: Correct error code symbol * Aristo: Update error code symbols * Aristo: Code cosmetics/comments * Aristo: Fix hashify schedule calculator why: Had a tendency to stop early leaving an incomplete job
2024-02-01 21:27:48 +00:00
# Miscelaneous handy helpers
AccRootUnacceptable
MptRootUnacceptable
Core db+friends fix sync bail outs (#2026) * Aristo: Update schedule runner for `hashify()` why: Width-first schedule walker overlooked resolvable vertex and ended in a deadlock situation * CoreDb+Aristo: Update error code for fringe condition * Ledger: Remove redundant extra MPT hashing statements why: These statement were used for troubleshooting * CoreDb: Couch medicine for legacy DB backend why: MPT will occasionally enter inconsistent states after deletion, some dirty fix is randomly added to mitgate that.
2024-02-14 10:02:09 +00:00
MptRootMissing
Core db update providing additional results code interface (#1776) * Split `core_db/base.nim` into several sources * Rename `core_db/legacy.nim` => `core_db/legacy_db.nim` * Update `CoreDb` API, dual methods returning `Result[]` or plain value detail: Plain value methods implemet the legacy API, they defect on error results * Redesign `CoreDB` direct backend access why: Made the `backend` directive integral part of the API * Discontinue providing unused or otherwise available functions details: + setTransactionID() removed, not used and not easily replicable in Aristo + maybeGet() removed, available via direct backend access + newPhk() removed, never used & was experimental anyway * Update/reorg backend API why: + Added error print function `$$()` + General descriptor completion (and optional validation) via `bless()` * Update `Aristo`/`Kvt` exception handling why: Avoid `CatchableError` exceptions, rather pass them as error code where appropriate. * More `CoreDB` compliant `Aristo` and `Kvt` methods details: + Providing functions like `contains()`, `getVtxRc()` (returns `Result[]`). + Additional error code: `NotImplemented` * Rewrite/reorg of Aristo DB constructor why: Previously used global object `DefaultQidLayoutRef` as default initialiser. This object was created at compile time which lead to non-gc safe functions. * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/aristo/aristo_transcode.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> --------- Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-09-26 09:21:13 +00:00
NotImplemented
Core db simplify new api storage modes (#2075) * Aristo+Kvt: Fix backend `dup()` function in api setup why: Backend object is subject to an inheritance cascade which was not taken care of, before. Only the base object was duplicated. * Kvt: Simplify DB clone/peers management * Aristo: Simplify DB clone/peers management * Aristo: Adjust unit test for working with memory DB only why: This currently causes some memory corruption persumably in the `libc` background layer. * CoredDb+Kvt: Simplify API for KVT why: Simplified storage models (was over engineered) for better performance and code maintenance. * CoredDb+Aristo: Simplify API for `Aristo` why: Only single database state needed here. Accessing a similar state will be implemented from outside this module using a context layer. This gives better performance and improves code maintenance. * Fix Copyright headers * CoreDb: Turn off API tracking why: CI would ot go through. Was accidentally turned on.
2024-03-14 22:17:43 +00:00
TrieInvalid
Aristo: Update unit test suite (#2002) * Aristo: Update unit test suite * Aristo/Kvt: Fix iterators why: Generic iterators were not properly updated after backend change * Aristo: Add sub-trie deletion functionality why: For storage tries linked to an account payload vertex ID, a the whole storage trie needs to be deleted with the account. * Aristo: Reserve vertex ID numbers for static custom state roots why: Static custom state roots may be controlled by an application, e.g. for a receipt or a transaction root. The `Aristo` functions are agnostic of what the static state roots are when different from the internal tree vertex ID 1. details; The `merge()` function applied to a non-static state root (assumed to be a storage root) will check the payload of an accounts leaf and mark its Merkle keys to be re-checked. * Aristo: Correct error code symbol * Aristo: Update error code symbols * Aristo: Code cosmetics/comments * Aristo: Fix hashify schedule calculator why: Had a tendency to stop early leaving an incomplete job
2024-02-01 21:27:48 +00:00
VidContextLocked
Aristo db traversal helpers (#1638) * Misc fixes detail: * Fix de-serialisation for account leafs * Update node recovery from unit tests * Remove `LegacyAccount` from `PayloadRef` object why: Legacy accounts use a hash key as storage root which is detrimental to the working of the Aristo database which uses a vertex ID. * Dissolve `hashify_helper` into `aristo_utils` and `aristo_transcode` why: Functions are of general interest so they should live in first level code files. * Added left/right iterators over leaf nodes * Some helper/wrapper functions that might be useful
2023-07-12 23:03:14 +00:00
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 11:47:47 +00:00
# End