status-im/nimbus-eth1
2
0
Fork 0
mirror of https://github.com/status-im/nimbus-eth1.git synced 2025-03-01 12:20:49 +00:00
Code Issues Projects Releases Wiki Activity
nimbus-eth1/execution_chain/db/aristo/aristo_desc/desc_error.nim

207 lines
4.9 KiB
Nim
Raw Normal View History

Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 15:25:29 +01:00
# nimbus-eth1
aristo: fork support via layers/txframes (#2960) * aristo: fork support via layers/txframes This change reorganises how the database is accessed: instead holding a "current frame" in the database object, a dag of frames is created based on the "base frame" held in `AristoDbRef` and all database access happens through this frame, which can be thought of as a consistent point-in-time snapshot of the database based on a particular fork of the chain. In the code, "frame", "transaction" and "layer" is used to denote more or less the same thing: a dag of stacked changes backed by the on-disk database. Although this is not a requirement, in practice each frame holds the change set of a single block - as such, the frame and its ancestors leading up to the on-disk state represents the state of the database after that block has been applied. "committing" means merging the changes to its parent frame so that the difference between them is lost and only the cumulative changes remain - this facility enables frames to be combined arbitrarily wherever they are in the dag. In particular, it becomes possible to consolidate a set of changes near the base of the dag and commit those to disk without having to re-do the in-memory frames built on top of them - this is useful for "flattening" a set of changes during a base update and sending those to storage without having to perform a block replay on top. Looking at abstractions, a side effect of this change is that the KVT and Aristo are brought closer together by considering them to be part of the "same" atomic transaction set - the way the code gets organised, applying a block and saving it to the kvt happens in the same "logical" frame - therefore, discarding the frame discards both the aristo and kvt changes at the same time - likewise, they are persisted to disk together - this makes reasoning about the database somewhat easier but has the downside of increased memory usage, something that perhaps will need addressing in the future. Because the code reasons more strictly about frames and the state of the persisted database, it also makes it more visible where ForkedChain should be used and where it is still missing - in particular, frames represent a single branch of history while forkedchain manages multiple parallel forks - user-facing services such as the RPC should use the latter, ie until it has been finalized, a getBlock request should consider all forks and not just the blocks in the canonical head branch. Another advantage of this approach is that `AristoDbRef` conceptually becomes more simple - removing its tracking of the "current" transaction stack simplifies reasoning about what can go wrong since this state now has to be passed around in the form of `AristoTxRef` - as such, many of the tests and facilities in the code that were dealing with "stack inconsistency" are now structurally prevented from happening. The test suite will need significant refactoring after this change. Once this change has been merged, there are several follow-ups to do: * there's no mechanism for keeping frames up to date as they get committed or rolled back - TODO * naming is confused - many names for the same thing for legacy reason * forkedchain support is still missing in lots of code * clean up redundant logic based on previous designs - in particular the debug and introspection code no longer makes sense * the way change sets are stored will probably need revisiting - because it's a stack of changes where each frame must be interrogated to find an on-disk value, with a base distance of 128 we'll at minimum have to perform 128 frame lookups for *every* database interaction - regardless, the "dag-like" nature will stay * dispose and commit are poorly defined and perhaps redundant - in theory, one could simply let the GC collect abandoned frames etc, though it's likely an explicit mechanism will remain useful, so they stay for now More about the changes: * `AristoDbRef` gains a `txRef` field (todo: rename) that "more or less" corresponds to the old `balancer` field * `AristoDbRef.stack` is gone - instead, there's a chain of `AristoTxRef` objects that hold their respective "layer" which has the actual changes * No more reasoning about "top" and "stack" - instead, each `AristoTxRef` can be a "head" that "more or less" corresponds to the old single-history `top` notion and its stack * `level` still represents "distance to base" - it's computed from the parent chain instead of being stored * one has to be careful not to use frames where forkedchain was intended - layers are only for a single branch of history! * fix layer vtop after rollback * engine fix * Fix test_txpool * Fix test_rpc * Fix copyright year * fix simulator * Fix copyright year * Fix copyright year * Fix tracer * Fix infinite recursion bug * Remove aristo and kvt empty files * Fic copyright year * Fix fc chain_kvt * ForkedChain refactoring * Fix merge master conflict * Fix copyright year * Reparent txFrame * Fix test * Fix txFrame reparent again * Cleanup and fix test * UpdateBase bugfix and fix test * Fixe newPayload bug discovered by hive * Fix engine api fcu * Clean up call template, chain_kvt, andn txguid * Fix copyright year * work around base block loading issue * Add test * Fix updateHead bug * Fix updateBase bug * Change func commitBase to proc commitBase * Touch up and fix debug mode crash --------- Co-authored-by: jangko <jangko128@gmail.com>
2025-02-06 08:04:50 +01:00
# Copyright (c) 2023-2025 Status Research & Development GmbH
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 15:25:29 +01:00
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or
# http://www.apache.org/licenses/LICENSE-2.0)
# * MIT license ([LICENSE-MIT](LICENSE-MIT) or
# http://opensource.org/licenses/MIT)
# at your option. This file may not be copied, modified, or distributed
# except according to those terms.
type
AristoError* = enum
NothingSerious = 0
Aristo uses pre classified tree types cont2 (#2397) * Provide dedicated functions for fetching accounts and storage trees why: Different prototypes for each class `account`, `generic` and `storage`. * Remove `fetchPayload()` and other cruft from API, `aristo_fetch`, etc. * Fix typos, debugging left overs, comments
2024-06-19 12:40:00 +00:00
# Cache checker `checkCache()`
CheckAnyVidDeadStorageRoot
CheckAnyVidSharedStorageRoot
CheckAnyVtxEmptyKeyMissing
CheckAnyVtxEmptyKeyExpected
CheckAnyVtxEmptyKeyMismatch
CheckAnyVtxBranchLinksMissing
CheckAnyVtxLockWithoutKey
CheckAnyVTopUnset
CheckBeCacheGarbledVTop
CheckBeCacheKeyDangling
CheckBeCacheKeyNonEmpty
CheckBeGarbledVTop
CheckBeVtxBranchLinksMissing
CheckBeVtxInvalid
CheckBeVtxMissing
CheckStkKeyStrayZeroEntry
CheckStkVtxKeyMismatch
CheckRlxVtxIncomplete
CheckRlxVtxKeyMissing
CheckRlxVtxKeyMismatch
# De-serialiser from `blobify.nim`
No ext update (#2494) * Imported/rebase from `no-ext`, PR #2485 Store extension nodes together with the branch Extension nodes must be followed by a branch - as such, it makes sense to store the two together both in the database and in memory: * fewer reads, writes and updates to traverse the tree * simpler logic for maintaining the node structure * less space used, both memory and storage, because there are fewer nodes overall There is also a downside: hashes can no longer be cached for an extension - instead, only the extension+branch hash can be cached - this seems like a fine tradeoff since computing it should be fast. TODO: fix commented code * Fix merge functions and `toNode()` * Update `merkleSignCommit()` prototype why: Result is always a 32bit hash * Update short Merkle hash key generation details: Ethereum reference MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) This is specified in the yellow paper, appendix D. Different to the `Aristo` implementation, the reference MPT would not store such a node on the key-value database. Rather the RLP encoded node value is stored instead of a node link in a parent node is stored as a node link on the parent database. Only for the root hash, the top level node is always referred to by the hash. * Fix/update `Extension` sections why: Were commented out after removal of a dedicated `Extension` type which left the system disfunctional. * Clean up unused error codes * Update unit tests * Update docu --------- Co-authored-by: Jacek Sieka <jacek@status.im>
2024-07-16 19:47:59 +00:00
Deblob256LenUnsupported
Deblob64LenUnsupported
DeblobBranchGotLeafPrefix
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 14:26:25 +01:00
DeblobBranchTooShort
No ext update (#2494) * Imported/rebase from `no-ext`, PR #2485 Store extension nodes together with the branch Extension nodes must be followed by a branch - as such, it makes sense to store the two together both in the database and in memory: * fewer reads, writes and updates to traverse the tree * simpler logic for maintaining the node structure * less space used, both memory and storage, because there are fewer nodes overall There is also a downside: hashes can no longer be cached for an extension - instead, only the extension+branch hash can be cached - this seems like a fine tradeoff since computing it should be fast. TODO: fix commented code * Fix merge functions and `toNode()` * Update `merkleSignCommit()` prototype why: Result is always a 32bit hash * Update short Merkle hash key generation details: Ethereum reference MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) This is specified in the yellow paper, appendix D. Different to the `Aristo` implementation, the reference MPT would not store such a node on the key-value database. Rather the RLP encoded node value is stored instead of a node link in a parent node is stored as a node link on the parent database. Only for the root hash, the top level node is always referred to by the hash. * Fix/update `Extension` sections why: Were commented out after removal of a dedicated `Extension` type which left the system disfunctional. * Clean up unused error codes * Update unit tests * Update docu --------- Co-authored-by: Jacek Sieka <jacek@status.im>
2024-07-16 19:47:59 +00:00
DeblobCodeLenUnsupported
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 14:26:25 +01:00
DeblobExtSizeGarbled
DeblobLeafGotExtPrefix
No ext update (#2494) * Imported/rebase from `no-ext`, PR #2485 Store extension nodes together with the branch Extension nodes must be followed by a branch - as such, it makes sense to store the two together both in the database and in memory: * fewer reads, writes and updates to traverse the tree * simpler logic for maintaining the node structure * less space used, both memory and storage, because there are fewer nodes overall There is also a downside: hashes can no longer be cached for an extension - instead, only the extension+branch hash can be cached - this seems like a fine tradeoff since computing it should be fast. TODO: fix commented code * Fix merge functions and `toNode()` * Update `merkleSignCommit()` prototype why: Result is always a 32bit hash * Update short Merkle hash key generation details: Ethereum reference MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) This is specified in the yellow paper, appendix D. Different to the `Aristo` implementation, the reference MPT would not store such a node on the key-value database. Rather the RLP encoded node value is stored instead of a node link in a parent node is stored as a node link on the parent database. Only for the root hash, the top level node is always referred to by the hash. * Fix/update `Extension` sections why: Were commented out after removal of a dedicated `Extension` type which left the system disfunctional. * Clean up unused error codes * Update unit tests * Update docu --------- Co-authored-by: Jacek Sieka <jacek@status.im>
2024-07-16 19:47:59 +00:00
DeblobLeafSizeGarbled
storage: store root id together with vid, for better locality of refe… (#2449) The state and account MPT:s currenty share key space in the database based on that vertex id:s are assigned essentially randomly, which means that when two adjacent slot values from the same contract are accessed, they might reside at large distance from each other. Here, we prefix each vertex id by its root causing them to be sorted together thus bringing all data belonging to a particular contract closer together - the same effect also happens for the main state MPT whose nodes now end up clustered together more tightly. In the future, the prefix given to the storage keys can also be used to perform range operations such as reading all the storage at once and/or deleting an account with a batch operation. Notably, parts of the API already supported this rooting concept while parts didn't - this PR makes the API consistent by always working with a root+vid.
2024-07-04 15:46:52 +02:00
DeblobRVidLenUnsupported
No ext update (#2494) * Imported/rebase from `no-ext`, PR #2485 Store extension nodes together with the branch Extension nodes must be followed by a branch - as such, it makes sense to store the two together both in the database and in memory: * fewer reads, writes and updates to traverse the tree * simpler logic for maintaining the node structure * less space used, both memory and storage, because there are fewer nodes overall There is also a downside: hashes can no longer be cached for an extension - instead, only the extension+branch hash can be cached - this seems like a fine tradeoff since computing it should be fast. TODO: fix commented code * Fix merge functions and `toNode()` * Update `merkleSignCommit()` prototype why: Result is always a 32bit hash * Update short Merkle hash key generation details: Ethereum reference MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) This is specified in the yellow paper, appendix D. Different to the `Aristo` implementation, the reference MPT would not store such a node on the key-value database. Rather the RLP encoded node value is stored instead of a node link in a parent node is stored as a node link on the parent database. Only for the root hash, the top level node is always referred to by the hash. * Fix/update `Extension` sections why: Were commented out after removal of a dedicated `Extension` type which left the system disfunctional. * Clean up unused error codes * Update unit tests * Update docu --------- Co-authored-by: Jacek Sieka <jacek@status.im>
2024-07-16 19:47:59 +00:00
DeblobUnknown
DeblobVtxTooShort
DeblobWrongSize
DeblobWrongType
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 15:25:29 +01:00
Aristo uses pre classified tree types cont2 (#2397) * Provide dedicated functions for fetching accounts and storage trees why: Different prototypes for each class `account`, `generic` and `storage`. * Remove `fetchPayload()` and other cruft from API, `aristo_fetch`, etc. * Fix typos, debugging left overs, comments
2024-06-19 12:40:00 +00:00
# Deletion of vertex paths, `deleteXxx()`
DelAccRootNotAccepted
DelBranchExpexted
DelBranchWithoutRefs
DelDanglingStoTrie
DelLeafExpexted
DelPathNotFound
DelRootVidMissing
DelStoAccMissing
DelStoRootMissing
DelStoRootNotAccepted
DelVidStaleVtx
# Fetch functions from `aristo_fetch.nim`
Update storage tree admin (#2419) * Tighten `CoreDb` API for accounts why: Apart from cruft, the way to fetch the accounts state root via a `CoreDbColRef` record was unnecessarily complicated. * Extend `CoreDb` API for accounts to cover storage tries why: In future, this will make the notion of column objects obsolete. Storage trees will then be indexed by the account address rather than the vertex ID equivalent like a `CoreDbColRef`. * Apply new/extended accounts API to ledger and tests details: This makes the `distinct_ledger` module obsolete * Remove column object constructors why: They were needed as an abstraction of MPT sub-trees including storage trees. Now, storage trees are handled by the account (e.g. via address) they belong to and all other trees can be identified by a constant well known vertex ID. So there is no need for column objects anymore. Still there are some left-over column object methods wnich will be removed next. * Remove `serialise()` and `PayloadRef` from default Aristo API why: Not needed. `PayloadRef` was used for unstructured/unknown payload formats (account or blob) and `serialise()` was used for decodng `PayloadRef`. Now it is known in advance what the payload looks like. * Added query function `hasStorageData()` whether a storage area exists why: Useful for supporting `slotStateEmpty()` of the `CoreDb` API * In the `Ledger` replace `storage.stateEmpty()` by `slotStateEmpty()` * On Aristo, hide the storage root/vertex ID in the `PayloadRef` why: The storage vertex ID is fully controlled by Aristo while the `AristoAccount` object is controlled by the application. With the storage root part of the `AristoAccount` object, there was a useless administrative burden to keep that storage root field up to date. * Remove cruft, update comments etc. * Update changed MPT access paradigms why: Fixes verified proxy tests * Fluffy cosmetics
2024-06-27 09:01:26 +00:00
FetchAccInaccessible
FetchAccPathWithoutLeaf
FetchAccRootNotAccepted
Aristo uses pre classified tree types cont2 (#2397) * Provide dedicated functions for fetching accounts and storage trees why: Different prototypes for each class `account`, `generic` and `storage`. * Remove `fetchPayload()` and other cruft from API, `aristo_fetch`, etc. * Fix typos, debugging left overs, comments
2024-06-19 12:40:00 +00:00
FetchLeafKeyInvalid
FetchPathInvalid
Update storage tree admin (#2419) * Tighten `CoreDb` API for accounts why: Apart from cruft, the way to fetch the accounts state root via a `CoreDbColRef` record was unnecessarily complicated. * Extend `CoreDb` API for accounts to cover storage tries why: In future, this will make the notion of column objects obsolete. Storage trees will then be indexed by the account address rather than the vertex ID equivalent like a `CoreDbColRef`. * Apply new/extended accounts API to ledger and tests details: This makes the `distinct_ledger` module obsolete * Remove column object constructors why: They were needed as an abstraction of MPT sub-trees including storage trees. Now, storage trees are handled by the account (e.g. via address) they belong to and all other trees can be identified by a constant well known vertex ID. So there is no need for column objects anymore. Still there are some left-over column object methods wnich will be removed next. * Remove `serialise()` and `PayloadRef` from default Aristo API why: Not needed. `PayloadRef` was used for unstructured/unknown payload formats (account or blob) and `serialise()` was used for decodng `PayloadRef`. Now it is known in advance what the payload looks like. * Added query function `hasStorageData()` whether a storage area exists why: Useful for supporting `slotStateEmpty()` of the `CoreDb` API * In the `Ledger` replace `storage.stateEmpty()` by `slotStateEmpty()` * On Aristo, hide the storage root/vertex ID in the `PayloadRef` why: The storage vertex ID is fully controlled by Aristo while the `AristoAccount` object is controlled by the application. With the storage root part of the `AristoAccount` object, there was a useless administrative burden to keep that storage root field up to date. * Remove cruft, update comments etc. * Update changed MPT access paradigms why: Fixes verified proxy tests * Fluffy cosmetics
2024-06-27 09:01:26 +00:00
FetchPathNotFound
Fringe case portal proof for existing account without storage tree (#2613) detail: For practical reasons, ifsuch an account is asked for a slot, an empty proof list is returned. It is up to the user to provide an account proof that shows that there is no storage tree.
2024-09-11 20:27:42 +00:00
FetchPathStoRootMissing
Aristo uses pre classified tree types cont2 (#2397) * Provide dedicated functions for fetching accounts and storage trees why: Different prototypes for each class `account`, `generic` and `storage`. * Remove `fetchPayload()` and other cruft from API, `aristo_fetch`, etc. * Fix typos, debugging left overs, comments
2024-06-19 12:40:00 +00:00
FetchRootVidMissing
FetchStoRootNotAccepted
# Get functions from `aristo_get.nim`
GetFilNotFound
GetFqsNotFound
GetKeyNotFound
GetKeyUpdateNeeded
GetLstNotFound
GetTuvNotFound
GetVtxNotFound
Aristo db supporting forest and layered tx architecture (#1598) * Exclude some storage tests why: These test running on external dumps slipped through. The particular dumps were reported earlier as somehow dodgy. This was changed in `#1457` but having a second look, the change on hexary_interpolate.nim(350) might be incorrect. * Redesign `Aristo DB` descriptor for transaction based layers why: Previous descriptor layout made it cumbersome to push/pop database delta layers. The new architecture keeps each layer with the full delta set relative to the database backend. * Keep root ID as part of the `Patricia Trie` leaf path why; That way, forests are supported
2023-06-09 12:17:37 +01:00
# Path function `hikeUp()`
Aristo: Update unit test suite (#2002) * Aristo: Update unit test suite * Aristo/Kvt: Fix iterators why: Generic iterators were not properly updated after backend change * Aristo: Add sub-trie deletion functionality why: For storage tries linked to an account payload vertex ID, a the whole storage trie needs to be deleted with the account. * Aristo: Reserve vertex ID numbers for static custom state roots why: Static custom state roots may be controlled by an application, e.g. for a receipt or a transaction root. The `Aristo` functions are agnostic of what the static state roots are when different from the internal tree vertex ID 1. details; The `merge()` function applied to a non-static state root (assumed to be a storage root) will check the payload of an accounts leaf and mark its Merkle keys to be re-checked. * Aristo: Correct error code symbol * Aristo: Update error code symbols * Aristo: Code cosmetics/comments * Aristo: Fix hashify schedule calculator why: Had a tendency to stop early leaving an incomplete job
2024-02-01 21:27:48 +00:00
HikeBranchMissingEdge
Aristo db update foreground caching (#1605) * Fix vertex ID generator state handling for rocksdb backend why: * Key error in walk iterator * Needs to be loaded when opening the database * Use non-zero sub-table prefixes for rocksdb why: Handy for debugging * Fix error code for missing key on rocksdb backend why: Previously returned `VOID_HASH_KEY` rather than `GetKeyNotFound` * Explicitly copy vertex data between internal table and function/result argument why: Function argument or return reference may still refer to the same data object. * Updated error symbols why: Error symbol names for the hike module now start with the prefix `Hike`. * Write back modified branch node into local top layer cache why: With the backend available, the source of the branch node references might not be the top layer cache. So any change must be explicitely recorded.
2023-06-22 12:13:24 +01:00
HikeBranchTailEmpty
Core db+aristo fixes and tx handling updates (#2164) * Aristo: Rename journal related sources and functions why: Previously, the naming was hinged on the phrases `fifo`, `filter` etc. which reflect the inner workings of cascaded filters. This was unfortunate for reading/understanding the source code for actions where the focus is the journal as a whole. * Aristo: Fix buffer overflow (path length truncating error) * Aristo: Tighten `hikeUp()` stop check, update error code why: Detect dangling vertex links. These are legit with `snap` sync processing but not with regular processing. * Aristo: Raise assert in regular mode `merge()` at a dangling link/edge why: With `snap` sync processing, partial trees are ok and can be amended. Not so in regular mode. Previously there was only a debug message when a non-legit dangling edge was encountered. * Aristo: Make sure that vertices are copied before modification why: Otherwise vertices from lower layers might also be modified * Aristo: Fix relaxed mode for validity checker `check()` * Remove cruft * Aristo: Update API for transaction handling details: + Split `aristo_tx.nim` into sub-modules + Split `forkWith()` into `findTx()` + `forkTx()` + Removed `forkTop()`, `forkBase()` (now superseded by new `forkTx()`) * CoreDb+Aristo: Fix initialiser (missing methods)
2024-05-03 17:38:17 +00:00
HikeDanglingEdge
Aristo: Update unit test suite (#2002) * Aristo: Update unit test suite * Aristo/Kvt: Fix iterators why: Generic iterators were not properly updated after backend change * Aristo: Add sub-trie deletion functionality why: For storage tries linked to an account payload vertex ID, a the whole storage trie needs to be deleted with the account. * Aristo: Reserve vertex ID numbers for static custom state roots why: Static custom state roots may be controlled by an application, e.g. for a receipt or a transaction root. The `Aristo` functions are agnostic of what the static state roots are when different from the internal tree vertex ID 1. details; The `merge()` function applied to a non-static state root (assumed to be a storage root) will check the payload of an accounts leaf and mark its Merkle keys to be re-checked. * Aristo: Correct error code symbol * Aristo: Update error code symbols * Aristo: Code cosmetics/comments * Aristo: Fix hashify schedule calculator why: Had a tendency to stop early leaving an incomplete job
2024-02-01 21:27:48 +00:00
HikeEmptyPath
Core db update storage root management for sub tries (#1964) * Aristo: Re-phrase `LayerDelta` and `LayerFinal` as object references why: Avoids copying in some cases * Fix copyright header * Aristo: Verify `leafTie.root` function argument for `merge()` proc why: Zero root will lead to inconsistent DB entry * Aristo: Update failure condition for hash labels compiler `hashify()` why: Node need not be rejected as long as links are on the schedule. In that case, `redo[]` is to become `wff.base[]` at a later stage. This amends an earlier fix, part of #1952 by also testing against the target nodes of the `wff.base[]` sets. * Aristo: Add storage root glue record to `hashify()` schedule why: An account leaf node might refer to a non-resolvable storage root ID. Storage root node chains will end up at the storage root. So the link `storage-root->account-leaf` needs an extra item in the schedule. * Aristo: fix error code returned by `fetchPayload()` details: Final error code is implied by the error code form the `hikeUp()` function. * CoreDb: Discard `createOk` argument in API `getRoot()` function why: Not needed for the legacy DB. For the `Arsto` DB, a lazy approach is implemented where a stprage root node is created on-the-fly. * CoreDb: Prevent `$$` logging in some cases why: Logging the function `$$` is not useful when it is used for internal use, i.e. retrieving an an error text for logging. * CoreDb: Add `tryHashFn()` to API for pretty printing why: Pretty printing must not change the hashification status for the `Aristo` DB. So there is an independent API wrapper for getting the node hash which never updated the hashes. * CoreDb: Discard `update` argument in API `hash()` function why: When calling the API function `hash()`, the latest state is always wanted. For a version that uses the current state as-is without checking, the function `tryHash()` was added to the backend. * CoreDb: Update opaque vertex ID objects for the `Aristo` backend why: For `Aristo`, vID objects encapsulate a numeric `VertexID` referencing a vertex (rather than a node hash as used on the legacy backend.) For storage sub-tries, there might be no initial vertex known when the descriptor is created. So opaque vertex ID objects are supported without a valid `VertexID` which will be initalised on-the-fly when the first item is merged. * CoreDb: Add pretty printer for opaque vertex ID objects * Cosmetics, printing profiling data * CoreDb: Fix segfault in `Aristo` backend when creating MPT descriptor why: Missing initialisation error * CoreDb: Allow MPT to inherit shared context on `Aristo` backend why: Creates descriptors with different storage roots for the same shared `Aristo` DB descriptor. * Cosmetics, update diagnostic message items for `Aristo` backend * Fix Copyright year
2024-01-11 19:11:38 +00:00
HikeLeafUnexpected
Aristo: Update unit test suite (#2002) * Aristo: Update unit test suite * Aristo/Kvt: Fix iterators why: Generic iterators were not properly updated after backend change * Aristo: Add sub-trie deletion functionality why: For storage tries linked to an account payload vertex ID, a the whole storage trie needs to be deleted with the account. * Aristo: Reserve vertex ID numbers for static custom state roots why: Static custom state roots may be controlled by an application, e.g. for a receipt or a transaction root. The `Aristo` functions are agnostic of what the static state roots are when different from the internal tree vertex ID 1. details; The `merge()` function applied to a non-static state root (assumed to be a storage root) will check the payload of an accounts leaf and mark its Merkle keys to be re-checked. * Aristo: Correct error code symbol * Aristo: Update error code symbols * Aristo: Code cosmetics/comments * Aristo: Fix hashify schedule calculator why: Had a tendency to stop early leaving an incomplete job
2024-02-01 21:27:48 +00:00
HikeNoLegs
HikeRootMissing
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 12:47:47 +01:00
Experimental MP-trie (#1573) * Experimental MP-trie why: Deleting records is a infeasible with the current structure * Added vertex ID recycling management Todo: Provide some unit tests * DB layout update why: Main news is the separation of `Merkel` hashes into an extra table. details: The code fragments cover conversion between compact MPT records and Aristo DB records as well as some rudimentary cache handling for the `Merkel` hashes (i.e. the extra table entries.) todo: Add some simple unit test for the descriptor record (currently used for vertex ID management, only.) * Updated vertex ID recycling management details: added simple unit tests (mainly testing ABI) * docu update
2023-05-11 15:25:29 +01:00
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 12:47:47 +01:00
# Merge leaf `merge()`
merge: avoid hike allocations (#2472) hike allocations (and the garbage collection maintenance that follows) are responsible for some 10% of cpu time (not wall time!) at this point - this PR avoids them by stepping through the layers one step at a time, simplifying the code at the same time.
2024-07-11 13:26:46 +02:00
MergeHikeFailed # Ooops, internal error
Aristo uses pre classified tree types (#2385) * Remove unused `merge*()` functions (for production) details: Some functionality moved to test suite * Make sure that only `AccountData` leaf type is exactly used on VertexID(1) * clean up payload type * Provide dedicated functions for merging accounts and storage trees why: Storage trees are always linked to an account, so there is no need for an application to fiddle about (e.e. creating, re-cycling) with storage tree vertex IDs. * CoreDb: Disable tracer functionality why: Must be updated to accommodate new/changed `Aristo` functions. * CoreDb: Use new `mergeXXX()` functions why: Makes explicit vertex ID management obsolete for creating new storage trees. * Remove `mergePayload()` and other cruft from API, `aristo_merge`, etc. * clean up merge functions details: The merge implementation `mergePayloadImpl()` does not need to be super generic anymore as all the edge cases are covered by the specialised functions `mergeAccountPayload()`, `mergeGenericData()`, and `mergeStorageData()`. * No tracer available at the moment, so disable offending tests
2024-06-18 11:14:02 +00:00
MergeAccRootNotAccepted
MergeStoRootNotAccepted
leaf cache fixes (#2637) * Add missing leaf cache update when a leaf turns to a branch with two leaves (on merge) and vice versa (on delete) - this could lead to stale leaves being returned from the cache causing validation failures - it didn't happen because the leaf caches were not being used efficiently :) * Replace `seq` with `ArrayBuf` in `Hike` allowing it to become allocation-free - this PR also works around an inefficiency in nim in returning large types via a `var` parameter * Use the leaf cache instead of `getVtxRc` to fetch recent leaves - this makes the vertex cache more efficient at caching branches because fewer leaf requests pass through it.
2024-09-19 10:39:06 +02:00
MergeNoAction
Aristo uses pre classified tree types (#2385) * Remove unused `merge*()` functions (for production) details: Some functionality moved to test suite * Make sure that only `AccountData` leaf type is exactly used on VertexID(1) * clean up payload type * Provide dedicated functions for merging accounts and storage trees why: Storage trees are always linked to an account, so there is no need for an application to fiddle about (e.e. creating, re-cycling) with storage tree vertex IDs. * CoreDb: Disable tracer functionality why: Must be updated to accommodate new/changed `Aristo` functions. * CoreDb: Use new `mergeXXX()` functions why: Makes explicit vertex ID management obsolete for creating new storage trees. * Remove `mergePayload()` and other cruft from API, `aristo_merge`, etc. * clean up merge functions details: The merge implementation `mergePayloadImpl()` does not need to be super generic anymore as all the edge cases are covered by the specialised functions `mergeAccountPayload()`, `mergeGenericData()`, and `mergeStorageData()`. * No tracer available at the moment, so disable offending tests
2024-06-18 11:14:02 +00:00
MergeRootVidMissing
Update storage tree admin (#2419) * Tighten `CoreDb` API for accounts why: Apart from cruft, the way to fetch the accounts state root via a `CoreDbColRef` record was unnecessarily complicated. * Extend `CoreDb` API for accounts to cover storage tries why: In future, this will make the notion of column objects obsolete. Storage trees will then be indexed by the account address rather than the vertex ID equivalent like a `CoreDbColRef`. * Apply new/extended accounts API to ledger and tests details: This makes the `distinct_ledger` module obsolete * Remove column object constructors why: They were needed as an abstraction of MPT sub-trees including storage trees. Now, storage trees are handled by the account (e.g. via address) they belong to and all other trees can be identified by a constant well known vertex ID. So there is no need for column objects anymore. Still there are some left-over column object methods wnich will be removed next. * Remove `serialise()` and `PayloadRef` from default Aristo API why: Not needed. `PayloadRef` was used for unstructured/unknown payload formats (account or blob) and `serialise()` was used for decodng `PayloadRef`. Now it is known in advance what the payload looks like. * Added query function `hasStorageData()` whether a storage area exists why: Useful for supporting `slotStateEmpty()` of the `CoreDb` API * In the `Ledger` replace `storage.stateEmpty()` by `slotStateEmpty()` * On Aristo, hide the storage root/vertex ID in the `PayloadRef` why: The storage vertex ID is fully controlled by Aristo while the `AristoAccount` object is controlled by the application. With the storage root part of the `AristoAccount` object, there was a useless administrative burden to keep that storage root field up to date. * Remove cruft, update comments etc. * Update changed MPT access paradigms why: Fixes verified proxy tests * Fluffy cosmetics
2024-06-27 09:01:26 +00:00
MergeStoAccMissing
Core db and aristo maintenance update (#2014) * Aristo: Update error return code why: Failing of `Aristo` function `delete()` might fail because there is no such data item on the db. This must return a single error code as is done with `fetch()`. * Ledger: Better error handling why: The `expect()` clauses have been replaced by raising asserts indicating the error from the database backend. Also, `delete()` failures are legitimate if the item to delete does not exist. * Aristo: Delete function must always leave a label on DB for `hashify()` why: The `hashify()` uses the labels left bu `merge()` and `delete()` to compile (and optimise) a scheduler for subsequent hashing. Originally, the labels were not used for deleted entries and `delete()` still had some edge case where the deletion label was not properly handled. * Aristo: Update `hashify()` scheduler, remove buggy optimisation why: Was left over from version without virtual state roots which did not know about account payload leaf vertices referring to storage roots. * Aristo: Label storage trie account in `delete()` similar to `merge()` details; The `delete()` function applied to a non-static state root (assumed to be a storage root) will check the payload of an accounts leaf and mark its Merkle keys to be re-checked when runninh `hashify()` * Aristo: Clean up and re-org recycled vertex IDs in `hashify()` why: Re-organising the recycled vertex IDs list intends to reduce the size of the list. This list is organised as a LIFO (or stack.) By reorganising it in a way so that the least vertex ID numbers are on top, the list will be kept smaller as observed on some examples (less than 30%.) * CoreDb: Accept storage trie deletion requests in non-initialised state why: Due to lazy initialisation, the root vertex ID might not yet exist. So the `Aristo` database handlers would reject this call with an error and this condition needs to be handled by the API (which realises the lazy feature.) * Cosmetics & code massage, prettify logging * fix missing import
2024-02-08 16:32:16 +00:00
Update aristo journal functionality (#2155) * Aristo: Code cosmetics, e.g. update some CamelCase names * CoreDb+Aristo: Provide oldest known state root implied details: The Aristo journal allows to recover earlier but not all state roots. * Aristo: Fix journal backward index operator, e.g. `[^1]` * Aristo: Fix journal updater why: The `fifosStore()` store function slightly misinterpreted the update instructions when translation is to database `put()` functions. The effect was that the journal was ever growing due to stale entries which were never deleted. * CoreDb+Aristo: Provide utils for purging stale data from the KVT details: See earlier patch, not all state roots are available. This patch provides a mapping from some state root to a block number and allows to remove all KVT data related to a particular block number * Aristo+Kvt: Implement a clean up schedule for expired data in KVT why: For a single state ledger like `Aristo`, there is only a limited backlog of states. So KVT data (i.e. headers etc.) are cleaned up regularly * Fix copyright year
2024-04-26 13:43:52 +00:00
Aristo db fixes after storage slots dump tests added (#1595) * Fix missing Merkle key removal in `merge()` * Accept optional root hash argument in `hashify()` why: For importing a full database, there will be no proof data except the root key. So this can be used to check and set the root key in the database descriptor. also: Associate vertex ID to `hashify()` error return code * Added Aristo Trie traversal function why: * step along leaf vertices in sorted order * tree/trie consistency checks when debugging * Enabled storage slots test data for Aristo DB
2023-06-02 11:04:29 +01:00
# Neighbour vertex, tree traversal `nearbyRight()` and `nearbyLeft()`
NearbyBeyondRange
NearbyBranchError
NearbyDanglingLink
NearbyEmptyHike
NearbyFailed
NearbyLeafExpected
NearbyNestingTooDeep
NearbyPathTailUnexpected
NearbyUnexpectedVtx
Aristo db update delete functionality (#1621) * Fix missing branch checks in transcoder why: Symmetry problem. `Blobify()` allowed for encoding degenerate branch vertices while `Deblobify()` rejected decoding wrongly encoded data. * Update memory backend so that it rejects storing bogus vertices. why: Error behaviour made similar to the rocks DB backend. * Make sure that leaf vertex IDs are not repurposed why: This makes it easier to record leaf node changes * Update error return code for next()/right() traversal why: Returning offending vertex ID (besides error code) helps debugging * Update Merkle hasher for deleted nodes why: Not implemented, yet also: Provide cache & backend consistency check functions. This was partly re-implemented from `hashifyCheck()` * Simplify some unit tests * Fix delete function why: Was conceptually wrong
2023-06-30 23:22:33 +01:00
NearbyVidInvalid
Aristo db fixes after storage slots dump tests added (#1595) * Fix missing Merkle key removal in `merge()` * Accept optional root hash argument in `hashify()` why: For importing a full database, there will be no proof data except the root key. So this can be used to check and set the root key in the database descriptor. also: Associate vertex ID to `hashify()` error return code * Added Aristo Trie traversal function why: * step along leaf vertices in sorted order * tree/trie consistency checks when debugging * Enabled storage slots test data for Aristo DB
2023-06-02 11:04:29 +01:00
Core db use differential tx layers for aristo and kvt (#1949) * Fix kvt headers * Provide differential layers for KVT transaction stack why: Significant performance improvement * Provide abstraction layer for database top cache layer why: This will eventually implemented as a differential database layers or transaction layers. The latter is needed to improve performance. behavioural changes: Zero vertex and keys (i.e. delete requests) are not optimised out until the last layer is written to the database. * Provide differential layers for Aristo transaction stack why: Significant performance improvement
2023-12-19 12:39:23 +00:00
Aristo uses pre classified tree types cont2 (#2397) * Provide dedicated functions for fetching accounts and storage trees why: Different prototypes for each class `account`, `generic` and `storage`. * Remove `fetchPayload()` and other cruft from API, `aristo_fetch`, etc. * Fix typos, debugging left overs, comments
2024-06-19 12:40:00 +00:00
# Path/nibble/key conversions in `aisto_path.nim`
PathExpected64Nibbles
PathAtMost64Nibbles
PathExpectedLeaf
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 14:26:25 +01:00
Aristo db allow shorter than 64 nibbles path keys (#1864) * Aristo: Single `FetchPathNotFound` error in `fetchXxx()` and `hasPath()` why: Missing path hike returns too many detailed reasons why it failed which becomes cumbersome to handle. also: Renamed `contains()` => `hasPath()` which disables the `in` operator on non-boolean `contains()` functions * Kvt: Renamed `contains()` => `hasKey()` why: which disables the `in` operator on non-boolean `contains()` functions * Aristo: Generalising `HashID` by variable length `PathID` why: There are cases when the `Aristo` database is to be used with shorter than 64 nibbles keys when handling transactions indexes with sequence IDs. caveat: This patch only works reliable for full length `PathID` values. Tests for shorter `PathID` values are currently missing.
2023-10-27 22:36:51 +01:00
Provide partial tree support for preloading tests (#2536) * Implement partial trees why: This is currently needed for unit tests to pre-load the database with test data similar to `proof` node pre-load. The basic features for `snap-sync` boundary proofs are available as well for future use. What is missing is the final proof verification and a complete storage data load/merge function (stub is available.) * Cosmetics, clean up
2024-07-29 20:15:17 +00:00
# Part/proof node errors
PartArgNotGenericRoot
Added portal proof nodes generation functionality (#2539) * Extracted `test_tx.testTxMergeProofAndKvpList()` => separate file * Fix serialiser why: Typo lead to duplicate rlp-encoded nodes in chain * Remove cruft * Implemnt portal proof nodes generators `partXxxTwig()` * Add unit test for portal proof nodes generator `partAccountTwig()` * Cosmetics * Simplify serialiser return code format * Fix proof generator for extension nodes why: Code was simply bonkers, not detected before the unit tests were adapted to check for just this. * Implemented portal proof nodes verifier `partUntwig()` * Cosmetics * Fix `testutp` cli poblem
2024-08-06 11:29:26 +00:00
PartArgNotInCore
Revive json tracer unit tests (#2538) * Some `Aristo` clean-ups/updates * Re-implemented core-db tracer functionality * Rename nimbus tracer `no-tracer.nim` => `tracer.nim` why: Restore original name for easy diff tracking with upcoming update * Update nimbus tracer using new core-db tracer functionality * Updating json tracer unit tests * Enable json tracer unit tests
2024-08-01 10:41:20 +00:00
PartArgRootAlreadyOnDatabase
Added portal proof nodes generation functionality (#2539) * Extracted `test_tx.testTxMergeProofAndKvpList()` => separate file * Fix serialiser why: Typo lead to duplicate rlp-encoded nodes in chain * Remove cruft * Implemnt portal proof nodes generators `partXxxTwig()` * Add unit test for portal proof nodes generator `partAccountTwig()` * Cosmetics * Simplify serialiser return code format * Fix proof generator for extension nodes why: Code was simply bonkers, not detected before the unit tests were adapted to check for just this. * Implemented portal proof nodes verifier `partUntwig()` * Cosmetics * Fix `testutp` cli poblem
2024-08-06 11:29:26 +00:00
PartArgRootAlreadyUsed
Provide partial tree support for preloading tests (#2536) * Implement partial trees why: This is currently needed for unit tests to pre-load the database with test data similar to `proof` node pre-load. The basic features for `snap-sync` boundary proofs are available as well for future use. What is missing is the final proof verification and a complete storage data load/merge function (stub is available.) * Cosmetics, clean up
2024-07-29 20:15:17 +00:00
PartChkChangedKeyNotInKeyTab
PartChkChangedVtxMissing
PartChkCoreKeyLookupFailed
PartChkCoreRVidLookupFailed
PartChkCoreVidLookupFailed
PartChkCoreVtxMissing
PartChkKeyTabCoreKeyMissing
PartChkKeyTabRootMissing
PartChkPerimeterVtxMustNotExist
PartChkVidKeyTabKeyMismatch
PartChkVidKeyTabLengthsDiffer
PartChkVidTabCoreRootMissing
PartChkVidTabVidMissing
Added portal proof nodes generation functionality (#2539) * Extracted `test_tx.testTxMergeProofAndKvpList()` => separate file * Fix serialiser why: Typo lead to duplicate rlp-encoded nodes in chain * Remove cruft * Implemnt portal proof nodes generators `partXxxTwig()` * Add unit test for portal proof nodes generator `partAccountTwig()` * Cosmetics * Simplify serialiser return code format * Fix proof generator for extension nodes why: Code was simply bonkers, not detected before the unit tests were adapted to check for just this. * Implemented portal proof nodes verifier `partUntwig()` * Cosmetics * Fix `testutp` cli poblem
2024-08-06 11:29:26 +00:00
PartChnBranchPathExhausted
Add portal proof functionality for non-existing keys/paths (#2610)
2024-09-11 09:39:45 +00:00
PartChnBranchVoidEdge
Added portal proof nodes generation functionality (#2539) * Extracted `test_tx.testTxMergeProofAndKvpList()` => separate file * Fix serialiser why: Typo lead to duplicate rlp-encoded nodes in chain * Remove cruft * Implemnt portal proof nodes generators `partXxxTwig()` * Add unit test for portal proof nodes generator `partAccountTwig()` * Cosmetics * Simplify serialiser return code format * Fix proof generator for extension nodes why: Code was simply bonkers, not detected before the unit tests were adapted to check for just this. * Implemented portal proof nodes verifier `partUntwig()` * Cosmetics * Fix `testutp` cli poblem
2024-08-06 11:29:26 +00:00
PartChnExtPfxMismatch
PartChnLeafPathMismatch
PartChnNodeConvError
Provide partial tree support for preloading tests (#2536) * Implement partial trees why: This is currently needed for unit tests to pre-load the database with test data similar to `proof` node pre-load. The basic features for `snap-sync` boundary proofs are available as well for future use. What is missing is the final proof verification and a complete storage data load/merge function (stub is available.) * Cosmetics, clean up
2024-07-29 20:15:17 +00:00
PartCtxNotAvailable
PartCtxStaleDescriptor
PartExtVtxExistsAlready
PartExtVtxHasVanished
PartExtVtxWasModified
PartGarbledExtsInProofs
PartMissingUplinkInternalError
PartNoMoreRootVidsLeft
PartPayloadAccRejected
PartPayloadAccRequired
PartRlp1r4ListEntries
PartRlp2Or17ListEntries
PartRlpBlobExpected
PartRlpBranchHashKeyExpected
PartRlpEmptyBlobExpected
PartRlpExtHashKeyExpected
PartRlpNodeException
PartRlpNonEmptyBlobExpected
PartRlpPayloadException
PartRootKeysDontMatch
PartRootVidsDontMatch
Added portal proof nodes generation functionality (#2539) * Extracted `test_tx.testTxMergeProofAndKvpList()` => separate file * Fix serialiser why: Typo lead to duplicate rlp-encoded nodes in chain * Remove cruft * Implemnt portal proof nodes generators `partXxxTwig()` * Add unit test for portal proof nodes generator `partAccountTwig()` * Cosmetics * Simplify serialiser return code format * Fix proof generator for extension nodes why: Code was simply bonkers, not detected before the unit tests were adapted to check for just this. * Implemented portal proof nodes verifier `partUntwig()` * Cosmetics * Fix `testutp` cli poblem
2024-08-06 11:29:26 +00:00
PartTrkEmptyPath
PartTrkFollowUpKeyMismatch
PartTrkGarbledNode
PartTrkLeafPfxMismatch
PartTrkLinkExpected
PartTrkPayloadMismatch
PartTrkRlpError
Provide partial tree support for preloading tests (#2536) * Implement partial trees why: This is currently needed for unit tests to pre-load the database with test data similar to `proof` node pre-load. The basic features for `snap-sync` boundary proofs are available as well for future use. What is missing is the final proof verification and a complete storage data load/merge function (stub is available.) * Cosmetics, clean up
2024-07-29 20:15:17 +00:00
PartVtxSlotWasModified
PartVtxSlotWasNotModified
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 14:26:25 +01:00
# RocksDB backend
RdbBeCantCreateTmpDir
Aristo use rocksdb cf instead of key pfx (#2332) * Use RocksDb column families instead of a prefixed single column why: Better performance * Use structural objects `VertexRef` and `HashKey` in LRU cache for RocksDb why: Avoids repeated de/serialisation
2024-06-10 12:04:22 +00:00
RdbBeDriverDelAdmError
RdbBeDriverDelKeyError
RdbBeDriverDelVtxError
RdbBeDriverGetAdmError
RdbBeDriverGetKeyError
RdbBeDriverGetVtxError
Aristo update rocksdb backend drivers (#2135) * Aristo+RocksDB: Update backend drivers why: RocksDB update allows use some of the newly provided methods which were previously implemented by using the very C backend (for the lack of NIM methods.) * Aristo+RocksDB: Simplify drivers wrapper * Kvt: Update backend drivers and wrappers similar to `Aristo` * Aristo+Kvm: Use column families for RocksDB * Aristo+MemoryDB: Code cosmetics * Aristo: Provide guest column family for export why: So `Kvt` can piggyback on `Aristo` so there avoiding to run a second DBMS system in parallel. * Kvt: Provide import mechanism for RoksDB guest column family why: So `Kvt` can piggyback on `Aristo` so there avoiding to run a second DBMS system in parallel. * CoreDb+Aristo: Run persistent `Kvt` DB piggybacked on `Aristo` why: Avoiding to run two DBMS systems in parallel. * Fix copyright year * Ditto
2024-04-16 20:39:11 +00:00
RdbBeDriverGuestError
Aristo use rocksdb cf instead of key pfx (#2332) * Use RocksDb column families instead of a prefixed single column why: Better performance * Use structural objects `VertexRef` and `HashKey` in LRU cache for RocksDb why: Avoids repeated de/serialisation
2024-06-10 12:04:22 +00:00
RdbBeDriverPutAdmError
RdbBeDriverPutKeyError
Triggered write event for kvt (#2351) * bump rockdb * Rename `KVT` objects related to filters according to `Aristo` naming details: filter* => delta* roFilter => balancer * Compulsory error handling if `persistent()` fails * Add return code to `reCentre()` why: Might eventually fail if re-centring is blocked. Some logic will be added in subsequent patch sets. * Add column families from earlier session to rocksdb in opening procedure why: All previously used CFs must be declared when re-opening an existing database. * Update `init()` and add rocksdb `reinit()` methods for changing parameters why: Opening a set column families (with different open options) must span at least the ones that are already on disk. * Provide write-trigger-event interface into `Aristo` backend why: This allows to save data from a guest application (think `KVT`) to get synced with the write cycle so the guest and `Aristo` save all atomically. * Use `KVT` with new column family interface from `Aristo` * Remove obsolete guest interface * Implement `KVT` piggyback on `Aristo` backend * CoreDb: Add separate `KVT`/`Aristo` backend mode for debugging * Remove `rocks_db` import from `persist()` function why: Some systems (i.p `fluffy` and friends) use the `Aristo` memory backend emulation and do not link against rocksdb when building the application. So this should fix that problem.
2024-06-13 18:15:11 +00:00
RdbBeDriverPutVtxError
Aristo update rocksdb backend drivers (#2135) * Aristo+RocksDB: Update backend drivers why: RocksDB update allows use some of the newly provided methods which were previously implemented by using the very C backend (for the lack of NIM methods.) * Aristo+RocksDB: Simplify drivers wrapper * Kvt: Update backend drivers and wrappers similar to `Aristo` * Aristo+Kvm: Use column families for RocksDB * Aristo+MemoryDB: Code cosmetics * Aristo: Provide guest column family for export why: So `Kvt` can piggyback on `Aristo` so there avoiding to run a second DBMS system in parallel. * Kvt: Provide import mechanism for RoksDB guest column family why: So `Kvt` can piggyback on `Aristo` so there avoiding to run a second DBMS system in parallel. * CoreDb+Aristo: Run persistent `Kvt` DB piggybacked on `Aristo` why: Avoiding to run two DBMS systems in parallel. * Fix copyright year * Ditto
2024-04-16 20:39:11 +00:00
RdbBeDriverWriteError
Triggered write event for kvt (#2351) * bump rockdb * Rename `KVT` objects related to filters according to `Aristo` naming details: filter* => delta* roFilter => balancer * Compulsory error handling if `persistent()` fails * Add return code to `reCentre()` why: Might eventually fail if re-centring is blocked. Some logic will be added in subsequent patch sets. * Add column families from earlier session to rocksdb in opening procedure why: All previously used CFs must be declared when re-opening an existing database. * Update `init()` and add rocksdb `reinit()` methods for changing parameters why: Opening a set column families (with different open options) must span at least the ones that are already on disk. * Provide write-trigger-event interface into `Aristo` backend why: This allows to save data from a guest application (think `KVT`) to get synced with the write cycle so the guest and `Aristo` save all atomically. * Use `KVT` with new column family interface from `Aristo` * Remove obsolete guest interface * Implement `KVT` piggyback on `Aristo` backend * CoreDb: Add separate `KVT`/`Aristo` backend mode for debugging * Remove `rocks_db` import from `persist()` function why: Some systems (i.p `fluffy` and friends) use the `Aristo` memory backend emulation and do not link against rocksdb when building the application. So this should fix that problem.
2024-06-13 18:15:11 +00:00
RdbBeTypeUnsupported
RdbBeWrSessionUnfinished
RdbBeWrTriggerActiveAlready
RdbBeWrTriggerNilFn
RdbGuestInstanceAborted
Aristo db update for short nodes key edge cases (#1887) * Aristo: Provide key-value list signature calculator detail: Simple wrappers around `Aristo` core functionality * Update new API for `CoreDb` details: + Renamed new API functions `contains()` => `hasKey()` or `hasPath()` which disables the `in` operator on non-boolean `contains()` functions + The functions `get()` and `fetch()` always return a not-found error if there is no item, available. The new functions `getOrEmpty()` and `mergeOrEmpty()` return an an empty `Blob` if there is no such key found. * Rewrite `core_apps.nim` using new API from `CoreDb` * Use `Aristo` functionality for calculating Merkle signatures details: For debugging, the `VerifyAristoForMerkleRootCalc` can be set so that `Aristo` results will be verified against the legacy versions. * Provide general interface for Merkle signing key-value tables details: Export `Aristo` wrappers * Activate `CoreDb` tests why: Now, API seems to be stable enough for general tests. * Update `toHex()` usage why: Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join` * Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify` why: + Different modules for different purposes + `aristo_serialise`: RLP encoding/decoding + `aristo_blobify`: Aristo database encoding/decoding * Compacted representation of small nodes' links instead of Keccak hashes why: Ethereum MPTs use Keccak hashes as node links if the size of an RLP encoded node is at least 32 bytes. Otherwise, the RLP encoded node value is used as a pseudo node link (rather than a hash.) Such a node is nor stored on key-value database. Rather the RLP encoded node value is stored instead of a lode link in a parent node instead. Only for the root hash, the top level node is always referred to by the hash. This feature needed an abstraction of the `HashKey` object which is now either a hash or a blob of length at most 31 bytes. This leaves two ways of representing an empty/void `HashKey` type, either as an empty blob of zero length, or the hash of an empty blob. * Update `CoreDb` interface (mainly reducing logger noise) * Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
RdbHashKeyExpected
Aristo db supporting forest and layered tx architecture (#1598) * Exclude some storage tests why: These test running on external dumps slipped through. The particular dumps were reported earlier as somehow dodgy. This was changed in `#1457` but having a second look, the change on hexary_interpolate.nim(350) might be incorrect. * Redesign `Aristo DB` descriptor for transaction based layers why: Previous descriptor layout made it cumbersome to push/pop database delta layers. The new architecture keeps each layer with the full delta set relative to the database backend. * Keep root ID as part of the `Patricia Trie` leaf path why; That way, forests are supported
2023-06-09 12:17:37 +01:00
Aristo uses pre classified tree types cont2 (#2397) * Provide dedicated functions for fetching accounts and storage trees why: Different prototypes for each class `account`, `generic` and `storage`. * Remove `fetchPayload()` and other cruft from API, `aristo_fetch`, etc. * Fix typos, debugging left overs, comments
2024-06-19 12:40:00 +00:00
Add items to `Aristo Trie` database (#1586) details: 1. Merging a leaf vertex merges a `Patricia Trie` path (while adding/modiying vertices) and adds a leaf node with payload 2. Merging a Merkel node merges a single vertex to the `Patricia Trie` and registers merkel hashes 3. Action 2 can be used before action 1 in order to construct a Merkel proof as required for handling `snap/1` data. 4. Unit tests show that action 3 is benign for now :)
2023-05-30 12:47:47 +01:00
# End
Reference in New Issue Copy Permalink