status-im
/
nimbus-eth1
mirror of https://github.com/status-im/nimbus-eth1.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth1/nimbus/db/aristo/aristo_init/memory_only.nim

106 lines
3.6 KiB
Nim
Raw Normal View History

Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
# nimbus-eth1
Core db update storage root management for sub tries (#1964) * Aristo: Re-phrase `LayerDelta` and `LayerFinal` as object references why: Avoids copying in some cases * Fix copyright header * Aristo: Verify `leafTie.root` function argument for `merge()` proc why: Zero root will lead to inconsistent DB entry * Aristo: Update failure condition for hash labels compiler `hashify()` why: Node need not be rejected as long as links are on the schedule. In that case, `redo[]` is to become `wff.base[]` at a later stage. This amends an earlier fix, part of #1952 by also testing against the target nodes of the `wff.base[]` sets. * Aristo: Add storage root glue record to `hashify()` schedule why: An account leaf node might refer to a non-resolvable storage root ID. Storage root node chains will end up at the storage root. So the link `storage-root->account-leaf` needs an extra item in the schedule. * Aristo: fix error code returned by `fetchPayload()` details: Final error code is implied by the error code form the `hikeUp()` function. * CoreDb: Discard `createOk` argument in API `getRoot()` function why: Not needed for the legacy DB. For the `Arsto` DB, a lazy approach is implemented where a stprage root node is created on-the-fly. * CoreDb: Prevent `$$` logging in some cases why: Logging the function `$$` is not useful when it is used for internal use, i.e. retrieving an an error text for logging. * CoreDb: Add `tryHashFn()` to API for pretty printing why: Pretty printing must not change the hashification status for the `Aristo` DB. So there is an independent API wrapper for getting the node hash which never updated the hashes. * CoreDb: Discard `update` argument in API `hash()` function why: When calling the API function `hash()`, the latest state is always wanted. For a version that uses the current state as-is without checking, the function `tryHash()` was added to the backend. * CoreDb: Update opaque vertex ID objects for the `Aristo` backend why: For `Aristo`, vID objects encapsulate a numeric `VertexID` referencing a vertex (rather than a node hash as used on the legacy backend.) For storage sub-tries, there might be no initial vertex known when the descriptor is created. So opaque vertex ID objects are supported without a valid `VertexID` which will be initalised on-the-fly when the first item is merged. * CoreDb: Add pretty printer for opaque vertex ID objects * Cosmetics, printing profiling data * CoreDb: Fix segfault in `Aristo` backend when creating MPT descriptor why: Missing initialisation error * CoreDb: Allow MPT to inherit shared context on `Aristo` backend why: Creates descriptors with different storage roots for the same shared `Aristo` DB descriptor. * Cosmetics, update diagnostic message items for `Aristo` backend * Fix Copyright year
2024-01-11 19:11:38 +00:00
# Copyright (c) 2023-2024 Status Research & Development GmbH
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or
# http://www.apache.org/licenses/LICENSE-2.0)
# * MIT license ([LICENSE-MIT](LICENSE-MIT) or
# http://opensource.org/licenses/MIT)
# at your option. This file may not be copied, modified, or distributed
# except according to those terms.
## Non persistent constructors for Aristo DB
## =========================================
##
{.push raises: [].}
import
../aristo_desc,
Aristo db implement filter storage scheduler (#1713) * Rename FilterID => QueueID why: The current usage does not identify a particular filter but uses it as storage tag to manage it on the database (to be organised in a set of FIFOs or queues.) * Split `aristo_filter` source into sub-files why: Make space for filter management API * Store filter queue IDs in pairs on the backend why: Any pair will will describe a FIFO accessed by bottom/top IDs * Reorg some source file names why: The "aristo_" prefix for make local/private files is tedious to use, so removed. * Implement filter slot scheduler details: Filters will be stored on the database on cascaded FIFOs. When a FIFO queue is full, some filter items are bundled together and stored on the next FIFO.
2023-08-25 22:53:59 +00:00
../aristo_desc/desc_backend,
"."/[init_common, memory_db]
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
Aristo db use filter betw backend and tx cache (#1678) * Provide deep copy for each transaction layer why: Localising changes. Selective deep copy was just overlooked. * Generalise vertex ID generator state reorg function `vidReorg()` why: makes it somewhat easier to handle when saving layers. * Provide dummy back end descriptor `NoneBackendRef` * Optional read-only filter between backend and transaction cache why: Some staging area for accumulating changes to the backend DB. This will eventually be an access layer for emulating a backend with multiple/historic state roots. * Re-factor `persistent()` with filter between backend/tx-cache => `stow()` why: The filter provides an abstraction from the physically stored data on disk. So, there can be several MPT instances using the same disk data with different state roots. Of course, all the MPT instances should not differ too much for practical reasons :). TODO: Filter administration tools need to be provided.
2023-08-10 20:01:28 +00:00
type
Aristo db integrate hashify into tx (#1679) * Renamed type `NoneBackendRef` => `VoidBackendRef` * Clarify names: `BE=filter+backend` and `UBE=backend (unfiltered)` why: Most functions used full names as `getVtxUnfilteredBackend()` or `getKeyBackend()`. After defining abbreviations (and its meaning) it seems easier to use `getVtxUBE()` and `getKeyBE()`. * Integrate `hashify()` process into transaction logic why: Is now transparent unless explicitly controlled. details: Cache changes imply setting a `dirty` flag which in turn triggers `hashify()` processing in transaction and `pack()` directives. * Removed `aristo_tx.exec()` directive why: Inconsistent implementation, functionality will be provided with a different paradigm.
2023-08-11 17:23:57 +00:00
VoidBackendRef* = ref object of TypedBackendRef
Aristo db api extensions for use as core db backend (#1754) * Update docu * Update Aristo/Kvt constructor prototype why: Previous version used an `enum` value to indicate what backend is to be used. This was replaced by using the backend object type. * Rewrite `hikeUp()` return code into `Result[Hike,(Hike,AristoError)]` why: Better code maintenance. Previously, the `Hike` object was returned. It had an internal error field so partial success was also available on a failure. This error field has been removed. * Use `openArray[byte]` rather than `Blob` in functions prototypes * Provide synchronised multi instance transactions why: The `CoreDB` object was geared towards the legacy DB which used a single transaction for the key-value backend DB. Different state roots are provided by the backend database, so all instances work directly on the same backend. Aristo db instances have different in-memory mappings (aka different state roots) and the transactions are on top of there mappings. So each instance might run different transactions. Multi instance transactions are a compromise to converge towards the legacy behaviour. The synchronised transactions span over all instances available at the time when base transaction was opened. Instances created later are unaffected. * Provide key-value pair database iterator why: Needed in `CoreDB` for `replicate()` emulation also: Some update of internal code * Extend API (i.e. prototype variants) why: Needed for `CoreDB` geared towards the legacy backend which has a more basic API than Aristo.
2023-09-15 15:23:53 +00:00
## Dummy descriptor type, used as `nil` reference
MemOnlyBackend* = VoidBackendRef|MemBackendRef
Aristo db use filter betw backend and tx cache (#1678) * Provide deep copy for each transaction layer why: Localising changes. Selective deep copy was just overlooked. * Generalise vertex ID generator state reorg function `vidReorg()` why: makes it somewhat easier to handle when saving layers. * Provide dummy back end descriptor `NoneBackendRef` * Optional read-only filter between backend and transaction cache why: Some staging area for accumulating changes to the backend DB. This will eventually be an access layer for emulating a backend with multiple/historic state roots. * Re-factor `persistent()` with filter between backend/tx-cache => `stow()` why: The filter provides an abstraction from the physically stored data on disk. So, there can be several MPT instances using the same disk data with different state roots. Of course, all the MPT instances should not differ too much for practical reasons :). TODO: Filter administration tools need to be provided.
2023-08-10 20:01:28 +00:00
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
export
Aristo db implement filter serialisation for storage (#1695) * Remove concept of empty/blind filters why: Not needed. A non-existent filter is is coded as a nil reference. * Slightly generalised backend iterators why: * VertexID as key for the ID generator state makes no sense * there will be more tables addressed by non-VertexID keys * Store serialised/blobified vertices on memory backend why: This is more in line with the RocksDB backend so more appropriate for testing when comparing behaviour. For a speedy memory database, a backend-less variant should be used. * Drop the `Aristo` prefix from names `AristoLayerRef`, etc. * Suppress compiler warning why: duplicate imports * Add filter serialisation transcoder why: Will be used as storage format
2023-08-18 19:46:55 +00:00
BackendType,
Aristo update rocksdb backend drivers (#2135) * Aristo+RocksDB: Update backend drivers why: RocksDB update allows use some of the newly provided methods which were previously implemented by using the very C backend (for the lack of NIM methods.) * Aristo+RocksDB: Simplify drivers wrapper * Kvt: Update backend drivers and wrappers similar to `Aristo` * Aristo+Kvm: Use column families for RocksDB * Aristo+MemoryDB: Code cosmetics * Aristo: Provide guest column family for export why: So `Kvt` can piggyback on `Aristo` so there avoiding to run a second DBMS system in parallel. * Kvt: Provide import mechanism for RoksDB guest column family why: So `Kvt` can piggyback on `Aristo` so there avoiding to run a second DBMS system in parallel. * CoreDb+Aristo: Run persistent `Kvt` DB piggybacked on `Aristo` why: Avoiding to run two DBMS systems in parallel. * Fix copyright year * Ditto
2024-04-16 20:39:11 +00:00
GuestDbRef,
Aristo cull journal related stuff (#2288) * Remove all journal related stuff * Refactor function names journal*() => delta*(), filter*() => delta*() * remove `trg` fileld from `FilterRef` why: Same as `kMap[$1]` * Re-type FilterRef.src as `HashKey` why: So it is directly comparable to `kMap[$1]` * Moved `vGen[]` field from `LayerFinalRef` to `LayerDeltaRef` why: Then a separate `FilterRef` type is not needed, anymore * Rename `roFilter` field in `AristoDbRef` => `balancer` why: New name more appropriate. * Replace `FilterRef` by `LayerDeltaRef` type why: This allows to avoid copying into the `balancer` (see next patch set) most of the time. Typically, only one instance is running on the backend and the `balancer` is only used as a stage before saving data. * Refactor way how to store data persistently why: Avoid useless copy when staging `top` layer for persistently saving to backend. * Fix copyright header?
2024-06-03 20:10:35 +00:00
MemBackendRef
Aristo db fixes n updates needed for filter fifo (#1728) * Set scheduler state as part of the backend descriptor details: Moved type definitions `QidLayoutRef` and `QidSchedRef` to `desc_structural.nim` so that it shares the same folder as `desc_backend.nim` * Automatic filter queue table initialisation in backend details: Scheduler can be tweaked or completely disabled * Updated backend unit tests details: + some code clean up/beautification, reads better now + disabled persistent filters so that there is no automated filter management which will be implemented next * Prettify/update unit tests source code details: Mostly replacing the `check()` paradigm by `xCheck()` * Somewhat simplified backend type management why: Backend objects are labelled with a `BackendType` symbol where the `BackendVoid` label is implicitly assumed for a `nil` backend object reference. To make it easier, a `kind()` function is used now applicable to `nil` references as well. * Fix DB storage layout for filter objects why: Need to store the filter ID with the object * Implement reverse [] index on fifo why: An integer index argument on `[]` retrieves the QueueID (label) of the fifo item while a QueueID argument on `[]` retrieves the index (so it is inverse to the former variant). * Provide iterator over filters as fifo why: This iterator goes along the cascased fifo structure (i.e. in historical order)
2023-09-05 13:57:20 +00:00
Aristo db api extensions for use as core db backend (#1754) * Update docu * Update Aristo/Kvt constructor prototype why: Previous version used an `enum` value to indicate what backend is to be used. This was replaced by using the backend object type. * Rewrite `hikeUp()` return code into `Result[Hike,(Hike,AristoError)]` why: Better code maintenance. Previously, the `Hike` object was returned. It had an internal error field so partial success was also available on a failure. This error field has been removed. * Use `openArray[byte]` rather than `Blob` in functions prototypes * Provide synchronised multi instance transactions why: The `CoreDB` object was geared towards the legacy DB which used a single transaction for the key-value backend DB. Different state roots are provided by the backend database, so all instances work directly on the same backend. Aristo db instances have different in-memory mappings (aka different state roots) and the transactions are on top of there mappings. So each instance might run different transactions. Multi instance transactions are a compromise to converge towards the legacy behaviour. The synchronised transactions span over all instances available at the time when base transaction was opened. Instances created later are unaffected. * Provide key-value pair database iterator why: Needed in `CoreDB` for `replicate()` emulation also: Some update of internal code * Extend API (i.e. prototype variants) why: Needed for `CoreDB` geared towards the legacy backend which has a more basic API than Aristo.
2023-09-15 15:23:53 +00:00
# ------------------------------------------------------------------------------
# Public helpers
# ------------------------------------------------------------------------------
proc kind*(
be: BackendRef;
): BackendType =
## Retrieves the backend type symbol for a `be` backend database argument
## where `BackendVoid` is returned for the`nil` backend.
if be.isNil:
BackendVoid
else:
be.TypedBackendRef.beKind
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
# ------------------------------------------------------------------------------
# Public database constuctors, destructor
# ------------------------------------------------------------------------------
Core db update providing additional results code interface (#1776) * Split `core_db/base.nim` into several sources * Rename `core_db/legacy.nim` => `core_db/legacy_db.nim` * Update `CoreDb` API, dual methods returning `Result[]` or plain value detail: Plain value methods implemet the legacy API, they defect on error results * Redesign `CoreDB` direct backend access why: Made the `backend` directive integral part of the API * Discontinue providing unused or otherwise available functions details: + setTransactionID() removed, not used and not easily replicable in Aristo + maybeGet() removed, available via direct backend access + newPhk() removed, never used & was experimental anyway * Update/reorg backend API why: + Added error print function `$$()` + General descriptor completion (and optional validation) via `bless()` * Update `Aristo`/`Kvt` exception handling why: Avoid `CatchableError` exceptions, rather pass them as error code where appropriate. * More `CoreDB` compliant `Aristo` and `Kvt` methods details: + Providing functions like `contains()`, `getVtxRc()` (returns `Result[]`). + Additional error code: `NotImplemented` * Rewrite/reorg of Aristo DB constructor why: Previously used global object `DefaultQidLayoutRef` as default initialiser. This object was created at compile time which lead to non-gc safe functions. * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/aristo/aristo_transcode.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> --------- Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-09-26 09:21:13 +00:00
proc init*(
T: type AristoDbRef; # Target type
B: type MemBackendRef; # Backend type
): T =
## Memory backend constructor.
##
## If the `qidLayout` argument is set `QidLayoutRef(nil)`, the a backend
## database will not provide filter history management. Providing a different
## scheduler layout shoud be used with care as table access with different
## layouts might render the filter history data unmanageable.
##
when B is MemBackendRef:
Aristo cull journal related stuff (#2288) * Remove all journal related stuff * Refactor function names journal*() => delta*(), filter*() => delta*() * remove `trg` fileld from `FilterRef` why: Same as `kMap[$1]` * Re-type FilterRef.src as `HashKey` why: So it is directly comparable to `kMap[$1]` * Moved `vGen[]` field from `LayerFinalRef` to `LayerDeltaRef` why: Then a separate `FilterRef` type is not needed, anymore * Rename `roFilter` field in `AristoDbRef` => `balancer` why: New name more appropriate. * Replace `FilterRef` by `LayerDeltaRef` type why: This allows to avoid copying into the `balancer` (see next patch set) most of the time. Typically, only one instance is running on the backend and the `balancer` is only used as a stage before saving data. * Refactor way how to store data persistently why: Avoid useless copy when staging `top` layer for persistently saving to backend. * Fix copyright header?
2024-06-03 20:10:35 +00:00
AristoDbRef(top: LayerRef.init(), backend: memoryBackend())
Core db update providing additional results code interface (#1776) * Split `core_db/base.nim` into several sources * Rename `core_db/legacy.nim` => `core_db/legacy_db.nim` * Update `CoreDb` API, dual methods returning `Result[]` or plain value detail: Plain value methods implemet the legacy API, they defect on error results * Redesign `CoreDB` direct backend access why: Made the `backend` directive integral part of the API * Discontinue providing unused or otherwise available functions details: + setTransactionID() removed, not used and not easily replicable in Aristo + maybeGet() removed, available via direct backend access + newPhk() removed, never used & was experimental anyway * Update/reorg backend API why: + Added error print function `$$()` + General descriptor completion (and optional validation) via `bless()` * Update `Aristo`/`Kvt` exception handling why: Avoid `CatchableError` exceptions, rather pass them as error code where appropriate. * More `CoreDB` compliant `Aristo` and `Kvt` methods details: + Providing functions like `contains()`, `getVtxRc()` (returns `Result[]`). + Additional error code: `NotImplemented` * Rewrite/reorg of Aristo DB constructor why: Previously used global object `DefaultQidLayoutRef` as default initialiser. This object was created at compile time which lead to non-gc safe functions. * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/aristo/aristo_transcode.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> * Update nimbus/db/core_db/legacy_db.nim Co-authored-by: Kim De Mey <kim.demey@gmail.com> --------- Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-09-26 09:21:13 +00:00
Aristo db api extensions for use as core db backend (#1754) * Update docu * Update Aristo/Kvt constructor prototype why: Previous version used an `enum` value to indicate what backend is to be used. This was replaced by using the backend object type. * Rewrite `hikeUp()` return code into `Result[Hike,(Hike,AristoError)]` why: Better code maintenance. Previously, the `Hike` object was returned. It had an internal error field so partial success was also available on a failure. This error field has been removed. * Use `openArray[byte]` rather than `Blob` in functions prototypes * Provide synchronised multi instance transactions why: The `CoreDB` object was geared towards the legacy DB which used a single transaction for the key-value backend DB. Different state roots are provided by the backend database, so all instances work directly on the same backend. Aristo db instances have different in-memory mappings (aka different state roots) and the transactions are on top of there mappings. So each instance might run different transactions. Multi instance transactions are a compromise to converge towards the legacy behaviour. The synchronised transactions span over all instances available at the time when base transaction was opened. Instances created later are unaffected. * Provide key-value pair database iterator why: Needed in `CoreDB` for `replicate()` emulation also: Some update of internal code * Extend API (i.e. prototype variants) why: Needed for `CoreDB` geared towards the legacy backend which has a more basic API than Aristo.
2023-09-15 15:23:53 +00:00
proc init*(
T: type AristoDbRef; # Target type
B: type MemOnlyBackend; # Backend type
): T =
## Memory backend constructor.
Aristo db fixes n updates needed for filter fifo (#1728) * Set scheduler state as part of the backend descriptor details: Moved type definitions `QidLayoutRef` and `QidSchedRef` to `desc_structural.nim` so that it shares the same folder as `desc_backend.nim` * Automatic filter queue table initialisation in backend details: Scheduler can be tweaked or completely disabled * Updated backend unit tests details: + some code clean up/beautification, reads better now + disabled persistent filters so that there is no automated filter management which will be implemented next * Prettify/update unit tests source code details: Mostly replacing the `check()` paradigm by `xCheck()` * Somewhat simplified backend type management why: Backend objects are labelled with a `BackendType` symbol where the `BackendVoid` label is implicitly assumed for a `nil` backend object reference. To make it easier, a `kind()` function is used now applicable to `nil` references as well. * Fix DB storage layout for filter objects why: Need to store the filter ID with the object * Implement reverse [] index on fifo why: An integer index argument on `[]` retrieves the QueueID (label) of the fifo item while a QueueID argument on `[]` retrieves the index (so it is inverse to the former variant). * Provide iterator over filters as fifo why: This iterator goes along the cascased fifo structure (i.e. in historical order)
2023-09-05 13:57:20 +00:00
##
## If the `qidLayout` argument is set `QidLayoutRef(nil)`, the a backend
## database will not provide filter history management. Providing a different
## scheduler layout shoud be used with care as table access with different
## layouts might render the filter history data unmanageable.
##
Aristo db api extensions for use as core db backend (#1754) * Update docu * Update Aristo/Kvt constructor prototype why: Previous version used an `enum` value to indicate what backend is to be used. This was replaced by using the backend object type. * Rewrite `hikeUp()` return code into `Result[Hike,(Hike,AristoError)]` why: Better code maintenance. Previously, the `Hike` object was returned. It had an internal error field so partial success was also available on a failure. This error field has been removed. * Use `openArray[byte]` rather than `Blob` in functions prototypes * Provide synchronised multi instance transactions why: The `CoreDB` object was geared towards the legacy DB which used a single transaction for the key-value backend DB. Different state roots are provided by the backend database, so all instances work directly on the same backend. Aristo db instances have different in-memory mappings (aka different state roots) and the transactions are on top of there mappings. So each instance might run different transactions. Multi instance transactions are a compromise to converge towards the legacy behaviour. The synchronised transactions span over all instances available at the time when base transaction was opened. Instances created later are unaffected. * Provide key-value pair database iterator why: Needed in `CoreDB` for `replicate()` emulation also: Some update of internal code * Extend API (i.e. prototype variants) why: Needed for `CoreDB` geared towards the legacy backend which has a more basic API than Aristo.
2023-09-15 15:23:53 +00:00
when B is VoidBackendRef:
Core db update storage root management for sub tries (#1964) * Aristo: Re-phrase `LayerDelta` and `LayerFinal` as object references why: Avoids copying in some cases * Fix copyright header * Aristo: Verify `leafTie.root` function argument for `merge()` proc why: Zero root will lead to inconsistent DB entry * Aristo: Update failure condition for hash labels compiler `hashify()` why: Node need not be rejected as long as links are on the schedule. In that case, `redo[]` is to become `wff.base[]` at a later stage. This amends an earlier fix, part of #1952 by also testing against the target nodes of the `wff.base[]` sets. * Aristo: Add storage root glue record to `hashify()` schedule why: An account leaf node might refer to a non-resolvable storage root ID. Storage root node chains will end up at the storage root. So the link `storage-root->account-leaf` needs an extra item in the schedule. * Aristo: fix error code returned by `fetchPayload()` details: Final error code is implied by the error code form the `hikeUp()` function. * CoreDb: Discard `createOk` argument in API `getRoot()` function why: Not needed for the legacy DB. For the `Arsto` DB, a lazy approach is implemented where a stprage root node is created on-the-fly. * CoreDb: Prevent `$$` logging in some cases why: Logging the function `$$` is not useful when it is used for internal use, i.e. retrieving an an error text for logging. * CoreDb: Add `tryHashFn()` to API for pretty printing why: Pretty printing must not change the hashification status for the `Aristo` DB. So there is an independent API wrapper for getting the node hash which never updated the hashes. * CoreDb: Discard `update` argument in API `hash()` function why: When calling the API function `hash()`, the latest state is always wanted. For a version that uses the current state as-is without checking, the function `tryHash()` was added to the backend. * CoreDb: Update opaque vertex ID objects for the `Aristo` backend why: For `Aristo`, vID objects encapsulate a numeric `VertexID` referencing a vertex (rather than a node hash as used on the legacy backend.) For storage sub-tries, there might be no initial vertex known when the descriptor is created. So opaque vertex ID objects are supported without a valid `VertexID` which will be initalised on-the-fly when the first item is merged. * CoreDb: Add pretty printer for opaque vertex ID objects * Cosmetics, printing profiling data * CoreDb: Fix segfault in `Aristo` backend when creating MPT descriptor why: Missing initialisation error * CoreDb: Allow MPT to inherit shared context on `Aristo` backend why: Creates descriptors with different storage roots for the same shared `Aristo` DB descriptor. * Cosmetics, update diagnostic message items for `Aristo` backend * Fix Copyright year
2024-01-11 19:11:38 +00:00
AristoDbRef(top: LayerRef.init())
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
Aristo db api extensions for use as core db backend (#1754) * Update docu * Update Aristo/Kvt constructor prototype why: Previous version used an `enum` value to indicate what backend is to be used. This was replaced by using the backend object type. * Rewrite `hikeUp()` return code into `Result[Hike,(Hike,AristoError)]` why: Better code maintenance. Previously, the `Hike` object was returned. It had an internal error field so partial success was also available on a failure. This error field has been removed. * Use `openArray[byte]` rather than `Blob` in functions prototypes * Provide synchronised multi instance transactions why: The `CoreDB` object was geared towards the legacy DB which used a single transaction for the key-value backend DB. Different state roots are provided by the backend database, so all instances work directly on the same backend. Aristo db instances have different in-memory mappings (aka different state roots) and the transactions are on top of there mappings. So each instance might run different transactions. Multi instance transactions are a compromise to converge towards the legacy behaviour. The synchronised transactions span over all instances available at the time when base transaction was opened. Instances created later are unaffected. * Provide key-value pair database iterator why: Needed in `CoreDB` for `replicate()` emulation also: Some update of internal code * Extend API (i.e. prototype variants) why: Needed for `CoreDB` geared towards the legacy backend which has a more basic API than Aristo.
2023-09-15 15:23:53 +00:00
elif B is MemBackendRef:
Aristo cull journal related stuff (#2288) * Remove all journal related stuff * Refactor function names journal*() => delta*(), filter*() => delta*() * remove `trg` fileld from `FilterRef` why: Same as `kMap[$1]` * Re-type FilterRef.src as `HashKey` why: So it is directly comparable to `kMap[$1]` * Moved `vGen[]` field from `LayerFinalRef` to `LayerDeltaRef` why: Then a separate `FilterRef` type is not needed, anymore * Rename `roFilter` field in `AristoDbRef` => `balancer` why: New name more appropriate. * Replace `FilterRef` by `LayerDeltaRef` type why: This allows to avoid copying into the `balancer` (see next patch set) most of the time. Typically, only one instance is running on the backend and the `balancer` is only used as a stage before saving data. * Refactor way how to store data persistently why: Avoid useless copy when staging `top` layer for persistently saving to backend. * Fix copyright header?
2024-06-03 20:10:35 +00:00
AristoDbRef(top: LayerRef.init(), backend: memoryBackend())
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
Aristo db api extensions for use as core db backend (#1754) * Update docu * Update Aristo/Kvt constructor prototype why: Previous version used an `enum` value to indicate what backend is to be used. This was replaced by using the backend object type. * Rewrite `hikeUp()` return code into `Result[Hike,(Hike,AristoError)]` why: Better code maintenance. Previously, the `Hike` object was returned. It had an internal error field so partial success was also available on a failure. This error field has been removed. * Use `openArray[byte]` rather than `Blob` in functions prototypes * Provide synchronised multi instance transactions why: The `CoreDB` object was geared towards the legacy DB which used a single transaction for the key-value backend DB. Different state roots are provided by the backend database, so all instances work directly on the same backend. Aristo db instances have different in-memory mappings (aka different state roots) and the transactions are on top of there mappings. So each instance might run different transactions. Multi instance transactions are a compromise to converge towards the legacy behaviour. The synchronised transactions span over all instances available at the time when base transaction was opened. Instances created later are unaffected. * Provide key-value pair database iterator why: Needed in `CoreDB` for `replicate()` emulation also: Some update of internal code * Extend API (i.e. prototype variants) why: Needed for `CoreDB` geared towards the legacy backend which has a more basic API than Aristo.
2023-09-15 15:23:53 +00:00
proc init*(
T: type AristoDbRef; # Target type
): T =
## Shortcut for `AristoDbRef.init(VoidBackendRef)`
AristoDbRef.init VoidBackendRef
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
Coeredb related clean up and maint fixes (#2360) * Fix initialiser why: Possible crash (app profiling, tracer etc.) * Update column family options processing why: Same for kvt as for aristo * Move `AristoDbDualRocks` backend type to the test suite why: So it is not available for production * Fix typos in API jump table why: Used for tracing and app profiling only. Needed some update * Purged CoreDb legacy API why: Not needed anymore, was transitionary and disabled. * Rename `flush` argument to `eradicate` in a DB close context why: The word `eradicate` leaves no doubt what is meant * Rename `stoFlush()` -> `stoDelete()` * Rename `core_apps_newapi` -> `core_apps` (not so new anymore)
2024-06-14 11:19:48 +00:00
proc finish*(db: AristoDbRef; eradicate = false) =
## Backend destructor. The argument `eradicate` indicates that a full
## database deletion is requested. If set `false` the outcome might differ
## depending on the type of backend (e.g. the `BackendMemory` backend will
## always eradicate on close.)
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
##
Aristo db implement distributed backend access (#1688) * Fix hashing algorithm why: Particular case where a sub-tree is on the backend, linked by an Extension vertex to the top level. * Update backend verification to report `dirty` top layer * Implement distributed merge of backend filters * Implement distributed backend access management details: Implemented and tested as described in chapter 5 of the `README.md` file.
2023-08-17 13:42:01 +00:00
## In case of distributed descriptors accessing the same backend, all
## distributed descriptors will be destroyed.
##
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
## This distructor may be used on already *destructed* descriptors.
Aristo db implement distributed backend access (#1688) * Fix hashing algorithm why: Particular case where a sub-tree is on the backend, linked by an Extension vertex to the top level. * Update backend verification to report `dirty` top layer * Implement distributed merge of backend filters * Implement distributed backend access management details: Implemented and tested as described in chapter 5 of the `README.md` file.
2023-08-17 13:42:01 +00:00
##
Aristo+Kvt: Let destructor crash when `nil` argument is given (#2080) why: Ignoring `nil` objects was handy for a while but eventually led to lazy programming which in turn led to double destructor calls for the rocks-db.
2024-03-15 14:20:00 +00:00
if not db.backend.isNil:
Coeredb related clean up and maint fixes (#2360) * Fix initialiser why: Possible crash (app profiling, tracer etc.) * Update column family options processing why: Same for kvt as for aristo * Move `AristoDbDualRocks` backend type to the test suite why: So it is not available for production * Fix typos in API jump table why: Used for tracing and app profiling only. Needed some update * Purged CoreDb legacy API why: Not needed anymore, was transitionary and disabled. * Rename `flush` argument to `eradicate` in a DB close context why: The word `eradicate` leaves no doubt what is meant * Rename `stoFlush()` -> `stoDelete()` * Rename `core_apps_newapi` -> `core_apps` (not so new anymore)
2024-06-14 11:19:48 +00:00
db.backend.closeFn eradicate
Aristo+Kvt: Let destructor crash when `nil` argument is given (#2080) why: Ignoring `nil` objects was handy for a while but eventually led to lazy programming which in turn led to double destructor calls for the rocks-db.
2024-03-15 14:20:00 +00:00
discard db.getCentre.forgetOthers()
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
# ------------------------------------------------------------------------------
# End
# ------------------------------------------------------------------------------