status-im
/
nimbus-eth1
mirror of https://github.com/status-im/nimbus-eth1.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth1/nimbus/db/aristo/aristo_init/persistent.nim

63 lines
2.2 KiB
Nim
Raw Normal View History

Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
# nimbus-eth1
# Copyright (c) 2021 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or
# http://www.apache.org/licenses/LICENSE-2.0)
# * MIT license ([LICENSE-MIT](LICENSE-MIT) or
# http://opensource.org/licenses/MIT)
# at your option. This file may not be copied, modified, or distributed
# except according to those terms.
## Persistent constructor for Aristo DB
## ====================================
##
## This module automatically pulls in the persistent backend library at the
## linking stage (e.g. `rocksdb`) which can be avoided for pure memory DB
## applications by importing `./aristo_init/memory_only` (rather than
## `./aristo_init/persistent`.)
##
{.push raises: [].}
import
results,
../aristo_desc,
Aristo db api extensions for use as core db backend (#1754) * Update docu * Update Aristo/Kvt constructor prototype why: Previous version used an `enum` value to indicate what backend is to be used. This was replaced by using the backend object type. * Rewrite `hikeUp()` return code into `Result[Hike,(Hike,AristoError)]` why: Better code maintenance. Previously, the `Hike` object was returned. It had an internal error field so partial success was also available on a failure. This error field has been removed. * Use `openArray[byte]` rather than `Blob` in functions prototypes * Provide synchronised multi instance transactions why: The `CoreDB` object was geared towards the legacy DB which used a single transaction for the key-value backend DB. Different state roots are provided by the backend database, so all instances work directly on the same backend. Aristo db instances have different in-memory mappings (aka different state roots) and the transactions are on top of there mappings. So each instance might run different transactions. Multi instance transactions are a compromise to converge towards the legacy behaviour. The synchronised transactions span over all instances available at the time when base transaction was opened. Instances created later are unaffected. * Provide key-value pair database iterator why: Needed in `CoreDB` for `replicate()` emulation also: Some update of internal code * Extend API (i.e. prototype variants) why: Needed for `CoreDB` geared towards the legacy backend which has a more basic API than Aristo.
2023-09-15 15:23:53 +00:00
"."/[rocks_db, memory_only]
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
export
Aristo db use filter betw backend and tx cache (#1678) * Provide deep copy for each transaction layer why: Localising changes. Selective deep copy was just overlooked. * Generalise vertex ID generator state reorg function `vidReorg()` why: makes it somewhat easier to handle when saving layers. * Provide dummy back end descriptor `NoneBackendRef` * Optional read-only filter between backend and transaction cache why: Some staging area for accumulating changes to the backend DB. This will eventually be an access layer for emulating a backend with multiple/historic state roots. * Re-factor `persistent()` with filter between backend/tx-cache => `stow()` why: The filter provides an abstraction from the physically stored data on disk. So, there can be several MPT instances using the same disk data with different state roots. Of course, all the MPT instances should not differ too much for practical reasons :). TODO: Filter administration tools need to be provided.
2023-08-10 20:01:28 +00:00
RdbBackendRef,
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
memory_only
# ------------------------------------------------------------------------------
# Public database constuctors, destructor
# ------------------------------------------------------------------------------
Aristo db api extensions for use as core db backend (#1754) * Update docu * Update Aristo/Kvt constructor prototype why: Previous version used an `enum` value to indicate what backend is to be used. This was replaced by using the backend object type. * Rewrite `hikeUp()` return code into `Result[Hike,(Hike,AristoError)]` why: Better code maintenance. Previously, the `Hike` object was returned. It had an internal error field so partial success was also available on a failure. This error field has been removed. * Use `openArray[byte]` rather than `Blob` in functions prototypes * Provide synchronised multi instance transactions why: The `CoreDB` object was geared towards the legacy DB which used a single transaction for the key-value backend DB. Different state roots are provided by the backend database, so all instances work directly on the same backend. Aristo db instances have different in-memory mappings (aka different state roots) and the transactions are on top of there mappings. So each instance might run different transactions. Multi instance transactions are a compromise to converge towards the legacy behaviour. The synchronised transactions span over all instances available at the time when base transaction was opened. Instances created later are unaffected. * Provide key-value pair database iterator why: Needed in `CoreDB` for `replicate()` emulation also: Some update of internal code * Extend API (i.e. prototype variants) why: Needed for `CoreDB` geared towards the legacy backend which has a more basic API than Aristo.
2023-09-15 15:23:53 +00:00
proc init*[W: MemOnlyBackend|RdbBackendRef](
T: type AristoDbRef;
B: type W;
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
basePath: string;
Aristo db fixes n updates needed for filter fifo (#1728) * Set scheduler state as part of the backend descriptor details: Moved type definitions `QidLayoutRef` and `QidSchedRef` to `desc_structural.nim` so that it shares the same folder as `desc_backend.nim` * Automatic filter queue table initialisation in backend details: Scheduler can be tweaked or completely disabled * Updated backend unit tests details: + some code clean up/beautification, reads better now + disabled persistent filters so that there is no automated filter management which will be implemented next * Prettify/update unit tests source code details: Mostly replacing the `check()` paradigm by `xCheck()` * Somewhat simplified backend type management why: Backend objects are labelled with a `BackendType` symbol where the `BackendVoid` label is implicitly assumed for a `nil` backend object reference. To make it easier, a `kind()` function is used now applicable to `nil` references as well. * Fix DB storage layout for filter objects why: Need to store the filter ID with the object * Implement reverse [] index on fifo why: An integer index argument on `[]` retrieves the QueueID (label) of the fifo item while a QueueID argument on `[]` retrieves the index (so it is inverse to the former variant). * Provide iterator over filters as fifo why: This iterator goes along the cascased fifo structure (i.e. in historical order)
2023-09-05 13:57:20 +00:00
qidLayout = DefaultQidLayoutRef;
Aristo db api extensions for use as core db backend (#1754) * Update docu * Update Aristo/Kvt constructor prototype why: Previous version used an `enum` value to indicate what backend is to be used. This was replaced by using the backend object type. * Rewrite `hikeUp()` return code into `Result[Hike,(Hike,AristoError)]` why: Better code maintenance. Previously, the `Hike` object was returned. It had an internal error field so partial success was also available on a failure. This error field has been removed. * Use `openArray[byte]` rather than `Blob` in functions prototypes * Provide synchronised multi instance transactions why: The `CoreDB` object was geared towards the legacy DB which used a single transaction for the key-value backend DB. Different state roots are provided by the backend database, so all instances work directly on the same backend. Aristo db instances have different in-memory mappings (aka different state roots) and the transactions are on top of there mappings. So each instance might run different transactions. Multi instance transactions are a compromise to converge towards the legacy behaviour. The synchronised transactions span over all instances available at the time when base transaction was opened. Instances created later are unaffected. * Provide key-value pair database iterator why: Needed in `CoreDB` for `replicate()` emulation also: Some update of internal code * Extend API (i.e. prototype variants) why: Needed for `CoreDB` geared towards the legacy backend which has a more basic API than Aristo.
2023-09-15 15:23:53 +00:00
): Result[T, AristoError] =
## Generic constructor, `basePath` argument is ignored for memory backend
## databases (which also unconditionally succeed initialising.)
Aristo db fixes n updates needed for filter fifo (#1728) * Set scheduler state as part of the backend descriptor details: Moved type definitions `QidLayoutRef` and `QidSchedRef` to `desc_structural.nim` so that it shares the same folder as `desc_backend.nim` * Automatic filter queue table initialisation in backend details: Scheduler can be tweaked or completely disabled * Updated backend unit tests details: + some code clean up/beautification, reads better now + disabled persistent filters so that there is no automated filter management which will be implemented next * Prettify/update unit tests source code details: Mostly replacing the `check()` paradigm by `xCheck()` * Somewhat simplified backend type management why: Backend objects are labelled with a `BackendType` symbol where the `BackendVoid` label is implicitly assumed for a `nil` backend object reference. To make it easier, a `kind()` function is used now applicable to `nil` references as well. * Fix DB storage layout for filter objects why: Need to store the filter ID with the object * Implement reverse [] index on fifo why: An integer index argument on `[]` retrieves the QueueID (label) of the fifo item while a QueueID argument on `[]` retrieves the index (so it is inverse to the former variant). * Provide iterator over filters as fifo why: This iterator goes along the cascased fifo structure (i.e. in historical order)
2023-09-05 13:57:20 +00:00
##
## If the `qidLayout` argument is set `QidLayoutRef(nil)`, the a backend
## database will not provide filter history management. Providing a different
## scheduler layout shoud be used with care as table access with different
## layouts might render the filter history data unmanageable.
##
Aristo db api extensions for use as core db backend (#1754) * Update docu * Update Aristo/Kvt constructor prototype why: Previous version used an `enum` value to indicate what backend is to be used. This was replaced by using the backend object type. * Rewrite `hikeUp()` return code into `Result[Hike,(Hike,AristoError)]` why: Better code maintenance. Previously, the `Hike` object was returned. It had an internal error field so partial success was also available on a failure. This error field has been removed. * Use `openArray[byte]` rather than `Blob` in functions prototypes * Provide synchronised multi instance transactions why: The `CoreDB` object was geared towards the legacy DB which used a single transaction for the key-value backend DB. Different state roots are provided by the backend database, so all instances work directly on the same backend. Aristo db instances have different in-memory mappings (aka different state roots) and the transactions are on top of there mappings. So each instance might run different transactions. Multi instance transactions are a compromise to converge towards the legacy behaviour. The synchronised transactions span over all instances available at the time when base transaction was opened. Instances created later are unaffected. * Provide key-value pair database iterator why: Needed in `CoreDB` for `replicate()` emulation also: Some update of internal code * Extend API (i.e. prototype variants) why: Needed for `CoreDB` geared towards the legacy backend which has a more basic API than Aristo.
2023-09-15 15:23:53 +00:00
when B is RdbBackendRef:
Aristo db code massage n cosmetics (#1745) * Rewrite remaining `AristoError` return code into `Result[void,AristoError]` why: Better code maintenance * Update import sections * Update Aristo DB paths why: More systematic so directory can be shared with other DB types * More cosmetcs * Update unit tests runners why: Proper handling of persistent and mem-only DB. The latter can be consistently triggered by an empty DB path.
2023-09-12 18:45:12 +00:00
let
be = ? rocksDbBackend(basePath, qidLayout)
vGen = block:
let rc = be.getIdgFn()
if rc.isErr:
be.closeFn(flush = false)
return err(rc.error)
rc.value
Aristo db implement filter serialisation for storage (#1695) * Remove concept of empty/blind filters why: Not needed. A non-existent filter is is coded as a nil reference. * Slightly generalised backend iterators why: * VertexID as key for the ID generator state makes no sense * there will be more tables addressed by non-VertexID keys * Store serialised/blobified vertices on memory backend why: This is more in line with the RocksDB backend so more appropriate for testing when comparing behaviour. For a speedy memory database, a backend-less variant should be used. * Drop the `Aristo` prefix from names `AristoLayerRef`, etc. * Suppress compiler warning why: duplicate imports * Add filter serialisation transcoder why: Will be used as storage format
2023-08-18 19:46:55 +00:00
ok AristoDbRef(top: LayerRef(vGen: vGen), backend: be)
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
else:
Aristo db api extensions for use as core db backend (#1754) * Update docu * Update Aristo/Kvt constructor prototype why: Previous version used an `enum` value to indicate what backend is to be used. This was replaced by using the backend object type. * Rewrite `hikeUp()` return code into `Result[Hike,(Hike,AristoError)]` why: Better code maintenance. Previously, the `Hike` object was returned. It had an internal error field so partial success was also available on a failure. This error field has been removed. * Use `openArray[byte]` rather than `Blob` in functions prototypes * Provide synchronised multi instance transactions why: The `CoreDB` object was geared towards the legacy DB which used a single transaction for the key-value backend DB. Different state roots are provided by the backend database, so all instances work directly on the same backend. Aristo db instances have different in-memory mappings (aka different state roots) and the transactions are on top of there mappings. So each instance might run different transactions. Multi instance transactions are a compromise to converge towards the legacy behaviour. The synchronised transactions span over all instances available at the time when base transaction was opened. Instances created later are unaffected. * Provide key-value pair database iterator why: Needed in `CoreDB` for `replicate()` emulation also: Some update of internal code * Extend API (i.e. prototype variants) why: Needed for `CoreDB` geared towards the legacy backend which has a more basic API than Aristo.
2023-09-15 15:23:53 +00:00
ok AristoDbRef.init(B, qidLayout)
Aristo db refactor tx paradim (#1674) * Better error handling why: Bail out on some error as early as possible before any changes. * Implement `fetch()` as opposite of `merge()` rationale: In the `Aristo` realm, the action named `fetch()` and `merge()` indicate leaf value related actions on the MPT, while actions `get()` and `put()` handle vertex or hash key related operations that constitute the MPT. * Re-factor `merge()` prototypes why: The most used variant of `merge()` should have the simplest prototype. * Persistent DB constructor needs to import `aristo/aristo_init/persistent` why: Most applications use memory DB anyway. This avoids linking `-lrocksdb` or any other back end libraries by default. * Re-factor transaction module why: Got the paradigm wrong. The transaction descriptor did replace the database one but should be handled separately.
2023-08-07 17:45:23 +00:00
# ------------------------------------------------------------------------------
# End
# ------------------------------------------------------------------------------