status-im
/
nimbus-eth1
mirror of https://github.com/status-im/nimbus-eth1.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth1/nimbus/db/aristo/aristo_init/rocks_db/rdb_desc.nim

82 lines
2.6 KiB
Nim
Raw Normal View History

Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
# nimbus-eth1
Update Nimbus codebase to use the new nim-rocksdb API. (#2054) * Bump nim-rocksdb. * Update codebase to use latest nim-rocksdb API. * Update copyright notices. * Fix memory leak due to allocCStringArray without deallocCStringArray. * Improve kvstore_rocksdb code. * Refactor and cleanup RocksStoreRef. * Update nim-rocksdb submodule to latest.
2024-03-05 04:54:42 +00:00
# Copyright (c) 2023-2024 Status Research & Development GmbH
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or
# http://www.apache.org/licenses/LICENSE-2.0)
# * MIT license ([LICENSE-MIT](LICENSE-MIT) or
# http://opensource.org/licenses/MIT)
# at your option. This file may not be copied, modified, or distributed
# except according to those terms.
## Rocks DB internal driver descriptor
## ===================================
{.push raises: [].}
import
Core db fix legacy db root vertex fetcher (#1899) * Using different `tmp` directories for `Kvt` and `Aristo` why: Closing one database would leave the other set of directories incomplete. * Code cosmetics, silence compiler * Fix typo `EMPTY_ROOT_HASH` vs. `EMPTY_CODE_HASH` * Fix copyright years
2023-11-20 20:22:27 +00:00
std/[tables, os],
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
eth/common,
Update Nimbus codebase to use the new nim-rocksdb API. (#2054) * Bump nim-rocksdb. * Update codebase to use latest nim-rocksdb API. * Update copyright notices. * Fix memory leak due to allocCStringArray without deallocCStringArray. * Improve kvstore_rocksdb code. * Refactor and cleanup RocksStoreRef. * Update nim-rocksdb submodule to latest.
2024-03-05 04:54:42 +00:00
rocksdb/lib/librocksdb,
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
rocksdb,
Bump stint to v2.0: new array backend (#1747) * Bump stint to v2.0: new array backend
2023-09-13 02:32:38 +00:00
stew/endians2,
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
../../aristo_desc,
Aristo db implement filter storage scheduler (#1713) * Rename FilterID => QueueID why: The current usage does not identify a particular filter but uses it as storage tag to manage it on the database (to be organised in a set of FIFOs or queues.) * Split `aristo_filter` source into sub-files why: Make space for filter management API * Store filter queue IDs in pairs on the backend why: Any pair will will describe a FIFO accessed by bottom/top IDs * Reorg some source file names why: The "aristo_" prefix for make local/private files is tedious to use, so removed. * Implement filter slot scheduler details: Filters will be stored on the database on cascaded FIFOs. When a FIFO queue is full, some filter items are bundled together and stored on the next FIFO.
2023-08-25 22:53:59 +00:00
../init_common
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
type
RdbInst* = object
Update Nimbus codebase to use the new nim-rocksdb API. (#2054) * Bump nim-rocksdb. * Update codebase to use latest nim-rocksdb API. * Update copyright notices. * Fix memory leak due to allocCStringArray without deallocCStringArray. * Improve kvstore_rocksdb code. * Refactor and cleanup RocksStoreRef. * Update nim-rocksdb submodule to latest.
2024-03-05 04:54:42 +00:00
dbOpts*: DbOptionsRef
store*: RocksDbReadWriteRef ## Rocks DB database handler
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
basePath*: string ## Database directory
Core db simplify new api storage modes (#2075) * Aristo+Kvt: Fix backend `dup()` function in api setup why: Backend object is subject to an inheritance cascade which was not taken care of, before. Only the base object was duplicated. * Kvt: Simplify DB clone/peers management * Aristo: Simplify DB clone/peers management * Aristo: Adjust unit test for working with memory DB only why: This currently causes some memory corruption persumably in the `libc` background layer. * CoredDb+Kvt: Simplify API for KVT why: Simplified storage models (was over engineered) for better performance and code maintenance. * CoredDb+Aristo: Simplify API for `Aristo` why: Only single database state needed here. Accessing a similar state will be implemented from outside this module using a context layer. This gives better performance and improves code maintenance. * Fix Copyright headers * CoreDb: Turn off API tracking why: CI would ot go through. Was accidentally turned on.
2024-03-14 22:17:43 +00:00
noFq*: bool ## No filter queues available
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
# Low level Rocks DB access for bulk store
Update Nimbus codebase to use the new nim-rocksdb API. (#2054) * Bump nim-rocksdb. * Update codebase to use latest nim-rocksdb API. * Update copyright notices. * Fix memory leak due to allocCStringArray without deallocCStringArray. * Improve kvstore_rocksdb code. * Refactor and cleanup RocksStoreRef. * Update nim-rocksdb submodule to latest.
2024-03-05 04:54:42 +00:00
envOpt*: ptr rocksdb_envoptions_t
impOpt*: ptr rocksdb_ingestexternalfileoptions_t
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
RdbKey* = array[1 + sizeof VertexID, byte]
## Sub-table key, <pfx> + VertexID
Update Nimbus codebase to use the new nim-rocksdb API. (#2054) * Bump nim-rocksdb. * Update codebase to use latest nim-rocksdb API. * Update copyright notices. * Fix memory leak due to allocCStringArray without deallocCStringArray. * Improve kvstore_rocksdb code. * Refactor and cleanup RocksStoreRef. * Update nim-rocksdb submodule to latest.
2024-03-05 04:54:42 +00:00
RdbTabs* = array[StorageType, Table[uint64,Blob]]
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
## Combined table for caching data to be stored/updated
const
Core db fix legacy db root vertex fetcher (#1899) * Using different `tmp` directories for `Kvt` and `Aristo` why: Closing one database would leave the other set of directories incomplete. * Code cosmetics, silence compiler * Fix typo `EMPTY_ROOT_HASH` vs. `EMPTY_CODE_HASH` * Fix copyright years
2023-11-20 20:22:27 +00:00
BaseFolder* = "nimbus" # Same as for Legacy DB
DataFolder* = "aristo" # Legacy DB has "data"
SstCache* = "bulkput" # Rocks DB bulk load file name in temp folder
TempFolder* = "tmp" # No `tmp` directory used with legacy DB
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
# ------------------------------------------------------------------------------
# Public functions
# ------------------------------------------------------------------------------
Core db fix legacy db root vertex fetcher (#1899) * Using different `tmp` directories for `Kvt` and `Aristo` why: Closing one database would leave the other set of directories incomplete. * Code cosmetics, silence compiler * Fix typo `EMPTY_ROOT_HASH` vs. `EMPTY_CODE_HASH` * Fix copyright years
2023-11-20 20:22:27 +00:00
func baseDir*(rdb: RdbInst): string =
rdb.basePath / BaseFolder
func dataDir*(rdb: RdbInst): string =
rdb.baseDir / DataFolder
func cacheDir*(rdb: RdbInst): string =
rdb.dataDir / TempFolder
func sstFilePath*(rdb: RdbInst): string =
rdb.cacheDir / SstCache
func toRdbKey*(id: uint64; pfx: StorageType): RdbKey =
Aristo db consolidate and clean up (#1699) * Removed dedicated transcoder tests why: will implicitely be provided by other tests: + encode/write -> hashify -> test_tx + decode/read -> merge raw nodes -> test_tx + de/blobfiy -> backend operations, taext_tx, test_backend, test_filter * Clarify how the vertex ID generator state is accessed from the backend why: This state is a list of unused vertex IDs. It was just stored somewhere on the backend which details were exposed when iterating over some sub-table(s). As there will be more such single information records, an admin sub-tables has been defined (formerly ID generator table) with dedicated access keys and type. Also, the iterator over the single ID generator state item has been removed. It must be accessed via the `get()` interface. * Remove trailing space from file name why: fixes windows bail out
2023-08-21 14:58:30 +00:00
let idKey = id.toBytesBE
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
result[0] = pfx.ord.byte
Aristo db consolidate and clean up (#1699) * Removed dedicated transcoder tests why: will implicitely be provided by other tests: + encode/write -> hashify -> test_tx + decode/read -> merge raw nodes -> test_tx + de/blobfiy -> backend operations, taext_tx, test_backend, test_filter * Clarify how the vertex ID generator state is accessed from the backend why: This state is a list of unused vertex IDs. It was just stored somewhere on the backend which details were exposed when iterating over some sub-table(s). As there will be more such single information records, an admin sub-tables has been defined (formerly ID generator table) with dedicated access keys and type. Also, the iterator over the single ID generator state item has been removed. It must be accessed via the `get()` interface. * Remove trailing space from file name why: fixes windows bail out
2023-08-21 14:58:30 +00:00
copyMem(addr result[1], unsafeAddr idKey, sizeof idKey)
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
Aristo db implement filter serialisation for storage (#1695) * Remove concept of empty/blind filters why: Not needed. A non-existent filter is is coded as a nil reference. * Slightly generalised backend iterators why: * VertexID as key for the ID generator state makes no sense * there will be more tables addressed by non-VertexID keys * Store serialised/blobified vertices on memory backend why: This is more in line with the RocksDB backend so more appropriate for testing when comparing behaviour. For a speedy memory database, a backend-less variant should be used. * Drop the `Aristo` prefix from names `AristoLayerRef`, etc. * Suppress compiler warning why: duplicate imports * Add filter serialisation transcoder why: Will be used as storage format
2023-08-18 19:46:55 +00:00
template toOpenArray*(vid: VertexID; pfx: StorageType): openArray[byte] =
Aristo db store filters on backend (#1703) * Simplify RocksDB sub-tables iterator * Implement `filter` storage on backend db details: Unit tests working
2023-08-22 18:44:54 +00:00
vid.uint64.toRdbKey(pfx).toOpenArray(0, sizeof uint64)
Aristo db implement filter storage scheduler (#1713) * Rename FilterID => QueueID why: The current usage does not identify a particular filter but uses it as storage tag to manage it on the database (to be organised in a set of FIFOs or queues.) * Split `aristo_filter` source into sub-files why: Make space for filter management API * Store filter queue IDs in pairs on the backend why: Any pair will will describe a FIFO accessed by bottom/top IDs * Reorg some source file names why: The "aristo_" prefix for make local/private files is tedious to use, so removed. * Implement filter slot scheduler details: Filters will be stored on the database on cascaded FIFOs. When a FIFO queue is full, some filter items are bundled together and stored on the next FIFO.
2023-08-25 22:53:59 +00:00
template toOpenArray*(qid: QueueID): openArray[byte] =
qid.uint64.toRdbKey(FilPfx).toOpenArray(0, sizeof uint64)
Aristo db consolidate and clean up (#1699) * Removed dedicated transcoder tests why: will implicitely be provided by other tests: + encode/write -> hashify -> test_tx + decode/read -> merge raw nodes -> test_tx + de/blobfiy -> backend operations, taext_tx, test_backend, test_filter * Clarify how the vertex ID generator state is accessed from the backend why: This state is a list of unused vertex IDs. It was just stored somewhere on the backend which details were exposed when iterating over some sub-table(s). As there will be more such single information records, an admin sub-tables has been defined (formerly ID generator table) with dedicated access keys and type. Also, the iterator over the single ID generator state item has been removed. It must be accessed via the `get()` interface. * Remove trailing space from file name why: fixes windows bail out
2023-08-21 14:58:30 +00:00
template toOpenArray*(aid: AdminTabID): openArray[byte] =
Aristo db store filters on backend (#1703) * Simplify RocksDB sub-tables iterator * Implement `filter` storage on backend db details: Unit tests working
2023-08-22 18:44:54 +00:00
aid.uint64.toRdbKey(AdmPfx).toOpenArray(0, sizeof uint64)
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
# ------------------------------------------------------------------------------
# End
# ------------------------------------------------------------------------------