status-im
/
nimbus-eth1
mirror of https://github.com/status-im/nimbus-eth1.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth1/nimbus/db/aristo/aristo_init/aristo_rocksdb/rdb_desc.nim

67 lines
2.3 KiB
Nim
Raw Normal View History

Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
# nimbus-eth1
# Copyright (c) 2021 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or
# http://www.apache.org/licenses/LICENSE-2.0)
# * MIT license ([LICENSE-MIT](LICENSE-MIT) or
# http://opensource.org/licenses/MIT)
# at your option. This file may not be copied, modified, or distributed
# except according to those terms.
## Rocks DB internal driver descriptor
## ===================================
{.push raises: [].}
import
std/tables,
eth/common,
rocksdb,
stint,
../../aristo_desc,
../aristo_init_common.nim
type
RdbInst* = object
store*: RocksDBInstance ## Rocks DB database handler
basePath*: string ## Database directory
# Low level Rocks DB access for bulk store
envOpt*: rocksdb_envoptions_t
impOpt*: rocksdb_ingestexternalfileoptions_t
RdbKey* = array[1 + sizeof VertexID, byte]
## Sub-table key, <pfx> + VertexID
Aristo db consolidate and clean up (#1699) * Removed dedicated transcoder tests why: will implicitely be provided by other tests: + encode/write -> hashify -> test_tx + decode/read -> merge raw nodes -> test_tx + de/blobfiy -> backend operations, taext_tx, test_backend, test_filter * Clarify how the vertex ID generator state is accessed from the backend why: This state is a list of unused vertex IDs. It was just stored somewhere on the backend which details were exposed when iterating over some sub-table(s). As there will be more such single information records, an admin sub-tables has been defined (formerly ID generator table) with dedicated access keys and type. Also, the iterator over the single ID generator state item has been removed. It must be accessed via the `get()` interface. * Remove trailing space from file name why: fixes windows bail out
2023-08-21 14:58:30 +00:00
RdbTabs* = array[StorageType,Table[uint64,Blob]]
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
## Combined table for caching data to be stored/updated
const
BaseFolder* = "nimbus" # Same as for Legacy DB has "backups"
DataFolder* = "aristo" # Legacy DB has "data"
BackupFolder* = "history" # Legacy DB has "backups"
TempFolder* = "tmp" # Not used with legacy DB
SstCache* = "bulkput" # Rocks DB bulk load file name in temp folder
# ------------------------------------------------------------------------------
# Public functions
# ------------------------------------------------------------------------------
Aristo db consolidate and clean up (#1699) * Removed dedicated transcoder tests why: will implicitely be provided by other tests: + encode/write -> hashify -> test_tx + decode/read -> merge raw nodes -> test_tx + de/blobfiy -> backend operations, taext_tx, test_backend, test_filter * Clarify how the vertex ID generator state is accessed from the backend why: This state is a list of unused vertex IDs. It was just stored somewhere on the backend which details were exposed when iterating over some sub-table(s). As there will be more such single information records, an admin sub-tables has been defined (formerly ID generator table) with dedicated access keys and type. Also, the iterator over the single ID generator state item has been removed. It must be accessed via the `get()` interface. * Remove trailing space from file name why: fixes windows bail out
2023-08-21 14:58:30 +00:00
proc toRdbKey*(id: uint64; pfx: StorageType): RdbKey =
let idKey = id.toBytesBE
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
result[0] = pfx.ord.byte
Aristo db consolidate and clean up (#1699) * Removed dedicated transcoder tests why: will implicitely be provided by other tests: + encode/write -> hashify -> test_tx + decode/read -> merge raw nodes -> test_tx + de/blobfiy -> backend operations, taext_tx, test_backend, test_filter * Clarify how the vertex ID generator state is accessed from the backend why: This state is a list of unused vertex IDs. It was just stored somewhere on the backend which details were exposed when iterating over some sub-table(s). As there will be more such single information records, an admin sub-tables has been defined (formerly ID generator table) with dedicated access keys and type. Also, the iterator over the single ID generator state item has been removed. It must be accessed via the `get()` interface. * Remove trailing space from file name why: fixes windows bail out
2023-08-21 14:58:30 +00:00
copyMem(addr result[1], unsafeAddr idKey, sizeof idKey)
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
Aristo db implement filter serialisation for storage (#1695) * Remove concept of empty/blind filters why: Not needed. A non-existent filter is is coded as a nil reference. * Slightly generalised backend iterators why: * VertexID as key for the ID generator state makes no sense * there will be more tables addressed by non-VertexID keys * Store serialised/blobified vertices on memory backend why: This is more in line with the RocksDB backend so more appropriate for testing when comparing behaviour. For a speedy memory database, a backend-less variant should be used. * Drop the `Aristo` prefix from names `AristoLayerRef`, etc. * Suppress compiler warning why: duplicate imports * Add filter serialisation transcoder why: Will be used as storage format
2023-08-18 19:46:55 +00:00
template toOpenArray*(vid: VertexID; pfx: StorageType): openArray[byte] =
Aristo db store filters on backend (#1703) * Simplify RocksDB sub-tables iterator * Implement `filter` storage on backend db details: Unit tests working
2023-08-22 18:44:54 +00:00
vid.uint64.toRdbKey(pfx).toOpenArray(0, sizeof uint64)
template toOpenArray*(fid: FilterID): openArray[byte] =
fid.uint64.toRdbKey(FilPfx).toOpenArray(0, sizeof uint64)
Aristo db consolidate and clean up (#1699) * Removed dedicated transcoder tests why: will implicitely be provided by other tests: + encode/write -> hashify -> test_tx + decode/read -> merge raw nodes -> test_tx + de/blobfiy -> backend operations, taext_tx, test_backend, test_filter * Clarify how the vertex ID generator state is accessed from the backend why: This state is a list of unused vertex IDs. It was just stored somewhere on the backend which details were exposed when iterating over some sub-table(s). As there will be more such single information records, an admin sub-tables has been defined (formerly ID generator table) with dedicated access keys and type. Also, the iterator over the single ID generator state item has been removed. It must be accessed via the `get()` interface. * Remove trailing space from file name why: fixes windows bail out
2023-08-21 14:58:30 +00:00
template toOpenArray*(aid: AdminTabID): openArray[byte] =
Aristo db store filters on backend (#1703) * Simplify RocksDB sub-tables iterator * Implement `filter` storage on backend db details: Unit tests working
2023-08-22 18:44:54 +00:00
aid.uint64.toRdbKey(AdmPfx).toOpenArray(0, sizeof uint64)
Aristo db with storage backends (#1603) * Generalised Aristo DB constructor for any type of backend details: * Records to be deleted are represented as key-void (rather than key-value) pairs by the put-function arguments * Allow direct driver access, iterators as example implementation and for testing. * Provide backend storage interface details: Stores the top layer onto backend tables * Implemented Rocks DB backend details: Transaction based `put()` functionality Iterators (based on direct RocksDB access)
2023-06-20 13:26:25 +00:00
# ------------------------------------------------------------------------------
# End
# ------------------------------------------------------------------------------