Commit Graph

17 Commits

Author SHA1 Message Date
Jordan Hrycaj debba5a620
Coeredb related clean up and maint fixes (#2360)
* Fix initialiser

why:
  Possible crash (app profiling, tracer etc.)

* Update column family options processing

why:
  Same for kvt as for aristo

* Move `AristoDbDualRocks` backend type to the test suite

why:
  So it is not available for production

* Fix typos in API jump table

why:
  Used for tracing and app profiling only. Needed some update

* Purged CoreDb legacy API

why:
  Not needed anymore, was transitionary and disabled.

* Rename `flush` argument to `eradicate` in a DB close context

why:
  The word `eradicate` leaves no doubt what is meant

* Rename `stoFlush()` -> `stoDelete()`

* Rename `core_apps_newapi` -> `core_apps` (not so new anymore)
2024-06-14 11:19:48 +00:00
Jordan Hrycaj 5a5cc6295e
Triggered write event for kvt (#2351)
* bump rockdb

* Rename `KVT` objects related to filters according to `Aristo` naming

details:
  filter* => delta*
  roFilter => balancer

* Compulsory error handling if `persistent()` fails

* Add return code to `reCentre()`

why:
  Might eventually fail if re-centring is blocked. Some logic will be
  added in subsequent patch sets.

* Add column families from earlier session to rocksdb in opening procedure

why:
  All previously used CFs must be declared when re-opening an existing
  database.

* Update `init()` and add rocksdb `reinit()` methods for changing parameters

why:
  Opening a set column families (with different open options) must span
  at least the ones that are already on disk.

* Provide write-trigger-event interface into `Aristo` backend

why:
  This allows to save data from a guest application (think `KVT`) to
  get synced with the write cycle so the guest and `Aristo` save all
  atomically.

* Use `KVT` with new column family interface from `Aristo`

* Remove obsolete guest interface

* Implement `KVT` piggyback on `Aristo` backend

* CoreDb: Add separate `KVT`/`Aristo` backend mode for debugging

* Remove `rocks_db` import from `persist()` function

why:
  Some systems (i.p `fluffy` and friends) use the `Aristo` memory
  backend emulation and do not link against rocksdb when building the
  application. So this should fix that problem.
2024-06-13 18:15:11 +00:00
Jordan Hrycaj f926222fec
Aristo cull journal related stuff (#2288)
* Remove all journal related stuff

* Refactor function names journal*() => delta*(), filter*() => delta*()

* remove `trg` fileld from `FilterRef`

why:
  Same as `kMap[$1]`

* Re-type FilterRef.src as `HashKey`

why:
  So it is directly comparable to `kMap[$1]`

* Moved `vGen[]` field from `LayerFinalRef` to `LayerDeltaRef`

why:
  Then a separate `FilterRef` type is not needed, anymore

* Rename `roFilter` field in `AristoDbRef` => `balancer`

why:
  New name more appropriate.

* Replace `FilterRef` by `LayerDeltaRef` type

why:
  This allows to avoid copying into the `balancer` (see next patch set)
  most of the time. Typically, only one instance is running on the backend
  and the `balancer` is only used as a stage before saving data.

* Refactor way how to store data persistently

why:
  Avoid useless copy when staging `top` layer for persistently saving to
  backend.

* Fix copyright header?
2024-06-03 20:10:35 +00:00
Jordan Hrycaj bda760f41d
Run coredb without journal (#2266)
* Add persistent last state stamp feature

why:
  This allows to run `CoreDb` without journal

* Start `CoreDb` without journal

* Remove journal related functions from `CoredDb`
2024-05-31 17:32:22 +00:00
andri lim 5db6ee07fb
Fix accounts_ledger savePoint conversion (#2226) 2024-05-26 17:42:02 +07:00
Jordan Hrycaj 9cc6e5a3aa
Aristo resume off line syncing on pre loaded database (#2203)
* Update some docu & messages

* Remove cruft from the ledger modules

* Must not overwrite genesis data on an initialised database

why:
  This will overwrite the global state of the Aristo single state DB.
  Otherwise resuming at the last synced state becomes impossible.

* Provide latest block number from journal

why:
  This relates the global state of the DB directly to the corresponding
  block number.

* Implemented unit test providing DB pre-load and resume
2024-05-22 13:41:14 +00:00
Jordan Hrycaj 143f2e99f5
Core db+aristo fixes and tx handling updates (#2164)
* Aristo: Rename journal related sources and functions

why:
  Previously, the naming was hinged on the phrases `fifo`, `filter` etc.
  which reflect the inner workings of cascaded filters. This was
  unfortunate for reading/understanding the source code for actions where
  the focus is the journal as a whole.

* Aristo: Fix buffer overflow (path length truncating error)

* Aristo: Tighten `hikeUp()` stop check, update error code

why:
  Detect dangling vertex links. These are legit with `snap` sync
  processing but not with regular processing.

* Aristo: Raise assert in regular mode `merge()` at a dangling link/edge

why:
  With `snap` sync processing, partial trees are ok and can be amended.
  Not so in regular mode.

  Previously there was only a debug message when a non-legit dangling edge
  was encountered.

* Aristo: Make sure that vertices are copied before modification

why:
  Otherwise vertices from lower layers might also be modified

* Aristo: Fix relaxed mode for validity checker `check()`

* Remove cruft

* Aristo: Update API for transaction handling

details:
+ Split `aristo_tx.nim` into sub-modules
+ Split `forkWith()` into `findTx()` + `forkTx()`
+ Removed `forkTop()`, `forkBase()` (now superseded by new `forkTx()`)

* CoreDb+Aristo: Fix initialiser (missing methods)
2024-05-03 17:38:17 +00:00
Jordan Hrycaj 961f63358e
Core db+aristo update recovery journal management (#2156)
* Aristo: Allow to define/set `FilterID` for journal filter records

why:
  After some changes, the `FilterID` is isomorphic to the `BlockNumber`
  scalar (well, the first 2^64 entries of a `BlockNumber`.)

  The needed change for `FilterID` is that the `FilterID(0)` value is
  valid part of the `FilterID` scalar. A non-valid `FilterID` entry is
  represented by `none(FilterID)`.

* Aristo: Split off function `persist()` as persistent version of `stow()`

why:
  In production, `stow(persistent=false,..)` is currently unused. So,
  using `persist()` rather than `stow(persistent=true,..)` improves
  readability and is better to maintain.

* CoreDb+Aristo: Store block numbers in journal records

why:
  This makes journal records searchable by block numbers

* Aristo: Rename some journal related functions

why:
  The name *journal* is more appropriate to api functions than something
   with *fifo* or *filter*.

* CoreDb+Aristo: Update last/oldest journal state retrieval

* CoreDb+Aristo: Register block number with state root in journal

why:
  No need anymore for extra lookup table `stRootToBlockNum` which maps
  a storage root -> block number.

* Aristo: Remove unused function `getFilUbe()` from api

* CoreDb: Remove now unused virtual table `stRootToBlockNum`

why:
  Was used to map a state root to a block number. This functionality
  is now embedded into the recovery journal backend.

* Turn of API tracking (will fail on `fluffy`)
2024-04-29 20:17:17 +00:00
Jordan Hrycaj 0d4ef023ed
Update aristo journal functionality (#2155)
* Aristo: Code cosmetics, e.g. update some CamelCase names

* CoreDb+Aristo: Provide oldest known state root implied

details:
  The Aristo journal allows to recover earlier but not all state roots.

* Aristo: Fix journal backward index operator, e.g. `[^1]`

* Aristo: Fix journal updater

why:
  The `fifosStore()` store function slightly misinterpreted the update
  instructions when translation is to database `put()` functions. The
  effect was that the journal was ever growing due to stale entries which
  were never deleted.

* CoreDb+Aristo: Provide utils for purging stale data from the KVT

details:
  See earlier patch, not all state roots are available. This patch
  provides a mapping from some state root to a block number and allows to
  remove all KVT data related to a particular block number

* Aristo+Kvt: Implement a clean up schedule for expired data in KVT

why:
  For a single state ledger like `Aristo`, there is only a limited
  backlog of states. So KVT data (i.e. headers etc.) are cleaned up
  regularly

* Fix copyright year
2024-04-26 13:43:52 +00:00
Jordan Hrycaj b9187e0493
Aristo selective read cashing for rocksdb backend (#2145)
* Aristo+Kvt: Better RocksDB profiling

why:
  Providing more detailed information, mainly for `Aristo`

* Aristo: Renamed journal `stats()` to `capacity()`

why:
  `Stats()` was a misnomer

* Aristo: Provide backend read caches for key and vertex IDs

why:
  Dedicated LRU caching for particular types gives a throughput advantage.
  The sizes of the LRU queues used for caching are currently constant
  but might be adjusted at a later time.

* Fix copyright year
2024-04-22 19:02:22 +00:00
Jordan Hrycaj 7d9e1d8607
Misc updates for full sync (#2140)
* Code cosmetics

* Aristo+Kvt: Fix api wrappers

why:
  Api setup killed the backend descriptor when backend mapping was
  disabled.

* Aristo: Implement masked profiling entries

why:
  Database backend should be listed but not counted in tally

* CoreDb: Simplify backend() methods

why:
  DBMS backend access Was provided very early and over engineered. Now
  there are only two backend machines, one for `Kvt` and the other one
  for an `Mpt` available only via new API.

* CoreDb: Code cleanup regarding descriptor types

* CoreDb: Refactor/redefine `persistent()` methods

why:
  There were `persistent()` methods for any type of caching storage
  facilities `Kvt`, `Mpt`, `Phk`, and `Acc`. Now there is only a single
  `persistent()` method storing all facilities in tandem (similar to
  how transactions work.)

  For non shared `Kvt` tables, there is now an extra storage method
  `saveOffSite()`.

* CoreDb lingo update: `trie` becomes `column`

why:
  Notion of a `trie` is pretty much hidden by the new `CoreDb` api.
  Revealed are sort of database columns for accounts an storage data,
  any of which have an internal state represented by a Keccack hash.
  So a `trie` or `MPT` becomes a `column` and a `rootHash` becomes a
  column state.

* Aristo: rename backend filed `filters` => `journal`

* Update full sync logging

details:
  + Disable eth handler noise while syncing
  + Log journal depth (if available)

* Fix copyright year

* Fix cruft and unwanted imports
2024-04-19 18:37:27 +00:00
Jordan Hrycaj 889a1165b0
Core db+aristo update tracer for non trivial operations (#2102)
* Remove cruft

* Docu/code cosmetics

* Aristo: Update `forkBase()`

why:
  Was not up to the job

* Update/correct tracer for running against `Aristo`

details:
  This patch makes sure that before creating a new `BaseVMState` the
  `CoreDb` context is adjusted to accommodate for the state root that
  is passed to the `BaseVMState` constructor.

* CpreDb+legacy: Always return current context with `ctxFromTx()`

why:
  There was an experimental setting trying to find the node with the
  proper setting in the KVT (not the hexary tie layer) which currently
  does not work reliable, probably due to `Ledger` caching effects.
2024-03-22 17:31:56 +00:00
Jordan Hrycaj 8ed40c78e0
Core db+aristo provides tracer funtionality (#2089)
* Aristo: Provide descriptor fork based on search in transaction stack

details:
  Try to find the tx that has a particular pair `(vertex-id,hash-key)`,
  and by extension try filter and backend if the former fails.

* Cleanup & docu

* CoreDb+Aristo: Implement context re-position to earlier in-memory state

why:
  It is a easy way to explore how there can be concurrent access to the
  same backend storage DB with different view states. This one can access
  an earlier state from the transaction stack.

* CoreDb+Aristo: Populate tracer stubs with real functionality

* Update `tracer.nim` to new API

why:
  Legacy API does not sufficiently support `Aristo`

* Fix logging problems in tracer

details:
  Debug logging turned off by default

* Fix function prototypes

* Add Copyright header

* Add tables import

why:
  For older compiler versions on CI
2024-03-21 10:45:57 +00:00
Jordan Hrycaj 99238ce0e4
Core db maintenance update (#2087)
* CoreDb+Aristo: Fix handler code

* Aristo+Kvt: Remove cruft

* Aristo+Kvt: The function `forkTop()` always provides a single transaction

why:
  Previously it provided a single squashed tx only if there were any. Now
  it will provide a blind one if there were none.

* Fix Copyright header
2024-03-20 15:15:56 +00:00
Jordan Hrycaj 0d73637f14
Core db simplify new api storage modes (#2075)
* Aristo+Kvt: Fix backend `dup()` function in api setup

why:
  Backend object is subject to an inheritance cascade which was not
  taken care of, before. Only the base object was duplicated.

* Kvt: Simplify DB clone/peers management

* Aristo: Simplify DB clone/peers management

* Aristo: Adjust unit test for working with memory DB only

why:
  This currently causes some memory corruption persumably in the
  `libc` background layer.

* CoredDb+Kvt: Simplify API for KVT

why:
  Simplified storage models (was over engineered) for better performance
  and code maintenance.

* CoredDb+Aristo: Simplify API for `Aristo`

why:
  Only single database state needed here. Accessing a similar state will
  be implemented from outside this module using a context layer. This
  gives better performance and improves code maintenance.

* Fix Copyright headers

* CoreDb: Turn off API tracking

why:
  CI would ot go through. Was accidentally turned on.
2024-03-14 22:17:43 +00:00
Jordan Hrycaj 3e1e493368
Core n lega db update tracer api (#2063)
* Aristo: Remove cruft

* Prettifying profile statistics printing & source code cosmetics

* Aristo/Kvt: API tools update

* CoreDb: Corrections, mostly API related

* CoreDb: Changed the tracer `logDb()` result

why:
  Only the key-value table is used from the logger database

* CoreDb: Update legacy tracer

why:
  The `contains()` directive did not correspond to `0 < get().len`

also:
  Copy `CoreDb` meta settings like `trackLegaApi`, `trackNewApi`, etc. to
  overlay tracer descriptor

* CoreDb: Extend/update tracer API

why:
  Get ready for accommodating `Aristo` tracer

* Fix missing import

why:
  Some CI compilers might have (cached?) a different NIM patch level

* Ditto
2024-03-07 19:24:05 +00:00
Jordan Hrycaj 587ca3abbe
Coredb use stackable api for aristo backend (#2060)
* Aristo/Kvt: Provide function hooks APIs

why:
  These APIs can be used for installing tracers, profiling functoinality,
  and other niceties on the databases.

* Aristo: Provide optional API profiling

details:
  It basically is a re-implementation of the `CoreDb` profiling
  implementation

* Kvt: Provide optional API profiling similar to `Aristo`

* CoreDb: Re-implementing profiling using `aristo_profile`

* Ledger: Re-implementing profiling using `aristo_profile`

* CoreDb: Update unit tests for maintainability

* update copyright dates
2024-02-29 21:10:24 +00:00