Commit Graph

230 Commits

Author SHA1 Message Date
Jordan Hrycaj 657379f484
Aristo db update merkle hasher (#1925)
* Register paths for added leafs because of trie re-balancing

why:
  While the payload would not change, the prefix in the leaf vertex
  would. So it needs to be flagged for hash recompilation for the
  `hashify()` module.

also:
  Make sure that `Hike` paths which might have vertex links into the
  backend filter are replaced by vertex copies before manipulating.
  Otherwise the vertices on the immutable filter might be involuntarily
  changed.

* Also check for paths where the leaf vertex is on the backend, already

why:
  A a path can have dome vertices on the top layer cache with the
  `Leaf` vertex on  the backend.

* Re-define a void `HashLabel` type.

why:
  A `HashLabel` type is a pair `(root-vertex-ID, Keccak-hash)`. Previously,
  a valid `HashLabel` consisted of a non-empty hash and a non-zero vertex
  ID. This definition leads to a non-unique representation of a void
  `HashLabel` with either root-ID or has void. This has been changed to
  the unique void `HashLabel` exactly if the hash entry is void.

* Update consistency checkers

* Re-org `hashify()` procedure

why:
  Syncing against block chain showed serious deficiencies which produced
  wrong hashes or simply bailed out with error.

  So all fringe cases (mainly due to deleted entries) could be integrated
  into the labelling schedule rather than handling separate fringe cases.
2023-12-04 20:39:26 +00:00
Jordan Hrycaj 5462c05dc6
Core db update api tracking (#1907)
* Fix copyright year

* Show elapsed times with enabled `CoreDb` API tracking

* Show elapsed times with enabled `LedgerRef` API tracking

* Reorg `CoreDb` auto destructors for `Aristo` DB

why:
  While `Aristo` supports some parallelism for concurrent database access,
  this comes with a price of management overhead. With a naive approach,
  the auto-destructor will slow down execution because the ledger and
  evm treat the database in a shared mode where a DB descriptor is just
  created and thrown away shortly after.

  This is reflected in the `Coredb` abstraction layer above `Aristo`/`Kvt`
  where a few `Shared` type descriptors are cached and a shared reference
  is returned rather than a disposable new object.

* For `CoreDb` support transaction level tracking

details:
  This is mainly an extra for the legacy DB as `Aristo` and `Kvt` support
  this already.

  Also return an error on the legacy DB backend when `persistent()` is
  called while there are transactions pending (the `persistent()` call
  does nothing otherwise on the legacy backend.)

* Clear compiler warnings (remove unused variables etc.)
2023-11-24 22:16:21 +00:00
Jordan Hrycaj 610e2d338d
Core db fix legacy db root vertex fetcher (#1899)
* Using different `tmp` directories for `Kvt` and `Aristo`

why:
  Closing one database would leave the other set of directories
  incomplete.

* Code cosmetics, silence compiler

* Fix typo `EMPTY_ROOT_HASH` vs. `EMPTY_CODE_HASH`

* Fix copyright years
2023-11-20 20:22:27 +00:00
Jordan Hrycaj 3e88589eb1
Optional accounts cache module for creating genesis (#1897)
* Split off `ReadOnlyStateDB` from `AccountStateDB` from `state_db.nim`

why:
  Apart from testing, applications use `ReadOnlyStateDB` as an easy
  way to access the accounts ledger. This is well supported by the
  `Aristo` db, but writable mode is only parially supported.

  The writable AccountStateDB` object for modifying accounts is not
  used by production code.

  So, for lecgacy and testing apps, the full support of the previous
  `AccountStateDB` is now enabled by `import db/state_db/read_write`
  and the `import db/state_db` provides read-only mode.

* Encapsulate `AccountStateDB` as `GenesisLedgerRef` or genesis creation

why:
  `AccountStateDB` has poor support for `Aristo` and is not widely used
   in favour of `AccountsLedger` (which will be abstracted as `ledger`.)

   Currently, using other than the `AccountStateDB` ledgers within the
   `GenesisLedgerRef` wrapper is experimental and test only. Eventually,
    the wrapper should disappear so that the `Ledger` object (which
    encapsulates `AccountsCache` and `AccountsLedger`) will prevail.

* For the `Ledger`, provide access to raw accounts `MPT`

why:
  This gives to the `CoreDbMptRef` descriptor from the `CoreDb` (which is
  the legacy version of 	CoreDxMptRef`.) For the new `ledger` API, the
  accounts are based on the `CoreDxMAccRef` descriptor which uses a
  particular sub-system for accounts while legacy applications use the
  `CoreDbPhkRef` equivalent of the `SecureHexaryTrie`.

  The only place where this feature will currently be used is the
 `genesis.nim` source file.

* Fix `Aristo` bugs, missing boundary checks, typos, etc.

* Verify root vertex in `MPT` and account constructors

why:
  Was missing so far, in particular the accounts constructor must
  verify `VertexID(1)

* Fix include file
2023-11-20 11:51:43 +00:00
Jordan Hrycaj 4825ab1566
Provide `Aristo` backend for `CoreDb` (#1895) 2023-11-16 20:53:44 +00:00
Jordan Hrycaj c47f021596
Core db and aristo updates for destructor and tx logic (#1894)
* Disable `TransactionID` related functions from `state_db.nim`

why:
  Functions `getCommittedStorage()` and `updateOriginalRoot()` from
  the `state_db` module are nowhere used. The emulation of a legacy
  `TransactionID` type functionality is administratively expensive to
  provide by `Aristo` (the legacy DB version is only partially
  implemented, anyway).

  As there is no other place where `TransactionID`s are used, they will
  not be provided by the `Aristo` variant of the `CoreDb`. For the
  legacy DB API, nothing will change.

* Fix copyright headers in source code

* Get rid of compiler warning

* Update Aristo code, remove unused `merge()` variant, export `hashify()`

why:
  Adapt to upcoming `CoreDb` wrapper

* Remove synced tx feature from `Aristo`

why:
+ This feature allowed to synchronise transaction methods like begin,
  commit, and rollback for a group of descriptors.
+ The feature is over engineered and not needed for `CoreDb`, neither
  is it complete (some convergence features missing.)

* Add debugging helpers to `Kvt`

also:
  Update database iterator, add count variable yield argument similar
  to `Aristo`.

* Provide optional destructors for `CoreDb` API

why;
  For the upcoming Aristo wrapper, this allows to control when certain
  smart destruction and update can take place. The auto destructor works
  fine in general when the storage/cache strategy is known and acceptable
  when creating descriptors.

* Add update option for `CoreDb` API function `hash()`

why;
  The hash function is typically used to get the state root of the MPT.
  Due to lazy hashing, this might be not available on the `Aristo` DB.
  So the `update` function asks for re-hashing the gurrent state changes
  if needed.

* Update API tracking log mode: `info` => `debug

* Use shared `Kvt` descriptor in new Ledger API

why:
  No need to create a new descriptor all the time
2023-11-16 19:35:03 +00:00
Kim De Mey bb88b50298
Add chronicles formatters for aristo db types (#1889) 2023-11-09 13:22:51 +01:00
Jordan Hrycaj 6e0397e276
Aristo and ledger small updates (#1888)
* Fix debug noise in `hashify()` for perfectly normal situation

why:
  Was previously considered a fixable error

* Fix test sample file names

why:
  The larger test file `goerli68161.txt.gz` is already in the local
  archive. So there is no need to use the smaller one from the external
  repo.

* Activate `accounts_cache` module from `db/ledger`

why:
  A copy of the original `accounts_cache.nim` source to be integrated
  into the `Ledger` module wrapper which allows to switch between
  different `accounts_cache` implementations unser tha same API.

details:
  At a later state, the `db/accounts_cache.nim` wrapper will be
  removed so that there is only one access to that module via
  `db/ledger/accounts_cache.nim`.

* Fix copyright headers in source code
2023-11-08 16:52:25 +00:00
Jordan Hrycaj 4feaa2cfab
Aristo db update for short nodes key edge cases (#1887)
* Aristo: Provide key-value list signature calculator

detail:
  Simple wrappers around `Aristo` core functionality

* Update new API for `CoreDb`

details:
+ Renamed new API functions `contains()` => `hasKey()` or `hasPath()`
  which disables the `in` operator on non-boolean 	`contains()` functions
+ The functions `get()` and `fetch()` always return a not-found error if
  there is no item, available. The new functions `getOrEmpty()` and
  `mergeOrEmpty()` return an an empty `Blob` if there is no such key
  found.

* Rewrite `core_apps.nim` using new API from `CoreDb`

* Use `Aristo` functionality for calculating Merkle signatures

details:
  For debugging, the `VerifyAristoForMerkleRootCalc` can be set so
  that `Aristo` results will be verified against the legacy versions.

* Provide general interface for Merkle signing key-value tables

details:
  Export `Aristo` wrappers

* Activate `CoreDb` tests

why:
  Now, API seems to be stable enough for general tests.

* Update `toHex()` usage

why:
  Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join`

* Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify`

why:
+ Different modules for different purposes
+ `aristo_serialise`: RLP encoding/decoding
+ `aristo_blobify`: Aristo database encoding/decoding

* Compacted representation of small nodes' links instead of Keccak hashes

why:
  Ethereum MPTs use Keccak hashes as node links if the size of an RLP
  encoded node is at least 32 bytes. Otherwise, the RLP encoded node
  value is used as a pseudo node link (rather than a hash.) Such a node
  is nor stored on key-value database. Rather the RLP encoded node value
  is stored instead of a lode link in a parent node instead. Only for
  the root hash, the top level node is always referred to by the hash.

  This feature needed an abstraction of the `HashKey` object which is now
  either a hash or a blob of length at most 31 bytes. This leaves two
  ways of representing an empty/void `HashKey` type, either as an empty
  blob of zero length, or the hash of an empty blob.

* Update `CoreDb` interface (mainly reducing logger noise)

* Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
jangko b0000eed8b
Add check copyright year linter to CI 2023-11-01 10:41:20 +07:00
Jordan Hrycaj 3fe0a49a5e
Aristo db allow shorter than 64 nibbles path keys (#1864)
* Aristo: Single `FetchPathNotFound` error in `fetchXxx()` and `hasPath()`

why:
  Missing path hike returns too many detailed reasons why it failed
  which becomes cumbersome to handle.

also:
  Renamed `contains()` => `hasPath()` which disables the `in` operator on
  non-boolean 	`contains()` functions

* Kvt: Renamed `contains()` => `hasKey()`

why:
  which disables the `in` operator on non-boolean 	`contains()` functions

* Aristo: Generalising `HashID` by variable length `PathID`

why:
  There are cases when the `Aristo` database is to be used with
  shorter than 64 nibbles keys when handling transactions indexes
  with sequence IDs.

caveat:
  This patch only works reliable for full length `PathID` values. Tests
  for shorter `PathID` values are currently missing.
2023-10-27 22:36:51 +01:00
Jordan Hrycaj 3198ad1bbd
Fix default pruning for ledger and update core db and ledger logging (#1861)
* Make sure that storage tries are not pruned (by default) on the new Ledger API

why:
  Pruning might kill some unwanted entries from storage tries ending up with an unstable database
  leading to crashes.

* Implement `CoreDb` and `LedgerRef` API tracing

details:
+ Locally enabled at compile time via constants `ProvideCoreDbLegacyAPI`
  and `EnableApiTracking` in either `base.nim` source
+ If enabled it can be selectively turned on/off via public switches in
  the `CoreDb` descriptor.

* Allow suppressing opportunistic `ifNecessaryGetXxx()` functions

why:
  Better troubleshooting when the system crashes (assertions will then
  most probably happen outside an `async` function.)
2023-10-25 15:03:09 +01:00
jangko c005281391
processBeaconBlockRoot in TxPool(EIP-4788) 2023-10-19 07:50:07 +07:00
Jordan Hrycaj e8ad950e0a
Ledger abstraction for accounts cache (#1824)
* Provide TDD/debug facility for inspecting `persistBlocks()` working

detail:
+ Make sure that the last block of a test sample is the first batch
  item in `persistBlocks()`.
+ Additionally, allow `AccountsCache` API tracing by setting the flag
  `extraTraceMessages = true` in the file `accounts_cache.nim`

* Overload AccountsCache by abstraction wrapper

details:
  Can facilitate CoreDb API switch, details in `ledger/README.md`.
2023-10-18 20:27:22 +01:00
Kim De Mey ca61a7009d
Bump submodules and related fixes (#1729)
* Bump submodules and related fixes

* Fix some warnings due to chronos and Nim bumps

* Seems macOS and Windows have different behavior
2023-10-17 14:19:50 +02:00
Jordan Hrycaj 786263c0b8
Core db update api and fix tracer methods (#1816)
* CoreDB: Re-org API

details:
  Legacy API internally uses vertex ID for root node abstraction

* Cosmetics: Move some unit test helpers to common sub-directory

* Extract constant from `accouns_cache.nim` => `constants.nim`

* Fix tracer methods

why:
  Logger dump data were wrongly dumped from the production database. This
  caused an assert exception when iterating over the persistent database
  (instead of the memory logger.) This event in turn was enabled after
  fixing another inconsistency which just set up an empty iterator. Unit
  tests failed to detect that.
2023-10-11 20:09:11 +01:00
Jordan Hrycaj 395580ff9d
Aristo and core db updates (#1800)
* Aristo: remove obsolete functions

* Aristo: Fix error code for non-available hash keys

why:
  Must not return `not-found` when the key is not available (i.e. the
  current changes were not hashified, yet.)

* CoreDB: Provide TDD and test framework
2023-10-03 12:56:13 +01:00
Jordan Hrycaj ea2b5b4c7e
Redefine result[] error object as ref (see issue #1796) (#1797)
details:
 This does not affect the legacy interface which is the production one,
 currently.
2023-10-02 19:05:17 +01:00
Jordan Hrycaj 6d132811ba
Core db update providing additional results code interface (#1776)
* Split `core_db/base.nim` into several sources

* Rename `core_db/legacy.nim` => `core_db/legacy_db.nim`

* Update `CoreDb` API, dual methods returning `Result[]` or plain value

detail:
  Plain value methods implemet the legacy API, they defect on error results

* Redesign `CoreDB` direct backend access

why:
  Made the `backend` directive integral part of the API

* Discontinue providing unused or otherwise available functions

details:
+ setTransactionID() removed, not used and not easily replicable in Aristo
+ maybeGet() removed, available via direct backend access
+ newPhk() removed, never used & was experimental anyway

* Update/reorg backend API

why:
+ Added error print function `$$()`
+ General descriptor completion (and optional validation) via `bless()`

* Update `Aristo`/`Kvt` exception handling

why:
  Avoid `CatchableError` exceptions, rather pass them as error code where
  appropriate.

* More `CoreDB` compliant `Aristo` and `Kvt` methods

details:
+ Providing functions like `contains()`, `getVtxRc()` (returns `Result[]`).
+ Additional error code: `NotImplemented`

* Rewrite/reorg of Aristo DB constructor

why:
  Previously used global object `DefaultQidLayoutRef` as default
  initialiser. This object was created at compile time which lead to
  non-gc safe functions.

* Update nimbus/db/core_db/legacy_db.nim

Co-authored-by: Kim De Mey <kim.demey@gmail.com>

* Update nimbus/db/aristo/aristo_transcode.nim

Co-authored-by: Kim De Mey <kim.demey@gmail.com>

* Update nimbus/db/core_db/legacy_db.nim

Co-authored-by: Kim De Mey <kim.demey@gmail.com>

---------

Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2023-09-26 10:21:13 +01:00
andri lim 6848b2010b
EIP-6780: Fix self destruct operation (#1777) 2023-09-26 09:03:21 +07:00
jangko cc7a7db74b
EIP-4788: Make it clear what is an EVM system call 2023-09-22 17:21:09 +07:00
andri lim 7af9e3dc53
Refactor beacon skeleton (#1761)
* Fix unlisted exception due to recent modification in CoreDB interface

* Refactor beacon skeleton

* More unlisted exception fix
2023-09-19 11:52:28 +07:00
Jordan Hrycaj 6bc55d4e6f
Core db aristo and kvt updates preparing for integration (#1760)
* Kvt: Implemented multi-descriptor access on the same backend

why:
  This behaviour mirrors the one of Aristo and can be used for
  simultaneous transactions on Aristo + Kvt

* Kvt: Update database iterators

why:
  Forgot to run on the top layer first

* Kvt: Misc fixes

* Aristo, use `openArray[byte]` rather than `Blob` in prototype

* Aristo, by default hashify right after cloning descriptor

why:
  Typically, a completed descriptor is expected after cloning. Hashing
  can be suppressed by argument flag.

* Aristo provides `replicate()` iterator, similar to legacy `replicate()`

* Aristo API fixes and updates

* CoreDB: Rename `legacy_persistent` => `legacy_rocksdb`

why:
  More systematic, will be in line with Aristo DB which might have
  more than one persistent backends

* CoreDB: Prettify API sources

why:
  Better to read and maintain

details:
  Annotating with custom pragmas which cleans up the prototypes

* CoreDB: Update MPT/put() prototype allowing `CatchableError`

why:
  Will be needed for Aristo API (legacy is OK with `RlpError`)
2023-09-18 21:20:28 +01:00
Jordan Hrycaj cd1d370543
Aristo db api extensions for use as core db backend (#1754)
* Update docu

* Update Aristo/Kvt constructor prototype

why:
  Previous version used an `enum` value to indicate what backend is to
  be used. This was replaced by using the backend object type.

* Rewrite `hikeUp()` return code into `Result[Hike,(Hike,AristoError)]`

why:
  Better code maintenance. Previously, the `Hike` object was returned. It
  had an internal error field so partial success was also available on
  a failure. This error field has been removed.

* Use `openArray[byte]` rather than `Blob` in functions prototypes

* Provide synchronised multi instance transactions

why:
  The `CoreDB` object was geared towards the legacy DB which used a single
  transaction for the key-value backend DB. Different state roots are
  provided by the backend database, so all instances work directly on the
  same backend.

  Aristo db instances have different in-memory mappings (aka different
  state roots) and the transactions are on top of there mappings. So each
  instance might run different transactions.

  Multi instance transactions are a compromise to converge towards the
  legacy behaviour. The synchronised transactions span over all instances
  available at the time when base transaction was opened. Instances
  created later are unaffected.

* Provide key-value pair database iterator

why:
  Needed in `CoreDB` for `replicate()` emulation

also:
  Some update of internal code

* Extend API (i.e. prototype variants)

why:
  Needed for `CoreDB` geared towards the legacy backend which has a more
  basic API than Aristo.
2023-09-15 16:23:53 +01:00
andri lim 56215ed83f
Bump stint to v2.0: new array backend (#1747)
* Bump stint to v2.0: new array backend
2023-09-13 09:32:38 +07:00
Jordan Hrycaj 8e00143313
Aristo db code massage n cosmetics (#1745)
* Rewrite remaining `AristoError` return code into `Result[void,AristoError]`

why:
  Better code maintenance

* Update import sections

* Update Aristo DB paths

why:
 More systematic so directory can be shared with other DB types

* More cosmetcs

* Update unit tests runners

why:
  Proper handling of persistent and mem-only DB. The latter can be
  consistently triggered by an empty DB path.
2023-09-12 19:45:12 +01:00
Jordan Hrycaj dda049cd43
Simple stupid key-value table companion for Aristo DB (#1746)
why:
  Additional tables needed for the `CoreDB` object with separate
  key-value table and MPT.

details:
+ Stripped down copy of Aristo DB to have a similar look'n feel. Otherwise
  it is just a posh way for accessing `Table` objects or `RocksDB` data.
+ No unit tests yet, will be tested on the go.
2023-09-12 19:44:45 +01:00
Jordan Hrycaj 8e46953390
Aristo db state root repos and reorg (#1744)
* Reorg of distributed backend access

details:
  Now handled via API provided in `aristo_desc`.

* Rename `checkCache()` => `checkTop()`

why:
  Better naming for top layer cache checker

also:
  Provide cascaded fifos checker

* Provide `eq` directive for finding filter by exact filter ID (think block number)

* Some code beautification (for better code reading)

* State root reposition and reorg

details:
  Repositioning is supported by forking a new descriptor. Reorg is then
  accomplished by writing this forked state on the backend database.
2023-09-11 21:38:49 +01:00
Jordan Hrycaj 070b06f809
Implement backend filter mechanics (#1730)
details:
* Tested features
  + Successively store filters with increasing filter ID (think block number)
  + Cascading through fifos, deeper fifos merge groups of filters
  + Fetch squash merged N top fifos
  + Delete N top fifos, push back merged fifo, continue storing
  + Fifo chain is verified by hashes and filter ID
* Not tested yet
  + Real live scenario (using data dumps)
  + Real filter data (only shallow filters used so far)
2023-09-05 19:00:40 +01:00
Jordan Hrycaj 3936d4d0ad
Aristo db fixes n updates needed for filter fifo (#1728)
* Set scheduler state as part of the backend descriptor

details:
  Moved type definitions `QidLayoutRef` and `QidSchedRef` to
 `desc_structural.nim` so that it shares the same folder as
  `desc_backend.nim`

* Automatic filter queue table initialisation in backend

details:
  Scheduler can be tweaked or completely disabled

* Updated backend unit tests

details:
+ some code clean up/beautification, reads better now
+ disabled persistent filters so that there is no automated filter
   management which will be implemented next

* Prettify/update unit tests source code

details:
  Mostly replacing the `check()` paradigm by `xCheck()`

* Somewhat simplified backend type management

why:
  Backend objects are labelled with a `BackendType` symbol where the
  `BackendVoid` label is implicitly assumed for a `nil` backend object
  reference.

  To make it easier, a `kind()` function is used now applicable to
  `nil` references as well.

* Fix DB storage layout for filter objects

why:
  Need to store the filter ID with the object

* Implement reverse [] index on fifo

why:
  An integer index argument on `[]` retrieves the QueueID (label) of the
  fifo item while a QueueID argument on `[]` retrieves the index (so
  it is inverse to the former variant).

* Provide iterator over filters as fifo

why:
  This iterator goes along the cascased fifo structure (i.e. in
  historical order)
2023-09-05 14:57:20 +01:00
Jordan Hrycaj f177f5bf11
Aristo db extend filter storage scheduler api (#1725)
* Add backwards index `[]` operator into fifo

also:
  Need another maintenance instruction: The last overflow queue must
  irrevocably delete some item in order to make space for a new one.

* Add re-org scheduler

details:
  Generates instructions how to extract and merge some leading entries

* Add filter ID selector

details:
  This allows to find the next filter now newer that a given filter ID

* Message update
2023-08-30 18:08:39 +01:00
andri lim 96fb355efe
Implement EIP-4788: Beacon block root in the EVM (#1722)
* Implement EIP-4788: Beacon block root in the EVM

* EIP-4788: Fix genesis.parentBeaconBlockRoot initialization
2023-08-30 23:29:48 +07:00
Jordan Hrycaj 465d694834
Aristo db implement filter storage scheduler (#1713)
* Rename FilterID => QueueID

why:
  The current usage does not identify a particular filter but uses it as
  storage tag to manage it on the database (to be organised in a set of
  FIFOs or queues.)

* Split `aristo_filter` source into sub-files

why:
  Make space for filter management API

* Store filter queue IDs in pairs on the backend

why:
  Any pair will will describe a FIFO accessed by bottom/top IDs

* Reorg some source file names

why:
  The "aristo_" prefix for make local/private files is tedious to
  use, so removed.

* Implement filter slot scheduler

details:
  Filters will be stored on the database on cascaded FIFOs. When a FIFO
  queue is full, some filter items are bundled together and stored on the
  next FIFO.
2023-08-25 23:53:59 +01:00
Jordan Hrycaj 124ac064c6
Aristo db store filters on backend (#1703)
* Simplify RocksDB sub-tables iterator

* Implement `filter` storage on backend db

details:
  Unit tests working
2023-08-22 19:44:54 +01:00
Jordan Hrycaj b9a4fd3137
Aristo db update serialisation (#1700)
* Remove unused unit test sources

* Redefine and document serialised data records for Aristo backend

why:
  Unique record types determined by marker byte, i.e. the last byte of a
  serialisation record. This just needed some tweaking after adding new
  record types.
2023-08-21 19:18:06 +01:00
Jordan Hrycaj 445fa75251
Aristo db consolidate and clean up (#1699)
* Removed dedicated transcoder tests

why:
  will implicitely be provided by other tests:
  + encode/write -> hashify -> test_tx
  + decode/read -> merge raw nodes -> test_tx
  + de/blobfiy -> backend operations, taext_tx, test_backend, test_filter

* Clarify how the vertex ID generator state is accessed from the backend

why:
  This state is a list of unused vertex IDs. It was just stored somewhere
  on the backend which details were exposed when iterating over some
  sub-table(s).

  As there will be more such single information records, an admin
  sub-tables has been defined (formerly ID generator table) with dedicated
  access keys and type. Also, the iterator over the single ID generator
  state item has been removed. It must be accessed via the `get()`
  interface.

* Remove trailing space from file name

why:
  fixes windows bail out
2023-08-21 15:58:30 +01:00
Jordan Hrycaj 4c9141ffac
Aristo db implement filter serialisation for storage (#1695)
* Remove concept of empty/blind filters

why:
  Not needed. A non-existent filter is is coded as a nil reference.

* Slightly generalised backend iterators

why:
 * VertexID as key for the ID generator state makes no sense
 * there will be more tables addressed by non-VertexID keys

* Store serialised/blobified vertices on memory backend

why:
  This is more in line with the RocksDB backend so more appropriate
  for testing when comparing behaviour. For a speedy memory database,
  a backend-less variant should be used.

* Drop the `Aristo` prefix from names `AristoLayerRef`, etc.

* Suppress compiler warning

why:
  duplicate imports

* Add filter serialisation transcoder

why:
  Will be used as storage format
2023-08-18 20:46:55 +01:00
jangko 467e6fffa6
Implement EIP-6780: SELFDESTRUCT only in same transaction 2023-08-18 10:22:34 +07:00
Jordan Hrycaj 3078c207ca
Aristo db implement distributed backend access (#1688)
* Fix hashing algorithm

why:
  Particular case where a sub-tree is on the backend, linked by an
  Extension vertex to the top level.

* Update backend verification to report `dirty` top layer

* Implement distributed merge of backend filters

* Implement distributed backend access management

details:
  Implemented and tested as described in chapter 5 of the `README.md`
  file.
2023-08-17 14:42:01 +01:00
jangko 5644749de9
Fix bugs discovered by engine api simulator 2023-08-17 11:08:01 +07:00
Jordan Hrycaj 8f21cf48a8
Distributed access architecture specs (#1680)
why:
  Manage access to different MPTs via the same database (makes sense
  only if the MPTs are not too different.)
2023-08-14 19:39:10 +01:00
Jordan Hrycaj 01fe172738
Aristo db integrate hashify into tx (#1679)
* Renamed type `NoneBackendRef` => `VoidBackendRef`

* Clarify names: `BE=filter+backend` and `UBE=backend (unfiltered)`

why:
  Most functions used full names as `getVtxUnfilteredBackend()` or
  `getKeyBackend()`. After defining abbreviations (and its meaning) it
   seems easier to use `getVtxUBE()` and `getKeyBE()`.

* Integrate `hashify()` process into transaction logic

why:
  Is now transparent unless explicitly controlled.

details:
  Cache changes imply setting a `dirty` flag which in turn triggers
  `hashify()` processing in transaction and `pack()` directives.

* Removed `aristo_tx.exec()` directive

why:
  Inconsistent implementation, functionality will be provided with a
  different paradigm.
2023-08-11 18:23:57 +01:00
Jordan Hrycaj 09fabd04eb
Aristo db use filter betw backend and tx cache (#1678)
* Provide deep copy for each transaction layer

why:
  Localising changes. Selective deep copy was just overlooked.

* Generalise vertex ID generator state reorg function `vidReorg()`

why:
  makes it somewhat easier to handle when saving layers.

* Provide dummy back end descriptor `NoneBackendRef`

* Optional read-only filter between backend and transaction cache

why:
  Some staging area for accumulating changes to the backend DB. This
  will eventually be an access layer for emulating a backend with
  multiple/historic state roots.

* Re-factor `persistent()` with filter between backend/tx-cache => `stow()`

why:
  The filter provides an abstraction from the physically stored data on
  disk. So, there can be several MPT instances using the same disk data
  with different state roots. Of course, all the MPT instances should
  not differ too much for practical reasons :).

TODO:
  Filter administration tools need to be provided.
2023-08-10 21:01:28 +01:00
Jordan Hrycaj 71c91e2280
Aristo db refactor tx paradim (#1674)
* Better error handling

why:
  Bail out on some error as early as possible before any changes.

* Implement `fetch()` as opposite of `merge()`

rationale:
  In the `Aristo` realm, the action named `fetch()` and `merge()` indicate
  leaf value related actions on the MPT, while actions `get()` and `put()`
   handle vertex or hash key related operations that constitute the MPT.

* Re-factor `merge()` prototypes

why:
  The most used variant of `merge()` should have the simplest prototype.

* Persistent DB constructor needs to import `aristo/aristo_init/persistent`

why:
  Most applications use memory DB anyway. This avoids linking `-lrocksdb`
  or any other back end libraries by default.

* Re-factor transaction module

why:
  Got the paradigm wrong. The transaction descriptor did replace the
  database one but should be handled separately.
2023-08-07 18:45:23 +01:00
Jordan Hrycaj 221e6c9e2f
Unified database frontend integration (#1670)
* Nimbus folder environment update

details:
* Integrated `CoreDbRef` for the sources in the `nimbus` sub-folder.
* The `nimbus` program does not compile yet as it needs the updates
  in the parallel `stateless` sub-folder.

* Stateless environment update

details:
* Integrated `CoreDbRef` for the sources in the `stateless` sub-folder.
* The `nimbus` program compiles now.

* Premix environment update

details:
* Integrated `CoreDbRef` for the sources in the `premix` sub-folder.

* Fluffy environment update

details:
* Integrated `CoreDbRef` for the sources in the `fluffy` sub-folder.

* Tools environment update

details:
* Integrated `CoreDbRef` for the sources in the `tools` sub-folder.

* Nodocker environment update

details:
* Integrated `CoreDbRef` for the sources in the
  `hive_integration/nodocker` sub-folder.

* Tests environment update

details:
* Integrated `CoreDbRef` for the sources in the `tests` sub-folder.
* The unit tests compile and run cleanly now.

* Generalise `CoreDbRef` to any `select_backend` supported database

why:
  Generalisation was just missed due to overcoming some compiler oddity
  which was tied to rocksdb for testing.

* Suppress compiler warning for `newChainDB()`

why:
  Warning was added to this function which must be wrapped so that
  any `CatchableError` is re-raised as `Defect`.

* Split off persistent `CoreDbRef` constructor into separate file

why:
  This allows to compile a memory only database version without linking
  the backend library.

* Use memory `CoreDbRef` database by default

detail:
 Persistent DB constructor needs to import `db/core_db/persistent

why:
 Most tests use memory DB anyway. This avoids linking `-lrocksdb` or
 any other backend by default.

* fix `toLegacyBackend()` availability check

why:
  got garbled after memory/persistent split.

* Clarify raw access to MPT for snap sync handler

why:
  Logically, `kvt` is not the raw access for the hexary trie (although
  this holds for the legacy database)
2023-08-04 12:10:09 +01:00
Jordan Hrycaj d3d172edee
Replace `CoreDbRef` oo-methods by closures (#1668)
why:
* Resolves some compiler coughing when it bails out on persitent
  db constructor inside `test()` caluses (works perfectly outside.)
* API looks cleaner and better to maintain for the price of slightly
  more work at the backend
2023-08-02 21:46:41 +01:00
Jordan Hrycaj 322f1c2e9e
Unified database frontend (#1661)
* Remove 32bit os support from `custom_network` unit test

also:
* Fix compilation annoyance #1648
* Fix unit test on Kiln (changed `merge` logic?)

* Hide unused sources do not compile

why:
* Get them out of the way before major update
* Import and function prototype mismatch -- maybe some changes got out
  of scope.

* Re-implemented `db_chain` as `core_db`

why:
  Hiding `TrieDatabaseRef` and `HexaryTrie` by default allows to replace
  the current db wrapper by some other one, e.g. Aristo

* Support compiler exception warnings for CoreDbRef base methods.

* Allow `pairs()` iterator on all memory based key-value tables

why:
  Previously only available for capture recorder.

* Backport `chain_db.nim` changes into its re-implementation `core_apps.nim`

* Fix exception annotation
2023-07-31 14:43:38 +01:00
jangko 81b7a8c682
graphql: add Shanghai and Cancun fields 2023-07-29 20:16:03 +07:00
Jordan Hrycaj 56d5c382d7
Aristo db traversal helpers (#1638)
* Misc fixes

detail:
* Fix de-serialisation for account leafs
* Update node recovery from unit tests

* Remove `LegacyAccount` from `PayloadRef` object

why:
  Legacy accounts use a hash key as storage root which is detrimental
  to the working of the Aristo database which uses a vertex ID.

* Dissolve `hashify_helper` into `aristo_utils` and `aristo_transcode`

why:
  Functions are of general interest so they should live in first level
  code files.

* Added left/right iterators over leaf nodes

* Some helper/wrapper functions that might be useful
2023-07-13 00:03:14 +01:00
Jordan Hrycaj 93a72025a1
Extended data Payload specs for the backend. (#1630)
why:
  For the main tree with root vertex ID 1, the leaf nodes hold the
  account data. These accounts may link to sub trees the storage root
  node ID of which must be registered here. There is no reverse key
  lookup on the backend.

note:
  These definitions are experimental. Also, there are some tests missing
  for validating Payload data conversions.
2023-07-05 21:27:48 +01:00