Commit Graph

392 Commits

Author SHA1 Message Date
Jordan Hrycaj 3078c207ca
Aristo db implement distributed backend access (#1688)
* Fix hashing algorithm

why:
  Particular case where a sub-tree is on the backend, linked by an
  Extension vertex to the top level.

* Update backend verification to report `dirty` top layer

* Implement distributed merge of backend filters

* Implement distributed backend access management

details:
  Implemented and tested as described in chapter 5 of the `README.md`
  file.
2023-08-17 14:42:01 +01:00
jangko 5644749de9
Fix bugs discovered by engine api simulator 2023-08-17 11:08:01 +07:00
Jordan Hrycaj 8f21cf48a8
Distributed access architecture specs (#1680)
why:
  Manage access to different MPTs via the same database (makes sense
  only if the MPTs are not too different.)
2023-08-14 19:39:10 +01:00
Jordan Hrycaj 01fe172738
Aristo db integrate hashify into tx (#1679)
* Renamed type `NoneBackendRef` => `VoidBackendRef`

* Clarify names: `BE=filter+backend` and `UBE=backend (unfiltered)`

why:
  Most functions used full names as `getVtxUnfilteredBackend()` or
  `getKeyBackend()`. After defining abbreviations (and its meaning) it
   seems easier to use `getVtxUBE()` and `getKeyBE()`.

* Integrate `hashify()` process into transaction logic

why:
  Is now transparent unless explicitly controlled.

details:
  Cache changes imply setting a `dirty` flag which in turn triggers
  `hashify()` processing in transaction and `pack()` directives.

* Removed `aristo_tx.exec()` directive

why:
  Inconsistent implementation, functionality will be provided with a
  different paradigm.
2023-08-11 18:23:57 +01:00
Jordan Hrycaj 09fabd04eb
Aristo db use filter betw backend and tx cache (#1678)
* Provide deep copy for each transaction layer

why:
  Localising changes. Selective deep copy was just overlooked.

* Generalise vertex ID generator state reorg function `vidReorg()`

why:
  makes it somewhat easier to handle when saving layers.

* Provide dummy back end descriptor `NoneBackendRef`

* Optional read-only filter between backend and transaction cache

why:
  Some staging area for accumulating changes to the backend DB. This
  will eventually be an access layer for emulating a backend with
  multiple/historic state roots.

* Re-factor `persistent()` with filter between backend/tx-cache => `stow()`

why:
  The filter provides an abstraction from the physically stored data on
  disk. So, there can be several MPT instances using the same disk data
  with different state roots. Of course, all the MPT instances should
  not differ too much for practical reasons :).

TODO:
  Filter administration tools need to be provided.
2023-08-10 21:01:28 +01:00
Jordan Hrycaj 71c91e2280
Aristo db refactor tx paradim (#1674)
* Better error handling

why:
  Bail out on some error as early as possible before any changes.

* Implement `fetch()` as opposite of `merge()`

rationale:
  In the `Aristo` realm, the action named `fetch()` and `merge()` indicate
  leaf value related actions on the MPT, while actions `get()` and `put()`
   handle vertex or hash key related operations that constitute the MPT.

* Re-factor `merge()` prototypes

why:
  The most used variant of `merge()` should have the simplest prototype.

* Persistent DB constructor needs to import `aristo/aristo_init/persistent`

why:
  Most applications use memory DB anyway. This avoids linking `-lrocksdb`
  or any other back end libraries by default.

* Re-factor transaction module

why:
  Got the paradigm wrong. The transaction descriptor did replace the
  database one but should be handled separately.
2023-08-07 18:45:23 +01:00
Jordan Hrycaj 221e6c9e2f
Unified database frontend integration (#1670)
* Nimbus folder environment update

details:
* Integrated `CoreDbRef` for the sources in the `nimbus` sub-folder.
* The `nimbus` program does not compile yet as it needs the updates
  in the parallel `stateless` sub-folder.

* Stateless environment update

details:
* Integrated `CoreDbRef` for the sources in the `stateless` sub-folder.
* The `nimbus` program compiles now.

* Premix environment update

details:
* Integrated `CoreDbRef` for the sources in the `premix` sub-folder.

* Fluffy environment update

details:
* Integrated `CoreDbRef` for the sources in the `fluffy` sub-folder.

* Tools environment update

details:
* Integrated `CoreDbRef` for the sources in the `tools` sub-folder.

* Nodocker environment update

details:
* Integrated `CoreDbRef` for the sources in the
  `hive_integration/nodocker` sub-folder.

* Tests environment update

details:
* Integrated `CoreDbRef` for the sources in the `tests` sub-folder.
* The unit tests compile and run cleanly now.

* Generalise `CoreDbRef` to any `select_backend` supported database

why:
  Generalisation was just missed due to overcoming some compiler oddity
  which was tied to rocksdb for testing.

* Suppress compiler warning for `newChainDB()`

why:
  Warning was added to this function which must be wrapped so that
  any `CatchableError` is re-raised as `Defect`.

* Split off persistent `CoreDbRef` constructor into separate file

why:
  This allows to compile a memory only database version without linking
  the backend library.

* Use memory `CoreDbRef` database by default

detail:
 Persistent DB constructor needs to import `db/core_db/persistent

why:
 Most tests use memory DB anyway. This avoids linking `-lrocksdb` or
 any other backend by default.

* fix `toLegacyBackend()` availability check

why:
  got garbled after memory/persistent split.

* Clarify raw access to MPT for snap sync handler

why:
  Logically, `kvt` is not the raw access for the hexary trie (although
  this holds for the legacy database)
2023-08-04 12:10:09 +01:00
Jordan Hrycaj d3d172edee
Replace `CoreDbRef` oo-methods by closures (#1668)
why:
* Resolves some compiler coughing when it bails out on persitent
  db constructor inside `test()` caluses (works perfectly outside.)
* API looks cleaner and better to maintain for the price of slightly
  more work at the backend
2023-08-02 21:46:41 +01:00
Jordan Hrycaj 322f1c2e9e
Unified database frontend (#1661)
* Remove 32bit os support from `custom_network` unit test

also:
* Fix compilation annoyance #1648
* Fix unit test on Kiln (changed `merge` logic?)

* Hide unused sources do not compile

why:
* Get them out of the way before major update
* Import and function prototype mismatch -- maybe some changes got out
  of scope.

* Re-implemented `db_chain` as `core_db`

why:
  Hiding `TrieDatabaseRef` and `HexaryTrie` by default allows to replace
  the current db wrapper by some other one, e.g. Aristo

* Support compiler exception warnings for CoreDbRef base methods.

* Allow `pairs()` iterator on all memory based key-value tables

why:
  Previously only available for capture recorder.

* Backport `chain_db.nim` changes into its re-implementation `core_apps.nim`

* Fix exception annotation
2023-07-31 14:43:38 +01:00
jangko 81b7a8c682
graphql: add Shanghai and Cancun fields 2023-07-29 20:16:03 +07:00
Jordan Hrycaj 56d5c382d7
Aristo db traversal helpers (#1638)
* Misc fixes

detail:
* Fix de-serialisation for account leafs
* Update node recovery from unit tests

* Remove `LegacyAccount` from `PayloadRef` object

why:
  Legacy accounts use a hash key as storage root which is detrimental
  to the working of the Aristo database which uses a vertex ID.

* Dissolve `hashify_helper` into `aristo_utils` and `aristo_transcode`

why:
  Functions are of general interest so they should live in first level
  code files.

* Added left/right iterators over leaf nodes

* Some helper/wrapper functions that might be useful
2023-07-13 00:03:14 +01:00
Jordan Hrycaj 93a72025a1
Extended data Payload specs for the backend. (#1630)
why:
  For the main tree with root vertex ID 1, the leaf nodes hold the
  account data. These accounts may link to sub trees the storage root
  node ID of which must be registered here. There is no reverse key
  lookup on the backend.

note:
  These definitions are experimental. Also, there are some tests missing
  for validating Payload data conversions.
2023-07-05 21:27:48 +01:00
Jordan Hrycaj ccf639fc3c
Aristo db transaction based interface (#1628)
* Provide transaction based interface for standard operations

* Provide unit tests for new Aristo interface using transactions

details:
  These new tests combine and replace several single-purpose tests.
  The now unused test sources will be kept for a while to be eventually
  removed.
2023-07-05 14:50:11 +01:00
Jordan Hrycaj ff6673beac
Aristo db tidy up a bit (#1625)
* Slightly tighten some self-check conditions

* Redefined the database descriptor object as reference (to the object)

why:
  The upcoming transaction wrapper will work with a database reference
  rather than the object itself

* Append state before `save()` to the Aristo descriptor

why:
  This stae was previously returned by the function. Appending it to
  a field of the Aristo descriptor seems easier to handle.
2023-07-04 19:24:03 +01:00
Jordan Hrycaj dd1c8ed6f2
Aristo db update delete functionality (#1621)
* Fix missing branch checks in transcoder

why:
  Symmetry problem. `Blobify()` allowed for encoding degenerate branch
  vertices while `Deblobify()` rejected decoding wrongly encoded data.

* Update memory backend so that it rejects storing bogus vertices.

why:
  Error behaviour made similar to the rocks DB backend.

* Make sure that leaf vertex IDs are not repurposed

why:
  This makes it easier to record leaf node changes

* Update error return code for next()/right() traversal

why:
  Returning offending vertex ID (besides error code) helps debugging

* Update Merkle hasher for deleted nodes

why:
  Not implemented, yet

also:
  Provide cache & backend consistency check functions. This was
  partly re-implemented from `hashifyCheck()`

* Simplify some unit tests

* Fix delete function

why:
  Was conceptually wrong
2023-06-30 23:22:33 +01:00
andri lim 26a8759c34
implementation of EIP-4844: Shard Blob Transactions (#1440)
* EIP-4844: add pointEvaluation precompiled contract

* EIP-4844: validate transaction and block header

* EIP-4844: implement DataHash Op Code

* EIP-4844: txPool support excessDataGas calculation

* EIP-4844: make sure tx produce correct txHash

* EIP-4844: node should not automatically broadcast blob tx to it's peers

* EIP-4844: add test cases

* EIP-4844: add EIP-4844 support to t8n tool

* EIP-4844: update nim-eth to branch eip-4844

* fix t8n transaction decoding

* add t8n test data

* EIP-4844: fix blobHash opcode

* disable blobHash test when evmc_enable
2023-06-24 20:56:44 +07:00
jangko 6544adf360
implement EIP-1153: Transient storage
new EVM opcodes:
- TLOAD  0xb3
- TSTORE 0xb4
2023-06-23 14:04:36 +07:00
Jordan Hrycaj 15cc9f962e
Aristo db update vertex caching when merging (#1606)
* Added missing deferred cleanup directive to sub-test functions

why:
  Rocksdb keeps the files locked for a short while leading to errors. This
  was previously solved my using different db sub-directories

* Provide vertex deep-copy function globally.

why:
  is just handy

* Avoid unnecessary vertex caching when merging proof nodes

also:
  Run all merge tests on the rocksdb backend
  Previously, proof node tests were run without backend
2023-06-22 20:21:33 +01:00
Jordan Hrycaj 83dbe87159
Aristo db update foreground caching (#1605)
* Fix vertex ID generator state handling for rocksdb backend

why:
 * Key error in walk iterator
 * Needs to be loaded when opening the database

* Use non-zero sub-table prefixes for rocksdb

why:
  Handy for debugging

* Fix error code for missing key on rocksdb backend

why:
  Previously returned `VOID_HASH_KEY` rather than `GetKeyNotFound`

* Explicitly copy vertex data between internal table and function/result argument

why:
  Function argument or return reference may still refer to the same data
  object.

* Updated error symbols

why:
  Error symbol names for the hike module now start with the prefix `Hike`.

* Write back modified branch node into local top layer cache

why:
  With the backend available, the source of the branch node references
  might not be the top layer cache. So any change must be explicitely
  recorded.
2023-06-22 12:13:24 +01:00
Jordan Hrycaj 4b66f93274
Aristo db with storage backends (#1603)
* Generalised Aristo DB constructor for any type of backend

details:
  * Records to be deleted are represented as key-void (rather than
    key-value) pairs by the put-function arguments
  * Allow direct driver access, iterators as example implementation and
    for testing.

* Provide backend storage interface

details:
  Stores the top layer onto backend tables

* Implemented Rocks DB backend

details:
  Transaction based `put()` functionality
  Iterators (based on direct RocksDB access)
2023-06-20 14:26:25 +01:00
Jordan Hrycaj d7f40516a7
Detach from snap/sync declarations & definitions (#1601)
why:
  Tests and some basic components were originally borrowed from the
  snap/sync implementation. These have fully been re-implemented.
2023-06-12 19:16:03 +01:00
Jordan Hrycaj 0308dfac4f
Aristo db address sup trie items properly (#1600)
* Fix include

why:
  Eth67 not default yet so that got missed

* Rename `LeafKey` => `LeafTie`

why:
  Name is a pen picture of what this object is for. Also, it avoids the
  ubiquitous term `key`.

* Provided `getOrVoid()` wrapper for `getOrDefault()`

also:
  Provide `isValid()` syntactic sugar for `.isNil.not`, `!= 0` etc.
  Reorg descriptor source, split into sub-sources

* Bundled `NodeKey` objects with root ID and called it `HashLabel`

why:
  `NodeKey` (aka repurposed Hash265) objects are unique only within a
  particular sub-trie (e.g. storage slots) which are kept separated
  (i.e non-interleaved) by design. This is not applied to the backend
  as the map VertexID->NodeKey labelling the nodes needs not be injective.

  For the in-memory database (transaction) layers, the injective map
  VertexID->(VertexID,NodeKey) is used where the first field of the image
  tuple is the root ID of the sub-trie the `NodeKey` object is valid. So
  identical storage tries for different accounts can be represented.
2023-06-12 14:48:47 +01:00
Jordan Hrycaj 932a2140f2
Aristo db supporting forest and layered tx architecture (#1598)
* Exclude some storage tests

why:
  These test running on external dumps slipped through. The particular
  dumps were reported earlier as somehow dodgy.

  This was changed in `#1457` but having a second look, the change on
  hexary_interpolate.nim(350) might be incorrect.

* Redesign `Aristo DB` descriptor for transaction based layers

why:
  Previous descriptor layout made it cumbersome to push/pop
  database delta layers.

  The new architecture keeps each layer with the full delta set
  relative to the database backend.

* Keep root ID as part of the `Patricia Trie` leaf path

why;
  That way, forests are supported
2023-06-09 12:17:37 +01:00
jangko 67aaf92c1d
bump submodules 2023-06-07 18:12:02 +07:00
Jordan Hrycaj 11bb33d0bc
Added delete fuctionality (#1596) 2023-06-02 20:21:46 +01:00
Jordan Hrycaj 099444ab3f
Aristo db fixes after storage slots dump tests added (#1595)
* Fix missing Merkle key removal in `merge()`

* Accept optional root hash argument in `hashify()`

why:
  For importing a full database, there will be no proof data except the
  root key. So this can be used to check and set the root key in the
  database descriptor.

also:
  Associate vertex ID to `hashify()` error return code

* Added Aristo Trie traversal function

why:
 * step along leaf vertices in sorted order
 * tree/trie consistency checks when debugging

* Enabled storage slots test data for Aristo DB
2023-06-02 11:04:29 +01:00
Jordan Hrycaj 2fc349feb9
Aristo db merkle hashify functionality added (#1593)
* Keep vertex ID generator state with each db-layer

why:
  The vertex ID generator state is part of the difference to the below
  layer

* Move otherwise unused source to test directory

* Add Merkle hash generator

also:
  * Verification facility for debugging
  * Empty Merkle key hashes encoded as `EMPTY_ROOT_HASH`
2023-05-30 22:21:15 +01:00
Jordan Hrycaj cd78458123
Add items to `Aristo Trie` database (#1586)
details:
1. Merging a leaf vertex merges a `Patricia Trie` path (while
   adding/modiying vertices) and adds a leaf node with payload
2. Merging a Merkel node merges a single vertex to the `Patricia Trie`
   and registers merkel hashes
3. Action 2 can be used before action 1 in order to construct a
   Merkel proof as required for handling `snap/1` data.
4. Unit tests show that action 3 is benign for now :)
2023-05-30 12:47:47 +01:00
jangko fb3b341aea store withdrawals along with txs in block body 2023-05-23 11:36:20 +03:00
Jordan Hrycaj ff0fc98fdf
Multi layer architecture 4 aristo db (#1581)
* Cosmetics, renamed fields (eVtx, bVtx) -> (eVid, bVid)

* Multilayered delta architecture for Aristo DB

details:
  Any VertexID or data retrieval needs to go down the rabbit hole and
  fetch/get/manipulate the bottom layer -- even without explicit
  backend.

* Direct reference to backend from top-level layer

why:
  Some services as the vid management needs to be synchronised among all
  layers. So access is optimised.
2023-05-14 18:43:01 +01:00
Jordan Hrycaj 605739ef4c
Experimental MP-trie (#1573)
* Experimental MP-trie

why:
  Deleting records is a infeasible with the current structure

* Added vertex ID recycling management

Todo:
  Provide some unit tests

* DB layout update

why:
  Main news is the separation of `Merkel` hashes into an extra table.

details:
  The code fragments cover conversion between compact MPT records and
  Aristo DB records as well as some rudimentary cache handling for
  the `Merkel` hashes (i.e. the extra table entries.)

todo:
  Add some simple unit test for the descriptor record (currently used
  for vertex ID management, only.)

* Updated vertex ID recycling management

details:
  added simple unit tests (mainly testing ABI)

* docu update
2023-05-11 15:25:29 +01:00
Adam Spitz 7f56e90654
Revert "Some work on speculative execution (#1552)" (#1561)
This reverts commit ddbdf34c3d.
2023-04-24 16:59:38 -04:00
Adam Spitz ddbdf34c3d
Some work on speculative execution (#1552)
* Recreating some of the speculative-execution code.

Not really using it yet. Also there's some new inefficiency in
memory.nim, but it's fixable - just haven't gotten around to it yet.

The big thing introduced here is the idea of "cells" for stack,
memory, and storage values. A cell is basically just a Future (though
there's also the option of making it an Identity - just a simple
distinct wrapper around a value - if you want to turn off the
asynchrony).

* Bumped nim-eth.

* Cleaned up a few comments.

* Bumped nim-secp256k1.

* Oops.

* Fixing a few compiler errors that show up with EVMC enabled.
2023-04-22 10:17:37 -04:00
Jordan Hrycaj 0a3bc102eb
Pre functional snap to full sync (#1546)
* Update sync scheduler pool mode

why:
  The pool mode allows to loop over active peers one after another. This
  is ideal for soft re-starting peers. As this is a two tier experience
  (start/stop, setup/release) the loop must be run twice. This is
  controlled by a more rigid re-definition of how to use the `poolMode`
  flag.

* Mitigate RLP serialiser deficiency

why:
  Currently, serialising the `BlockBody` in not conevrtible and need
  to be checked in the `eth` module. Currently a local fix for the
  wire protocol applies. Unit tests will stay (after this local solution
  will have been removed.)

* Code cosmetics and massage

details:
  Main part is `types.toStr()` as a unified function for logging block
  numbers.

* Allow to use a logical genesis replacement (start of history)

why:
  Snap sync will set up an arbitrary pivot at a block number different
  from zero. In fact, the higher the block number the better.

details:
  A non-genesis start of history will currently only affect the score
  values which were derived from the difficulty.

* Provide function to store the snap pivot block header in chain db

why:
  Together with the start of history facility, this allows to proceed
  with full syncing once snap has finished.

details:
  Snap db storage was switched from a sub-tables to the flat chain db.

* Provide database completeness and sanity checker

details:
  For debugging on smaller databases, only

* Implement snap -> full sync switch
2023-04-14 23:28:57 +01:00
Adam Spitz c58d83efd5
More work on stateless mode (#1535)
* Reviving more of the code for stateless mode.

* Made asyncProcessTransaction.

* More work on stateless mode.
2023-04-12 08:39:11 -04:00
jangko a31db89e0e
fix t8n tool bug and accounts cache bug
t8n: a silly bug contract address generator, should use original
tx nonce instead of read the nonce from sender address in state db.
Although in EVM contract address generated by reading nonce from state db
is correct, outside EVM that nonce value might have been modified,
thus generating incorrect contract address.

accounts cache: when clearing account storage, the originalValue
cache is not cleared, only the storageRoot set to empty storage root,
this will cause getStorage and getCommitedStorage return wrong value
if the originalValue cache contains old value.
2023-03-23 11:07:30 +07:00
jangko 0b97f5a2e6
t8n tools support withdrawals records 2023-03-21 18:36:22 +07:00
jangko 2a3c67b4be
refactor touched accounts, selfdestruct, and log
simplify EVM and delegete those things to accounts cache.
also no more manual state clearing, accounts cache will be
responsible for both collecting touched account and perform
state clearing.
2023-03-21 08:14:06 +07:00
jangko 633f135c72
align accounts cache with EIP158/161
Some nomenclature used in accounts cache are not what described
in EIP158/161, therefore causing confusion and introduce bugs.
Now it should be fixed.
2023-03-18 18:13:06 +07:00
Adam Spitz ec2bd4a9c5
More work on withdrawals (#1503)
* Gwei conversion should use u256 because u64 can overflow.

* Make withdrawals follow the EIP-158 state-clearing rules.

(i.e. Empty accounts should be deleted.)

* Allow the zero address in normalizeNumber.

(Necessary for one of the new withdrawals-related tests.)
2023-03-16 16:34:47 -04:00
Adam Spitz 11771f8444
Recreating some of the old stateless code that had bitrotted. (#1493)
Note that we're not really using it yet; I'm just refactoring and
adding new code, working in small steps.
2023-03-10 17:16:42 -05:00
Adam Spitz c921135e2e
Refactored to introduce distinct types AccountsTrie and StorageTrie. (#1490)
This is in preparation for working on stateless mode; it's useful to
have a clearer and more type-safe interface for accessing accounts
versus slots.
2023-03-10 13:42:37 -05:00
Jordan Hrycaj 6b9f3c9ac5
Silence compiler gossip after nim upgrade cont1 (#1455)
* Silence some compiler gossip -- part 5, common

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Silence some compiler gossip -- part 6, db, rpc, utils

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Silence some compiler gossip -- part 7, randomly collected source files

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Silence some compiler gossip -- part 8, assorted tests

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Clique update

why:
  More impossible exceptions (undoes temporary fix from previous PR)
2023-01-31 01:32:17 +00:00
Kim De Mey 8523d8b40e
Bump submodules and fix related issues (#1448)
* Updated to the latest nim-eth, nim-rocksdb, nim-web3

* Bump nimbus-eth2 module and fix related issues

Temporarily disabling Portal beacon light client network as it is
a lot of copy pasted code that did not yet take into account
forks. This will require a bigger rework and was not yet tested
in an actual network anyhow.

* More nimbus fixes after module bumps

---------

Co-authored-by: Adam Spitz <adamspitz@status.im>
Co-authored-by: jangko <jangko128@gmail.com>
2023-01-27 15:57:48 +01:00
Kim De Mey a669b51ec5
Bump Nim to 1.6 and resolve the related issues (#1445)
Two unresolved items currently:
- Three tests that are temporarily disabled as they fail in the
macro_assembler code, which seems to be due to an ambigious
identifier Stop (Ops and chronos ServerCommand enum).
- i386 CI disabled as it fails at Nim compilation already. Failed
tests where already ignored for this target.
2023-01-26 13:37:19 +01:00
Kim De Mey 9b1193c402
Revert "Updated to the latest nim-eth, nim-rocksdb, nim-web3. (#1441)" (#1443)
This reverts commit 1c1a3352b3.
2023-01-23 10:50:17 +01:00
Adam Spitz 1c1a3352b3
Updated to the latest nim-eth, nim-rocksdb, nim-web3. (#1441) 2023-01-21 15:04:22 +01:00
jangko 13e5b1a76b
fix persistHeaderToDb, allow POS block header to be canonical 2022-12-10 09:02:30 +07:00
jangko 94a94c5b65 implement better hardfork management 2022-12-02 13:51:42 +07:00
Jordan Hrycaj bba1bea4c8
Snap sync state save (#1302)
* Piecemeal trie inspection

details:
  Trie inspection will stop after maximum number of nodes visited.
  The inspection can be resumed using the returned state from the
  last session.

why:
  This feature allows for task switch between `piecemeal` sessions.

* Extract pivot helper code from `worker.nim` => `pivot_helper.nim`

* Accounts import will now return dangling paths from `proof` nodes

why:
  With proper bookkeeping, this can be used to start healing without
  analysing the the probably full trie.

* Update `unprocessed` account range handling

why:
  More generally, the API of a pairs of unprocessed intervals favours
  the first set and not before that is exhausted the second set comes
  into play.

  This was unfortunately implemented which caused the ranges to be
  unnecessarily fractioned. Now the number of range interval typically
  remains in the lower single digit numbers.

* Save sync state after end of downloading some accounts

details:
  restore/resume to be implemented later
2022-11-16 23:51:06 +00:00
Jordan Hrycaj e14fd4b96c
Prep for full sync after snap make 6 (#1291)
* Update log ticker, using time interval rather than ticker count

why:
  Counting and logging ticker occurrences is inherently imprecise. So
  time intervals are used.

* Use separate storage tables for snap sync data

* Left boundary proof update

why:
  Was not properly implemented, yet.

* Capture pivot in peer worker (aka buddy) tasks

why:
  The pivot environment is linked to the `buddy` descriptor. While
  there is a task switch, the pivot may change. So it is passed on as
  function argument `env` rather than retrieved from the buddy at
  the start of a sub-function.

* Split queues `fetchStorage` into `fetchStorageFull` and `fetchStoragePart`

* Remove obsolete account range returned from `GetAccountRange` message

why:
  Handler returned the wrong right value of the range. This range was
  for convenience, only.

* Prioritise storage slots if the queue becomes large

why:
  Currently, accounts processing is prioritised up until all accounts
  are downloaded. The new prioritisation has two thresholds for
  + start processing storage slots with a new worker
  + stop account processing and switch to storage processing

also:
  Provide api for `SnapTodoRanges` pair of range sets in `worker_desc.nim`

* Generalise left boundary proof for accounts or storage slots.

why:
  Detailed explanation how this works is documented with
  `snapdb_accounts.importAccounts()`.

  Instead of enforcing a left boundary proof (which is still the default),
  the importer functions return a list of `holes` (aka node paths) found in
  the argument ranges of leaf nodes. This in turn is used by the book
   keeping software for data download.

* Forgot to pass on variable in function wrapper

also:
  + Start healing not before 99% accounts covered (previously 95%)
  + Logging updated/prettified
2022-11-08 18:56:04 +00:00
jangko f732cba0c3
add additional constructor to BaseChainDB 2022-10-25 11:01:31 +07:00
Kim De Mey 74a83c1229
Bump nim-eth and remove all now unneeded p2p related imports (#1273) 2022-10-20 10:34:59 +02:00
jangko 16bc2de1cf
update EF test fixtures and fixes to pass all tests 2022-10-03 16:41:32 +07:00
jangko 86f6d284aa
initial beacon sync skeleton implementation 2022-09-17 09:08:55 +07:00
Jacek Sieka c2ed731fa5
eth: adapt to smaller eth_types (#1210) 2022-09-03 20:15:35 +02:00
Jordan Hrycaj 7d7e26d45f
Experimental bulk loader tests (#1187)
why:
  Rocksdb bulk loading might provide a slight advantage when loading
  larger data sets into the system
2022-08-12 16:42:07 +01:00
Jordan Hrycaj 5f0e89a41e
Snap accounts bulk import preparer (#1183)
* Provided common scheduler API, applied to `full` sync

* Use hexary trie as storage for proofs_db records

also:
 + Store metadata with account for keeping track of account state
 + add iterator over accounts

* Common scheduler API applied to `snap` sync

* Prepare for accounts bulk import

details:
+ Added some ad-hoc checks for proving accounts data received from the
  snap/1 (will be replaced by proper database version when ready)
+ Added code that dumps some of the received snap/1 data into a file
  (turned of by default, see `worker_desc.nim`)
2022-08-04 09:04:30 +01:00
jangko 2b4baff8ec
move block validation from execution payload generator to engine api 2022-07-04 20:36:30 +07:00
jangko 709d8ef255
fix markCanonicalChain bug 2022-07-04 19:32:14 +07:00
Jordan Hrycaj 134fe26997
Store proved snap accounts (#1145)
* Relocated `IntervalSets` to nim-stew repo

* Accumulate accounts on temporary kv-DB

why:
  Explore the data as returned from snap/1. Will be converted to a
  `eth/db` next.

details:
  Verify and accumulate per/state-root accounts downloaded via snap.

also:
  Some unit tests

* Replace `Table` by `TrieDatabaseRef` for accounts accumulator

* update ticker statistics

details:
  mean/variance based counter update

* allow persistent db for proved accounts

* rebase, and globally activate unit test

* fix statistics
2022-07-01 12:42:17 +01:00
jangko e274b347ae
add haveBlockAndState to BaseChainDB and use it in engine api 2022-06-30 16:11:43 +07:00
KonradStaniec fdb048ed21
Add getLogs implementation (#1143)
* Add eth_getLogs json-rpc endpoint in Fluffy and Nimbus
2022-06-29 17:44:08 +02:00
jangko 4a50b00c37
rpc: change getStorageAt param type to HexDataStr
if using HexQuantityStr, it will fail when given hex string with leading zeros
2022-06-27 17:47:59 +07:00
jangko b80eca0718
json-rpc: able to query finalized block and safe block header
engine-api:
- store safe block hash and finalized block hash

engine-api test:
- fix test case related to safe block hash and finalized block hash
2022-06-27 11:15:54 +07:00
jangko 69a1000d77
more engine api test 2022-06-15 15:56:45 +07:00
jangko 6fcd63cb58
modify hive simulators to run in CI 2022-04-21 12:01:18 +07:00
jangko f2f204293e
first step into styleCheck fixes 2022-04-14 08:39:50 +07:00
Jacek Sieka 1d6a9951d6
move rocksdb support to eth1 (#927)
* move rocksdb support to eth1

only used here / causes unnecessary build deps
2022-04-06 07:28:19 +02:00
jangko c704626ea8
mark the new chain as canonical when calling db.setHead
the new chain marked as canonical is from the new head down to
forking point. most of the time, the forking point is the parent
itself.
2022-03-26 13:23:08 +07:00
jangko e628c8dc88
temporary fix db.setHead before actual fix 2022-03-25 14:01:32 +07:00
jangko d41a3dedf0
write block ttd in `persistHeaderToDbWithoutSetHead` 2022-03-08 21:47:51 +07:00
jangko f782327fcf
reimplement engine API rpc kiln spec v2 2022-03-03 11:28:18 +07:00
jangko 73e28694b5
move ttd from vm/state to chain_db 2022-03-03 08:58:55 +07:00
jangko 9a1c8fc779
refactor genesis.nim 2022-03-03 08:28:00 +07:00
jangko 28cdfcaf6b
fix EIP-4399 'random' opcode
- fix previous implementation of EIP-4399
- now `random` opcode can be used with evmc_enabled
2022-02-08 20:23:40 +07:00
Zahary Karadjov 137eb97766 Initial implementation of the merge spec
Includes a simple test harness for the merge interop M1 milestone

This aims to enable connecting nimbus-eth2 to nimbus-eth1 within
the testing protocol described here:

https://github.com/status-im/nimbus-eth2/blob/amphora-merge-interop/docs/interop_merge.md

To execute the work-in-progress test, please run:

In terminal 1:
tests/amphora/launch-nimbus.sh

In terminal 2:
tests/amphora/check-merge-test-vectors.sh
2022-01-24 09:44:39 +02:00
Jordan Hrycaj 261c0b51a7
Redesign of BaseVMState descriptor (#923)
* Redesign of BaseVMState descriptor

why:
  BaseVMState provides an environment for executing transactions. The
  current descriptor also provides data that cannot generally be known
  within the execution environment, e.g. the total gasUsed which is
  available not before after all transactions have finished.

  Also, the BaseVMState constructor has been replaced by a constructor
  that does not need pre-initialised input of the account database.

also:
  Previous constructor and some fields are provided with a deprecated
  annotation (producing a lot of noise.)

* Replace legacy directives in production sources

* Replace legacy directives in unit test sources

* fix CI (missing premix update)

* Remove legacy directives

* chase CI problem

* rebased

* Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation

why:
  Constructing a new 'AccountsCache' descriptor can be avoided sometimes
  when the current state root is properly positioned already. Such a
  feature existed already as the update function 'initStateDB()' for the
  'BaseChanDB' where the accounts cache was linked into this desctiptor.

  The function 'initStateDB()' was removed and re-implemented into the
  'BaseVmState' constructor without optimisation. The old version was of
  restricted use as a wrong accounts cache state would unconditionally
  throw an exception rather than conceptually ask for a remedy.

  The optimised 'BaseVmState' re-initialisation has been implemented for
  the 'persistBlocks()' function.

also:
  moved some test helpers to 'test/replay' folder

* Remove unused & undocumented fields from Chain descriptor

why:
  Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
jangko baf508f6ae
move stateDB from VMState to chainDB
previously, every time the VMState was created, it will also create
new stateDB, and this action will nullify the advantages of cached accounts.

the new changes will conserve the accounts cache if the executed blocks
are contiguous. if not the stateDB need to be reinited.

this changes also allow rpcCallEvm and rpcEstimateGas executed properly
using current stateDB instead of creating new one each time they are called.
2021-10-28 18:57:08 +07:00
jangko 44394d9ffd
fixes nimbus evm tracer, add missing networkParams when constructing chainDB 2021-09-26 10:45:52 +07:00
jangko a3badea928
config: fix new config based on input from jamie and zahary 2021-09-18 17:34:51 +07:00
jangko 69f2a0f95a
config: replace stdlib parseOpt with nim-confutils
fixes #581
2021-09-18 17:34:46 +07:00
jangko 48d497580a
config: remove last instance of getConfiguration usage from nimbus code
this is a preparation for migration to confutils based config
although there is still some getConfiguration usage in tests code
it will be removed after new config arrived
2021-09-08 21:25:14 +07:00
Kim De Mey e37bafd47e
Add a first, simple, content network test (#760)
* Add a first, simple, content network test

* Use sqlite as nimbus_db_backend for fluffy tests

* Fix backend selection for sqlite
2021-07-15 15:12:33 +02:00
Jordan Hrycaj 491149c6d5 Eip225 clique/PoA consensus protocol
details:
  formal port from go-lang sources, compiles but will not do anything
  useful yet
2021-06-17 08:03:57 +01:00
jangko a0d10f5728
drop PublicNetwork enum usage and replace it with NetworkId
we cannot limit the `--networkid` switch to values available in
`PublicNetwork` enum. it should able to accept very wide range of
custom NetworkId.
2021-05-20 14:04:16 +07:00
jangko f2491e6307
fixes crappy custom genesis and chain config parser
instead of using stdlib/json, now we switch to json_serialization
the result is much tidier code and more robust when parsing
optional fields.

fixes #635
2021-05-13 16:04:08 +07:00
jangko f6a0e4bcbd
fixes wrong usage of `chainId` in places where it should be networkId
fixes #643
2021-05-12 09:45:09 +07:00
jangko 82d4f6a2d4
fixes getReceipts proc signature in db_chain.nim
instead of using header as input param, now getReceipts using
receiptRoot hash, the intention is clearer and less data passed around
when we only using receiptRoot instead of whole block header.
2021-04-24 10:51:05 +07:00
jangko 1728dd8d54
fix getTransactionCount bug in db_chain.nim
getTransactionCount will always return 0 because the txCount
variable is not returned
2021-04-24 10:47:37 +07:00
jangko 3db535aa39
EIP2929 implementation 2021-01-11 14:56:42 +07:00
jangko f2b483d6ad
access list implementation 2021-01-11 14:54:55 +07:00
jangko f987e86562
implement more eth rpc 2020-07-30 14:21:11 +07:00
jangko c9802edfce
setup block and state env for more complex eth rpc tests 2020-07-28 23:48:45 +07:00
jangko f82dff64fa
implement more eth rpc and keystore management 2020-07-23 14:54:32 +07:00
jangko 336efdb0c3
implement web3, net, and some eth namespace rpc 2020-07-22 23:57:55 +07:00
jangko 165f9fea2e
reduce warnings 2020-07-21 13:15:06 +07:00
jangko 9d0b399213
fix accounts_cache: move overlayStorage to originalStorage when not clearCache 2020-06-20 18:45:09 +07:00
jangko bd7e1fe2e5
reuse VMState and AccountsCache for better performance 2020-06-18 13:16:38 +07:00
jangko eabacb0a33
allow nimbus to read geth database 2020-06-18 13:16:25 +07:00
jangko 50816f2ebb
implement block witness data collection in accounts_cache 2020-06-03 20:50:13 +07:00
jangko 844071033a
fix at various places related to missing accounts_cache.persist call 2020-06-01 13:45:32 +07:00
jangko 8a4c8c6273
fix buggy isAlive from accounts_cache 2020-06-01 10:54:25 +07:00
jangko 71514a0a66
replace state_db with accounts_cache 2020-05-30 10:14:59 +07:00
andri lim c9e49bf68a
fix accounts cache init API 2020-04-29 12:00:44 +07:00
Jacek Sieka ed7dbab70f db: use kvstore from nim-eth 2020-04-27 18:40:45 +03:00
Jacek Sieka 4ade5797ee
rlp: don't use ranges / experimental features (#495) 2020-04-20 20:12:44 +02:00
andri lim edc19076e3 accounts cache: read-only operations should not create new account 2020-02-27 14:38:42 +02:00
andri lim 2fbabd25a4 implement aleth/geth/parity compatibility mode -- 100% pass test 2020-02-20 09:08:44 +02:00
andri lim 16a938d3fa fixes clearStorage bug 2020-01-20 18:36:58 +02:00
andri lim 5f24757cd7 use 'withValue' when retrieving account's storage 2020-01-19 19:17:37 +02:00
andri lim cf0d4481f3 [skip ci] rename 'isExists' to 'exists' 2020-01-19 19:17:37 +02:00
andri lim 727b477fca [skip ci] add accounts cache test 2020-01-19 19:17:37 +02:00
andri lim 5795bf52e0 [skip ci] implement accounts_cache 2020-01-19 19:17:37 +02:00
andri lim 1ffb992674 multi root state trie implementation 2019-12-10 12:34:48 +02:00
andri lim 97e89b3afe implement 'getCommittedStorage' 2019-11-14 10:11:35 +00:00
andri lim c4dcf5da30
reduce compiler warnings 2019-11-13 21:49:39 +07:00
andri lim 61f340ae87 implement EIP2200's SSTORE and gas cost 2019-11-12 15:51:48 +00:00
andri lim 3ad465d169 implement block validation and friends 2019-09-26 12:33:43 +03:00
Jacek Sieka 2763bd0dd5
std_shims -> stew 2019-07-07 12:12:01 +02:00
andri lim 4a7f58c7dc implement constantinople bitwise opcode 2019-05-13 10:26:28 +03:00
andri lim 85d9bd3958
implement EIP158, GST -4 +81 2019-04-17 11:03:52 +07:00
andri lim 8a9e4aba80
apply dao fork changes 2019-04-07 06:53:26 +07:00
andri lim f2f92f5608
fix block 1.487.668 problem: getScore int overflow 2019-04-05 16:29:05 +07:00
Ștefan Talpalaru a67edd693a
assert() -> doAssert() 2019-03-13 22:36:54 +01:00
andri lim e2311d798b
fix recurring symbol collision problem 2019-03-07 22:53:09 +07:00
andri lim 200f93a3b7 refactor incNonce 2019-02-27 13:30:18 +02:00
andri lim 2c032ad1ab refactor utils 2019-02-27 13:30:18 +02:00
andri lim 282824cd53 zah's request: add missing comments 2019-02-18 13:16:04 +02:00
andri lim c53e7fa07c fixes #228 2019-02-15 17:20:51 +02:00
andri lim 7c6c7dbe3a 'accountExist' to 'accountExists' 2019-02-06 18:42:03 +01:00
andri lim 4cc0ef427c add ccountExist and isDeadAccount to AccountStateDB 2019-02-06 18:42:03 +01:00
Yuriy Glukhov 481c6cf4ed Use nim-eth (#224) 2019-02-05 20:15:50 +01:00
andri lim 0c887b3711 move initializeEmptyDb to genesis.nim 2019-01-15 15:30:25 +02:00
Yuriy Glukhov 58ad0e472c Moved db backends to eth_trie 2019-01-13 15:31:59 +02:00
Ștefan Talpalaru 89044b84e7 update Nimbus header and refine db backend picking
- dynamically generated copyright year interval
- added the db backend to the header
- documented the db-backend-changing define, made it case insensitive
  and ensured wrong values would trigger compilation errors
2019-01-07 01:21:29 +02:00
andri lim 281b7dffac implement debug_setHead 2019-01-06 13:17:08 +02:00
andri lim e78fb72ef6 fixes #188 2019-01-06 13:16:45 +02:00
andri lim 041ed689aa fix silly typo mistake 2019-01-06 11:43:38 +02:00
andri lim 96b18cfce3 add select_backend module 2019-01-06 11:43:38 +02:00
andri lim 0487720769 fix capturedb bug 2019-01-06 11:43:38 +02:00
andri lim ab0fb7fdf0 fix getScore 32bit bug 2019-01-06 11:43:38 +02:00
andri lim 5fc134ac2a add lmdb backend 2019-01-06 11:43:38 +02:00
andri lim a6bf970b1b some cleanup 2019-01-06 11:43:38 +02:00
Ștefan Talpalaru 048a43b2f1
better error handling
- added some useful `chronicles` defines in the top-level nim.cfg
- converted some assert() calls to error codes to avoid a reported
  segfault with -d:release
- the recoverable errors are being recovered from, in persistWorkItem()
- the only case that's a show stopper raises an exception
2018-12-17 23:36:38 +01:00
andri lim 9c0bb70ba7 working makeReceipt 2018-12-16 15:38:51 +02:00
andri lim ad1202f98a initial make_receipt 2018-12-16 15:38:51 +02:00
andri lim 7a3ddfec87 correcting account storage retrieval 2018-12-05 22:01:29 +07:00
andri lim c954dc7d62 fix tests 2018-12-05 22:01:29 +07:00
andri lim 5509c66cce implement transaction tracer 2018-12-05 22:01:28 +07:00
andri lim a098285a7b put prune switch into usage 2018-12-05 21:58:34 +07:00
andri lim 21070d510d add TransactionKey to persistTransactions and add more fat to debug_traceTransaction 2018-11-28 09:25:36 +07:00
andri lim 161d6adbc7 implement persistTransactions and getBlockBody 2018-11-24 08:00:58 +07:00
Dustin Brody d97cc50afc fix eth_trie/constants missing compile error 2018-10-19 07:44:08 -07:00
Zahary Karadjov 41adca1abb Minimal changes required for building the LES branch 2018-10-16 03:10:01 +03:00
Zahary Karadjov 343cc4fa43 Populate the persistent databases with the empty RLP key.
Also implements transactional block persistence. Two issues
in the transaction processing code have been discovered that
might affect other usages such as the CALL instruction.

The main fix gets us past block 49000.

You may need to clean up your database.
2018-10-05 03:36:48 +03:00
Zahary Karadjov 74f6f0b738 Deal with the refactoring in status-im/nim-rlp#20 2018-09-26 12:58:23 +03:00
Zahary Karadjov 5c0fd16448 Fix an issue with the Sqlite3 tests (the in-memory DB was not properly used) 2018-09-25 16:01:38 +03:00
Zahary Karadjov 4e323df363 Implement a --datadir command-line options
By default, the database files will be written in the
platform-specific application data folder:

$HOME/AppData/Roaming/Nimbus/DB
$HOME/Library/Application Support/Nimbus/DB
$HOME/.cache/nimbus/db
2018-09-25 02:06:20 +03:00
tersec 690ce2cf97
Partly fix state_db getCode for 20 new working GeneralStateTests (#156)
* partly fix state_db getCode to get 20 new working GeneralStateTests; remove 2 functions which existed as workarounds; switch all remaining setBalance calls in GeneralStateTestRunner to subBalance (addBalance calls already changed)

* two of the 20 new tests don't work in 32-bit builds
2018-09-20 21:36:57 +00:00
Yuriy Glukhov 0adfaf0f31 s/increaseBalance/addBalance 2018-09-19 18:41:17 +03:00
Yuriy Glukhov 3d64c66b7a More state restoring 2018-09-19 18:41:17 +03:00
coffeepots 4f03c9cf2a
Merge pull request #139 from status-im/callOpCodeWork
Flesh out call op
2018-09-18 14:09:00 +01:00
Dustin Brody beb1befc6e 13 additional GeneralStateTests working; remove commented auto-ported Py-EVM validations which are implicit in UInt256; address comments to last commit (toSeq, hoising vmState.readOnlyStateDB, deltaBalance -> increaseBalance) 2018-09-17 13:41:32 +00:00
Zahary Karadjov 409d875beb Some helpers for tracing and debugging Trie issues (the code is no-op by default) 2018-09-16 00:20:20 +03:00
coffeepots 213aebde7d Update some comments, adding TODO item for getStateDb 2018-09-14 16:16:55 +01:00
Dustin Brody 7ba90fda54
48 new working GeneralStateTests, via fixing some overflows/underflows, creating a rudimentary balance commit/revert scheme, and implementing account deletion for the selfdestruct instruction (#146) 2018-09-14 13:43:40 +00:00
Yuriy Glukhov 62260c786d Caching DB backend 2018-09-12 17:53:15 +03:00
Yuriy Glukhov 3a1ec035b3 Various fixes towards state reconstruction 2018-09-07 14:00:37 +03:00
coffeepots 12834fba12
Merge branch 'master' into createOpContractAddr 2018-09-04 16:37:43 +01:00
Dustin Brody 4262ea8670 fix (1) overflow/signed confusion when reading large-valued storage entries; (2) adjust test fixture validity for GeneralStateTests; (3) remove incorrect usage of state trie for code (not codehash) storage; add ability to display state; (4) avoid double-hashing keys for already-hashed SecureHexaryTrie for storage; (5) add intentionally local workaround for empty code in GeneralStateTest fixtures lacking 0x prefix; (6) fix nonce parsing, which is always hex string in test fixtures 2018-09-04 17:55:30 +03:00
coffeepots 7ed2403c5c Add hasCodeOrNonce 2018-09-03 19:24:55 +01:00
coffeepots 1427d59f27 Change setNonce, getNonce to use AccountNonce, which is uint64 from UInt256 2018-09-01 16:03:06 +03:00
Jacek Sieka db202dc35f
replace logging module with nim-chronicles (fixes #38)
This blindly changes logging to nim-chronicles - issues that ensue:

* keeps gas cost computation logs hidden behind flag
* unclear if logScope is practical - for example, since vm is split over
many files, topics get lost when using simple top-level per-module
topics
* when passing named object around, scope should incliude the name of
the object but this is caught neither by logScope nor by dynamicLogScope
2018-08-29 16:13:03 -06:00
Yuriy Glukhov e3be8ca30a Separate Chain from ChainDB 2018-08-29 11:49:01 +03:00
coffeepots 67e9f47ac5 Add getTransactionKey 2018-08-21 13:33:37 +01:00
Yuriy Glukhov 55b5dbd719 Raise KeyError when key not found 2018-08-14 18:40:59 +03:00
Zahary Karadjov 350188bd29 address review comments 2018-08-09 12:25:37 +03:00
Zahary Karadjov d902be66ee Fix the genesis state hash computation
The account database needed to use the secure flavour of the Hexary
Trie. This means all keys are SHA3 hashed before being inserted.
2018-08-08 15:48:34 +03:00
Yuriy Glukhov e4929d4f60 Slightly reworked ChainDB interface to more consistent 2018-08-03 14:56:49 +03:00
Yuriy Glukhov c4f4a37d2c Added genesis initialization 2018-08-01 15:50:44 +03:00
Yuriy Glukhov 84515b78a9 Align DB backend interfaces with eth_trie. Init BaseChainDB in nimbus. 2018-07-30 16:05:32 +03:00
coffeepots e33bc85bfa Remove validateCanonicalAddress 2018-07-19 19:04:01 +01:00
Mamy Ratsimbazafy 4b5eada322
Refactor interpreter dispatch (#65)
* move forks constants, rename errors

* Move vm/utils to vm/interpreter/utils

* initial opcodes refactoring

* Add refactored Comparison & Bitwise Logic Operations

* Add sha3 and address, simplify macro, support pop 0

* balance, origin, caller, callValue

* fix gas copy opcodes gas costs, add callDataLoad/Size/Copy, CodeSize/Copy and gas price opcode

* Update with 30s, 40s, 50s opcodes + impl of balance + stack improvement

* add push, dup, swap, log, create and call operations

* finish opcode implementation

* Add the new dispatching logic

* Pass the opcode test

* Make test_vm_json compile

* halt execution without exceptions for Return, Revert, selfdestruct (fix #62)

* Properly catch and recover from EVM exceptions (stack underflow ...)

* Fix byte op

* Fix jump regressions

* Update for latest devel, don't import old dispatch code as quasiBoolean macro is broken by latest devel

* Fix sha3 regression on empty memory slice and until end of range slice

* Fix padding / range error on expXY_success (gas computation left)

* update logging procs

* Add tracing - expXY_success is not a regression, sload stub was accidentally passing the test

* Reuse the same stub as OO implementation

* Delete previous opcode implementation

* Delete object oriented fork code

* Delete exceptions that were used as control flows

* delete base.nim 🔥, yet another OO remnants

* Delete opcode table

* Enable omputed gotos and compile-time gas fees

* Revert const gasCosts -> generates SIGSEGV

* inline push, swap and dup opcodes

* loggers are now template again, why does this pass new tests?

* Trigger CI rebuild after rocksdb fix https://github.com/status-im/nim-rocksdb/pull/5

* Address review comment on "push" + VMTests in debug mode (not release)

* Address review comment: don't tag fork by default, make opcode impl grepable

* Static compilation fixes after rebasing

* fix the initialization of the VM database

* add a missing import

* Deactivate balance and sload test following #59

* Reactivate stack check (deactivated in #59, necessary to pass tests)

* Merge remaining opcodes implementation from #59

* Merge callDataLoad and codeCopy fixes, todo simplify see #67
2018-07-06 09:52:31 +02:00
zah 18b7bbb3b0 Implemented most of the stubbed-out state handling instructions (#59)
Merge note: currently cannot compile due to `quasiBoolean` (#63). This will be solved by https://github.com/status-im/nimbus/pull/65
----

* Implemented most of the stubbed out state handling instructions

The code compiles, but still fails at the moment due to incorrect

initialization of the VM. Don't merge yet. More commits will be
pushed in the coming days.

* Fixed crash

* trie put and del are void now

* getBlockTransactionData and getReceipts

* Working code for extcodesize0.json

* fix origin.json

* fix calldatasize1

* fix calldataloadSizeTooHighPartial

* fix calldataloadSizeTooHigh

* more efficient PushX implementation

* fix and, or, xor
2018-07-05 14:41:01 +02:00
Zahary Karadjov 3060452d96 review changes
* `usedBytes` was not named appropriately. it indicates the last
  populated position in the data (the last valid index) for slicing
  purposes (i.e. the position is passed to `toOpenArray`)

* Fixed the `==` operator for DBKeys

* Use one-byte key for the 'canonical head hash'
2018-06-26 13:46:54 +03:00
Zahary Karadjov 583c72fa54 Implement storage backends using RocksDB and SQLite 2018-06-26 13:46:54 +03:00
Yuriy Glukhov 1c79d1ab3d Persist* functions 2018-06-24 18:37:02 +03:00
Mamy Ratsimbazafy c26c751f9b
Reorganize VM/interpreter + cleanup (#52)
* Move and cleanup interpreter files - prepare for redesign of VM

* fix call comment aobut recursive dependencies

* memory: use a template again and avoid (?) a cstring-> string conversion

* Fix stack test regression

* Fix recursive dependency on logging_ops, test_vm_json compiles but regression :/

* Fix signextend regression

* Fix 3 signed test and sha3 test
2018-06-15 11:11:25 +02:00
Mamy Ratsimbazafy 90c3ca4a96
Gas refactoring - decouple opcode logic and gas (#49)
* Decoupling op logic and gas - introduce gasometer, rework opcode declaration

* Remove gas constants for gas opcode computation

* Remove gas constants for precompiled contracts

* make vm_types compile

* Make opcode, call and computation compile

* Distinguish between dynamic and complex gas costs, fix arithmetic

* Fix context and sha3

* update memory and storage ops

* Log opcode uses memory expansion code

* update/stub system_ops with gas costs

* Make test compile. Deactivate stub test_vm

* all tests compiles, opcode fails due to https://github.com/nim-lang/Nim/issues/8007 (const object variant in tables reset at runtime)

* Create an enum without holes - workaround: https://github.com/nim-lang/Nim/issues/8007

* Use arrays instead of tables for GasCosts, remove some unused imports - passing all basic tests!

* Make test_vm_json compile

* Fix test_vm_json - workaround https://github.com/nim-lang/Nim/issues/8015

* fix memory expansion cost bug

* Remove leftover special handling from before GckMemExpansion

* cleanup outdated comment, better align =

* Fix sha3 gas cost not taking memory expansion into account

* Improve gas error reporting of test_vm_json

* Fix gas computation regression due to mem expansion

* mass replace for memExpansion->RequestedMemSize was too eager

* fix log gas cost (no tests :/)

* missed a static FeeSchedule

* static as expression is fickle
2018-06-12 17:33:47 +02:00
Yuriy Glukhov 8bdf09683b Hash256/EthAddrres refactoring 2018-05-30 19:11:15 +03:00
Yuriy Glukhov 51b572d8b8 Some renames 2018-05-28 13:22:28 +03:00