Commit Graph

3171 Commits

Author SHA1 Message Date
jangko 6544adf360
implement EIP-1153: Transient storage
new EVM opcodes:
- TLOAD  0xb3
- TSTORE 0xb4
2023-06-23 14:04:36 +07:00
Jordan Hrycaj 15cc9f962e
Aristo db update vertex caching when merging (#1606)
* Added missing deferred cleanup directive to sub-test functions

why:
  Rocksdb keeps the files locked for a short while leading to errors. This
  was previously solved my using different db sub-directories

* Provide vertex deep-copy function globally.

why:
  is just handy

* Avoid unnecessary vertex caching when merging proof nodes

also:
  Run all merge tests on the rocksdb backend
  Previously, proof node tests were run without backend
2023-06-22 20:21:33 +01:00
Jordan Hrycaj 83dbe87159
Aristo db update foreground caching (#1605)
* Fix vertex ID generator state handling for rocksdb backend

why:
 * Key error in walk iterator
 * Needs to be loaded when opening the database

* Use non-zero sub-table prefixes for rocksdb

why:
  Handy for debugging

* Fix error code for missing key on rocksdb backend

why:
  Previously returned `VOID_HASH_KEY` rather than `GetKeyNotFound`

* Explicitly copy vertex data between internal table and function/result argument

why:
  Function argument or return reference may still refer to the same data
  object.

* Updated error symbols

why:
  Error symbol names for the hike module now start with the prefix `Hike`.

* Write back modified branch node into local top layer cache

why:
  With the backend available, the source of the branch node references
  might not be the top layer cache. So any change must be explicitely
  recorded.
2023-06-22 12:13:24 +01:00
Kim De Mey 2de9c95770
Remove a socket.close after destroy + bump nim-eth for uTP fixes (#1604) 2023-06-21 18:56:57 +02:00
Jordan Hrycaj 4b66f93274
Aristo db with storage backends (#1603)
* Generalised Aristo DB constructor for any type of backend

details:
  * Records to be deleted are represented as key-void (rather than
    key-value) pairs by the put-function arguments
  * Allow direct driver access, iterators as example implementation and
    for testing.

* Provide backend storage interface

details:
  Stores the top layer onto backend tables

* Implemented Rocks DB backend

details:
  Transaction based `put()` functionality
  Iterators (based on direct RocksDB access)
2023-06-20 14:26:25 +01:00
Kim De Mey dd4da7491d
Remove yield usage from portal stream and protocol (#1602) 2023-06-15 14:32:33 +02:00
Jordan Hrycaj d7f40516a7
Detach from snap/sync declarations & definitions (#1601)
why:
  Tests and some basic components were originally borrowed from the
  snap/sync implementation. These have fully been re-implemented.
2023-06-12 19:16:03 +01:00
Jordan Hrycaj 0308dfac4f
Aristo db address sup trie items properly (#1600)
* Fix include

why:
  Eth67 not default yet so that got missed

* Rename `LeafKey` => `LeafTie`

why:
  Name is a pen picture of what this object is for. Also, it avoids the
  ubiquitous term `key`.

* Provided `getOrVoid()` wrapper for `getOrDefault()`

also:
  Provide `isValid()` syntactic sugar for `.isNil.not`, `!= 0` etc.
  Reorg descriptor source, split into sub-sources

* Bundled `NodeKey` objects with root ID and called it `HashLabel`

why:
  `NodeKey` (aka repurposed Hash265) objects are unique only within a
  particular sub-trie (e.g. storage slots) which are kept separated
  (i.e non-interleaved) by design. This is not applied to the backend
  as the map VertexID->NodeKey labelling the nodes needs not be injective.

  For the in-memory database (transaction) layers, the injective map
  VertexID->(VertexID,NodeKey) is used where the first field of the image
  tuple is the root ID of the sub-trie the `NodeKey` object is valid. So
  identical storage tries for different accounts can be represented.
2023-06-12 14:48:47 +01:00
jangko 10aabd8ca1
bubble up validate transaction error message 2023-06-12 12:58:53 +07:00
jangko 8700d8b1e1
reduce compiler warnings 2023-06-12 12:58:53 +07:00
jangko 0128c82c52
fix initial hardfork transition in CommonRef 2023-06-12 11:06:04 +07:00
Jordan Hrycaj 932a2140f2
Aristo db supporting forest and layered tx architecture (#1598)
* Exclude some storage tests

why:
  These test running on external dumps slipped through. The particular
  dumps were reported earlier as somehow dodgy.

  This was changed in `#1457` but having a second look, the change on
  hexary_interpolate.nim(350) might be incorrect.

* Redesign `Aristo DB` descriptor for transaction based layers

why:
  Previous descriptor layout made it cumbersome to push/pop
  database delta layers.

  The new architecture keeps each layer with the full delta set
  relative to the database backend.

* Keep root ID as part of the `Patricia Trie` leaf path

why;
  That way, forests are supported
2023-06-09 12:17:37 +01:00
jangko 67aaf92c1d
bump submodules 2023-06-07 18:12:02 +07:00
Jordan Hrycaj 11bb33d0bc
Added delete fuctionality (#1596) 2023-06-02 20:21:46 +01:00
Jordan Hrycaj 099444ab3f
Aristo db fixes after storage slots dump tests added (#1595)
* Fix missing Merkle key removal in `merge()`

* Accept optional root hash argument in `hashify()`

why:
  For importing a full database, there will be no proof data except the
  root key. So this can be used to check and set the root key in the
  database descriptor.

also:
  Associate vertex ID to `hashify()` error return code

* Added Aristo Trie traversal function

why:
 * step along leaf vertices in sorted order
 * tree/trie consistency checks when debugging

* Enabled storage slots test data for Aristo DB
2023-06-02 11:04:29 +01:00
jangko cbd593f514
bump nim-eth and accompanying fixes 2023-05-31 19:10:17 +07:00
jangko 742759e607
optimize EVM memory extend 2023-05-31 10:14:26 +07:00
Jordan Hrycaj 2fc349feb9
Aristo db merkle hashify functionality added (#1593)
* Keep vertex ID generator state with each db-layer

why:
  The vertex ID generator state is part of the difference to the below
  layer

* Move otherwise unused source to test directory

* Add Merkle hash generator

also:
  * Verification facility for debugging
  * Empty Merkle key hashes encoded as `EMPTY_ROOT_HASH`
2023-05-30 22:21:15 +01:00
Jordan Hrycaj cd78458123
Add items to `Aristo Trie` database (#1586)
details:
1. Merging a leaf vertex merges a `Patricia Trie` path (while
   adding/modiying vertices) and adds a leaf node with payload
2. Merging a Merkel node merges a single vertex to the `Patricia Trie`
   and registers merkel hashes
3. Action 2 can be used before action 1 in order to construct a
   Merkel proof as required for handling `snap/1` data.
4. Unit tests show that action 3 is benign for now :)
2023-05-30 12:47:47 +01:00
Kim De Mey bf21484f70
Add EL block proofs verifable from historical_summaries (#1588) 2023-05-30 07:02:39 +02:00
Kim De Mey c9f3f82877
Add new Portal BlockBody type for Shanghai fork (#1589) 2023-05-30 06:56:54 +02:00
jangko 48930970ce
free more space of windows ci 2023-05-27 20:52:48 +07:00
jangko fb3b341aea store withdrawals along with txs in block body 2023-05-23 11:36:20 +03:00
jangko 9868eceed1 fix withdrawals problem in tx-pool 2023-05-23 11:36:20 +03:00
jangko 27393e9420 fix difficulty test 2023-05-23 11:36:20 +03:00
jangko 93ff745421 Add shanghai timestamp to MainNet, Goerli, and Sepolia 2023-05-23 11:36:20 +03:00
Jordan Hrycaj 4c865ec884
Snap sync update pivot updating via rpc (#1583)
* Unit tests update, code cosmetics

* Fix segfault with zombie handling

why:
  In order to save memory, the data records of zombie entries are removed
  and only the key (aka peer node) is kept. Consequently, logging these
  zombies can only be done by the key.

* Allow to accept V2 payload without `shanghaiTime` set while syncing

why:
  Currently, `shanghaiTime` is missing (alt least) while snap syncing. So
  beacon node headers can be processed regardless. Normal (aka strict)
  processing will be automatically restored when leaving snap sync mode.
2023-05-16 14:52:44 +01:00
jangko 4e332868b7
reduce compiler warnings 2023-05-16 11:15:10 +07:00
Jordan Hrycaj ff0fc98fdf
Multi layer architecture 4 aristo db (#1581)
* Cosmetics, renamed fields (eVtx, bVtx) -> (eVid, bVid)

* Multilayered delta architecture for Aristo DB

details:
  Any VertexID or data retrieval needs to go down the rabbit hole and
  fetch/get/manipulate the bottom layer -- even without explicit
  backend.

* Direct reference to backend from top-level layer

why:
  Some services as the vid management needs to be synchronised among all
  layers. So access is optimised.
2023-05-14 18:43:01 +01:00
Kim De Mey 33fd8b1fae
Add historical_roots with proof + test (#1579) 2023-05-11 19:08:34 +02:00
Jordan Hrycaj 605739ef4c
Experimental MP-trie (#1573)
* Experimental MP-trie

why:
  Deleting records is a infeasible with the current structure

* Added vertex ID recycling management

Todo:
  Provide some unit tests

* DB layout update

why:
  Main news is the separation of `Merkel` hashes into an extra table.

details:
  The code fragments cover conversion between compact MPT records and
  Aristo DB records as well as some rudimentary cache handling for
  the `Merkel` hashes (i.e. the extra table entries.)

todo:
  Add some simple unit test for the descriptor record (currently used
  for vertex ID management, only.)

* Updated vertex ID recycling management

details:
  added simple unit tests (mainly testing ABI)

* docu update
2023-05-11 15:25:29 +01:00
jangko 2871dbfddf
remove unused vmState param from t8n tool 2023-05-11 14:36:32 +07:00
jangko 4e58f9d79a
modexp precompile allow arbitrary input length 2023-05-11 07:21:03 +07:00
Kim De Mey 408394a2bd
Bump nim-eth and remove unneeded Defect raises (#1575) 2023-05-10 18:04:35 +02:00
Kim De Mey f565b2bc57
Bump nim-eth and accompanying nim-secp2561k module (#1574) 2023-05-09 21:40:59 +02:00
Kim De Mey 9097548a10
Adjust discv5_updateNodeInfo JSON-RPC to specify key and value (#1569) 2023-05-09 14:34:01 +02:00
jangko cdfc99e4f7
remove appveyor, travis, and azure scripts 2023-05-02 20:48:23 +07:00
Kim De Mey 2ba974a7cd
Adjust portal FindContent JSON-RPC according to latest specs (#1566)
Also remove the FindContentFull unspecced JSON-RPC as the new
FindContent specification replaces this functionality basically.
2023-04-28 09:04:21 +02:00
Kim De Mey acc2d53b52
Further clean-up of Portal and discv5 JSON-RPC (#1565)
- Removal of TODOs fixed in specification instead
- Fix some other TODOs
- General clean-up such as to0xHex usage
2023-04-27 14:07:57 +02:00
Jordan Hrycaj e1369a7c25
Improve full sync part behaviour 4 snap sync suite (#1564)
* Set maximum time for nodes to be banned.

why:
  Useless nodes are marked zombies and banned. They a kept in a table
  until flushed out by new connections. This works well if there are many
  connections. For the case that there are a few only, a maximum time is
  set. When expired, zombies are flushed automatically.

* Suspend full sync while block number at beacon block

details:
  Also allows to use external setting from file (2nd line)

* Resume state at full sync after restart (if any)
2023-04-26 16:46:42 +01:00
Jordan Hrycaj 68b2448ce1
Snap sync cosmetic code update (#1563)
* Relocate full sync descriptors from global `worker_desc.nim` to local pass

why:
  These settings are needed only for the full sync pass.

* Rename `pivotAccountsCoverage*()` => `accountsCoverage*()`

details:
  Extract from `worker_desc.nim` into separate source file.

* Relocate snap sync sub-descriptors

details:
  ..from global `worker_desc.nim` to local pass module `snap_pass_desc.nam`.

* Rename `SnapPivotRef` => `SnapPassPivotRef`

* Mostly removed `SnapPass` prefix from object type names

why:
  These objects are solely used on the snap pass.
2023-04-25 17:34:48 +01:00
Jordan Hrycaj d6ee672ba5
Fix pivot setup after switch to full sync (#1562)
* Cosmetics, update logging, docu

* Fix pivot hand-over after switch to full sync

why:
  Got garbled after code clean up
2023-04-25 13:24:32 +01:00
Adam Spitz 7f56e90654
Revert "Some work on speculative execution (#1552)" (#1561)
This reverts commit ddbdf34c3d.
2023-04-24 16:59:38 -04:00
Jordan Hrycaj c5e895aaab
Code reorg 4 snap sync suite (#1560)
* Rename `playXXX` => `passXXX`

why:
  Better purpose match

* Code massage, log message updates

* Moved `ticker.nim` to `misc` folder to be used the same by full and snap sync

why:
  Simplifies maintenance

* Move `worker/pivot*` => `worker/pass/pass_snap/*`

why:
  better for maintenance

* Moved helper source file => `pass/pass_snap/helper`

* Renamed ComError => GetError, `worker/com/` => `worker/get/`

* Keep ticker enable flag in worker descriptor

why:
  This allows to pass this flag with the descriptor and not an extra
  function argument when calling the setup function.

* Extracted setup/release code from `worker.nim` => `pass/pass_init.nim`
2023-04-24 21:24:07 +01:00
Adam Spitz ddbdf34c3d
Some work on speculative execution (#1552)
* Recreating some of the speculative-execution code.

Not really using it yet. Also there's some new inefficiency in
memory.nim, but it's fixable - just haven't gotten around to it yet.

The big thing introduced here is the idea of "cells" for stack,
memory, and storage values. A cell is basically just a Future (though
there's also the option of making it an Identity - just a simple
distinct wrapper around a value - if you want to turn off the
asynchrony).

* Bumped nim-eth.

* Cleaned up a few comments.

* Bumped nim-secp256k1.

* Oops.

* Fixing a few compiler errors that show up with EVMC enabled.
2023-04-22 10:17:37 -04:00
Jordan Hrycaj f40a066cc6
Update snap sync ready to succeed at lab test (#1556)
* Extract RocksDB timing tests from snap unit tests as separate module

why:
  Declutter, make space for more snap related unit tests.

* Renamed `undumpNextGroup()` => `undumpBlocks()`

why:
  Source file name is called `undump_blocks.nim` which should be sort
  of in sync with the method name(s).

* Implement snap/1 server method `getByteCodes()`

* Implement snap/1 client method `getByteCodes()`

* Implement faculty for handling contract code fetching via snap/1

* Provide persistent storage for contract code records

* Implement contract code snap sync fetch & store

* Code massage, cosmetics

* Unit tests for verifying snap sync snapshot dump

details:
  Use `undump_kvp.dumpAllDb()` to dump any database.
2023-04-21 22:11:04 +01:00
Jordan Hrycaj 0387afb7b1
Remove local block body rlp fix (#1555)
* Remove local block body rlp fix

why:
  Fix moved to `nim-eth` module

* Update nim-eth bumper
2023-04-21 20:08:18 +01:00
Kim De Mey 21ef655b9c
Fix dataRadius in result of Portal ping JSON-RPC (#1553) 2023-04-21 17:00:55 +02:00
Kim De Mey 04137f1c98
Fix name of fluffy config file and disable log colors for Windows (#1551) 2023-04-20 12:58:56 +02:00
Kim De Mey ff90f4fd22
Improve logging and logging options in Fluffy (#1548)
* Improve logging and logging options in Fluffy

- Allow selection of log format, including:
  - JSON
  - automatic selection based on tty
- Allow log levels per topic configured on cli
2023-04-19 17:01:01 +02:00