Commit Graph

3312 Commits

Author SHA1 Message Date
jangko 19ace7309a
EIP-1153: Fix tload stack underflow bug 2023-09-21 16:25:48 +07:00
jangko a72141ccdf
bump eth_tests 2023-09-21 16:06:51 +07:00
Daniel Sobol bd1a647987
Dockerfile for portal-hive development (#1759)
improves build time for a local dev container for a portal-hive test suite
2023-09-20 19:07:09 +03:00
Kim De Mey 53df21f000
Add clientInfo to ENR for easier testnet debugging (#1513)
* Add clientInfo to ENR for easier testnet debugging

* Remove git rev from ENR client info

This allows for a shorter ENR which allows more ENRs to be packed
in the Nodes reply. And for glados, only the client name is used
anyhow.
2023-09-20 12:19:40 +02:00
andri lim 7af9e3dc53
Refactor beacon skeleton (#1761)
* Fix unlisted exception due to recent modification in CoreDB interface

* Refactor beacon skeleton

* More unlisted exception fix
2023-09-19 11:52:28 +07:00
andri lim de634963ad
Selectively download nimbus-eth2 submodules (#1757) 2023-09-19 07:47:58 +07:00
Jordan Hrycaj 6bc55d4e6f
Core db aristo and kvt updates preparing for integration (#1760)
* Kvt: Implemented multi-descriptor access on the same backend

why:
  This behaviour mirrors the one of Aristo and can be used for
  simultaneous transactions on Aristo + Kvt

* Kvt: Update database iterators

why:
  Forgot to run on the top layer first

* Kvt: Misc fixes

* Aristo, use `openArray[byte]` rather than `Blob` in prototype

* Aristo, by default hashify right after cloning descriptor

why:
  Typically, a completed descriptor is expected after cloning. Hashing
  can be suppressed by argument flag.

* Aristo provides `replicate()` iterator, similar to legacy `replicate()`

* Aristo API fixes and updates

* CoreDB: Rename `legacy_persistent` => `legacy_rocksdb`

why:
  More systematic, will be in line with Aristo DB which might have
  more than one persistent backends

* CoreDB: Prettify API sources

why:
  Better to read and maintain

details:
  Annotating with custom pragmas which cleans up the prototypes

* CoreDB: Update MPT/put() prototype allowing `CatchableError`

why:
  Will be needed for Aristo API (legacy is OK with `RlpError`)
2023-09-18 21:20:28 +01:00
Kim De Mey 18b3ff3390
Correct the shortcut for ommershash verification post Shanghai (#1756)
* Correct the shortcut for ommershash verification post Shanghai

There was a faulty length check being done in the assumption that
the uncles field was already the list. However it is still the
RLP encoded data at that point, hence we check directly for the
RLP encoded value for empty list.

* Fix copy-paste error in withdrawal root verification

And print out the hashes in txs root and withdrawal root
verification
2023-09-18 14:50:07 +02:00
Kim De Mey a65b1c9062
Do not accept new offers if our contentQueue is full (#1753) 2023-09-16 16:06:11 +02:00
Jordan Hrycaj cd1d370543
Aristo db api extensions for use as core db backend (#1754)
* Update docu

* Update Aristo/Kvt constructor prototype

why:
  Previous version used an `enum` value to indicate what backend is to
  be used. This was replaced by using the backend object type.

* Rewrite `hikeUp()` return code into `Result[Hike,(Hike,AristoError)]`

why:
  Better code maintenance. Previously, the `Hike` object was returned. It
  had an internal error field so partial success was also available on
  a failure. This error field has been removed.

* Use `openArray[byte]` rather than `Blob` in functions prototypes

* Provide synchronised multi instance transactions

why:
  The `CoreDB` object was geared towards the legacy DB which used a single
  transaction for the key-value backend DB. Different state roots are
  provided by the backend database, so all instances work directly on the
  same backend.

  Aristo db instances have different in-memory mappings (aka different
  state roots) and the transactions are on top of there mappings. So each
  instance might run different transactions.

  Multi instance transactions are a compromise to converge towards the
  legacy behaviour. The synchronised transactions span over all instances
  available at the time when base transaction was opened. Instances
  created later are unaffected.

* Provide key-value pair database iterator

why:
  Needed in `CoreDB` for `replicate()` emulation

also:
  Some update of internal code

* Extend API (i.e. prototype variants)

why:
  Needed for `CoreDB` geared towards the legacy backend which has a more
  basic API than Aristo.
2023-09-15 16:23:53 +01:00
Kim De Mey cb4626b488
Initial implementation of Portal fluffy beacon chain bridge (#1751)
* Initial implementation of Portal fluffy beacon chain bridge

* Run windows ci for fluffy with -j1 to avoid OOM
2023-09-15 16:21:00 +02:00
jangko f4cf952a4c
Fix Nim binaries cache key of simulators CI 2023-09-14 12:25:23 +07:00
jangko dbb17ab68c
Fix Nim binaries cache key of main CI
Why:
Multiple CI share the same cache key will fail to save the cached files.

Also:
Fix simulators CI path to cached llvm-mingw on windows.
2023-09-14 10:23:15 +07:00
andri lim 73622459b1
Remove unused files because of rebase conflict (#1748) 2023-09-13 11:42:39 +07:00
andri lim 56215ed83f
Bump stint to v2.0: new array backend (#1747)
* Bump stint to v2.0: new array backend
2023-09-13 09:32:38 +07:00
Jordan Hrycaj 8e00143313
Aristo db code massage n cosmetics (#1745)
* Rewrite remaining `AristoError` return code into `Result[void,AristoError]`

why:
  Better code maintenance

* Update import sections

* Update Aristo DB paths

why:
 More systematic so directory can be shared with other DB types

* More cosmetcs

* Update unit tests runners

why:
  Proper handling of persistent and mem-only DB. The latter can be
  consistently triggered by an empty DB path.
2023-09-12 19:45:12 +01:00
Jordan Hrycaj dda049cd43
Simple stupid key-value table companion for Aristo DB (#1746)
why:
  Additional tables needed for the `CoreDB` object with separate
  key-value table and MPT.

details:
+ Stripped down copy of Aristo DB to have a similar look'n feel. Otherwise
  it is just a posh way for accessing `Table` objects or `RocksDB` data.
+ No unit tests yet, will be tested on the go.
2023-09-12 19:44:45 +01:00
Jordan Hrycaj 8e46953390
Aristo db state root repos and reorg (#1744)
* Reorg of distributed backend access

details:
  Now handled via API provided in `aristo_desc`.

* Rename `checkCache()` => `checkTop()`

why:
  Better naming for top layer cache checker

also:
  Provide cascaded fifos checker

* Provide `eq` directive for finding filter by exact filter ID (think block number)

* Some code beautification (for better code reading)

* State root reposition and reorg

details:
  Repositioning is supported by forking a new descriptor. Reorg is then
  accomplished by writing this forked state on the backend database.
2023-09-11 21:38:49 +01:00
andri lim b7f0be5468
Fix Windows simulators CI (#1743)
* update simulator cache key

* enable libbacktrace in hive simulators
2023-09-10 21:13:09 +07:00
andri lim 948c94763c
Bump nim-eth: Add closeWait to EthereumNode (#1742) 2023-09-09 13:54:58 +07:00
andri lim db6bd9af16
Switch to llvm-mingw for faster Windows CI (#1740) 2023-09-08 22:10:16 +07:00
andri lim 348a9aea6f
Close eth node after usage (#1741) 2023-09-08 21:37:50 +07:00
andri lim cf9553196e
When memory backend selected, no snap sync (#1738) 2023-09-08 15:21:59 +07:00
Kim De Mey b0442dc41a
Add instance/container select for version panel in grafana dashboard (#1737) 2023-09-08 09:05:00 +02:00
Kim De Mey c04f2477ac
Add version to fluffy cli flags and metrics (#1736)
Also update Grafana dashboard to incorperate the metrics and do
a confutils module bump.
2023-09-07 16:27:13 +02:00
Kim De Mey 2b23fe2ba4
Bump chronos submodule (#1735) 2023-09-07 16:15:58 +02:00
jangko 5fb0fc65ba
Implement beacon sync stub
- Prepare a test env for beacon sync in engine api simulator.
- Wiring beacon sync to the rest of subsystems.
2023-09-07 08:49:31 +07:00
jangko 6139152143
Add multiple clients support to engine api simulator 2023-09-06 18:51:02 +07:00
Jordan Hrycaj 070b06f809
Implement backend filter mechanics (#1730)
details:
* Tested features
  + Successively store filters with increasing filter ID (think block number)
  + Cascading through fifos, deeper fifos merge groups of filters
  + Fetch squash merged N top fifos
  + Delete N top fifos, push back merged fifo, continue storing
  + Fifo chain is verified by hashes and filter ID
* Not tested yet
  + Real live scenario (using data dumps)
  + Real filter data (only shallow filters used so far)
2023-09-05 19:00:40 +01:00
Jordan Hrycaj 3936d4d0ad
Aristo db fixes n updates needed for filter fifo (#1728)
* Set scheduler state as part of the backend descriptor

details:
  Moved type definitions `QidLayoutRef` and `QidSchedRef` to
 `desc_structural.nim` so that it shares the same folder as
  `desc_backend.nim`

* Automatic filter queue table initialisation in backend

details:
  Scheduler can be tweaked or completely disabled

* Updated backend unit tests

details:
+ some code clean up/beautification, reads better now
+ disabled persistent filters so that there is no automated filter
   management which will be implemented next

* Prettify/update unit tests source code

details:
  Mostly replacing the `check()` paradigm by `xCheck()`

* Somewhat simplified backend type management

why:
  Backend objects are labelled with a `BackendType` symbol where the
  `BackendVoid` label is implicitly assumed for a `nil` backend object
  reference.

  To make it easier, a `kind()` function is used now applicable to
  `nil` references as well.

* Fix DB storage layout for filter objects

why:
  Need to store the filter ID with the object

* Implement reverse [] index on fifo

why:
  An integer index argument on `[]` retrieves the QueueID (label) of the
  fifo item while a QueueID argument on `[]` retrieves the index (so
  it is inverse to the former variant).

* Provide iterator over filters as fifo

why:
  This iterator goes along the cascased fifo structure (i.e. in
  historical order)
2023-09-05 14:57:20 +01:00
Daniel Sobol 69a1a988b0
don't gossip data back to the node we rcvd it from [#1227] (#1719) 2023-09-04 12:21:01 +02:00
jangko 156f4f0140
Fix EIP-4844 and EIP-4788 genesis initialization 2023-08-31 10:58:31 +07:00
jangko 47fae1bb71
Add EVM tracer test case for EIP-2929 opcodes 2023-08-31 09:49:51 +07:00
Kim De Mey 7ae333ac8b
Add discv5 provided ENR directly to Portal protocol routing table (#1714)
* Add discv5 provided ENR directly to Portal protocol routing table

Previous version of getting the ENR from the discv5 routing table
would not work due to the order of first calling the talk protocol
handler and only after that the addEnr to the disc5 routing table.

Instead of changing this order, pass along the ENR directly to
avoid this additional getNode call.

* Still request ENR from discv5 if it wasn't passed via handshake
2023-08-30 20:38:15 +02:00
Jordan Hrycaj f177f5bf11
Aristo db extend filter storage scheduler api (#1725)
* Add backwards index `[]` operator into fifo

also:
  Need another maintenance instruction: The last overflow queue must
  irrevocably delete some item in order to make space for a new one.

* Add re-org scheduler

details:
  Generates instructions how to extract and merge some leading entries

* Add filter ID selector

details:
  This allows to find the next filter now newer that a given filter ID

* Message update
2023-08-30 18:08:39 +01:00
andri lim 96fb355efe
Implement EIP-4788: Beacon block root in the EVM (#1722)
* Implement EIP-4788: Beacon block root in the EVM

* EIP-4788: Fix genesis.parentBeaconBlockRoot initialization
2023-08-30 23:29:48 +07:00
jangko 57a22bbfab
Disable sealing engine when beacon engine takes it place. 2023-08-30 21:14:38 +07:00
jangko 0b0d478966
Allow EVM dispatcher to use different GasCostKind for each fork
Why?
Some opcodes such as labeled EIP-2929 changed their behavior from fixed
gas cost to dynamic gas cost.

This changes together with #1715 and #1717 will make the new EVM tracer
to produce trace result identical to geth.
2023-08-30 21:08:35 +07:00
Daniel Sobol 85134eb24b
add hidden --disable-poke option into fluffy [#1640] (#1711) 2023-08-30 10:01:00 +02:00
jangko 7a1fe5707c
Refactor engine api and cleanup web3 types conversion 2023-08-30 10:42:46 +07:00
jangko 7d113b839f
EVM cleanup 2023-08-29 07:12:31 +07:00
jangko 26620eb672
EVM embrace more EVMC types
Also embed evmc_status_code to computation.error, and make
the tracer produce cleaner output. No more "Revert opcode executed"
error message. We can distinguish error code between REVERT
and FAILURE in a more cleaner way.
2023-08-28 21:36:23 +07:00
jangko 00262a1d48
Optimize EVM Mstore8 and memory.writePadded
- mstore8 operation is simplified using one byte writer
- refactor writePadded and avoid unecessary allocations
2023-08-28 18:04:22 +07:00
jangko 80aec9ccd9
Fix EVM tracer: capture exception properly
Also fix EVM to support new tracer
2023-08-28 14:26:43 +07:00
Jordan Hrycaj 465d694834
Aristo db implement filter storage scheduler (#1713)
* Rename FilterID => QueueID

why:
  The current usage does not identify a particular filter but uses it as
  storage tag to manage it on the database (to be organised in a set of
  FIFOs or queues.)

* Split `aristo_filter` source into sub-files

why:
  Make space for filter management API

* Store filter queue IDs in pairs on the backend

why:
  Any pair will will describe a FIFO accessed by bottom/top IDs

* Reorg some source file names

why:
  The "aristo_" prefix for make local/private files is tedious to
  use, so removed.

* Implement filter slot scheduler

details:
  Filters will be stored on the database on cascaded FIFOs. When a FIFO
  queue is full, some filter items are bundled together and stored on the
  next FIFO.
2023-08-25 23:53:59 +01:00
andri lim 91704cd3ae
Add current computation to each EVM tracer interface (#1712) 2023-08-25 16:07:20 +07:00
Kim De Mey 7aead6151e
Add utpTransfer bool to recursiveFindContent JSON-RPC result (#1710)
Rename portal protocol ContentInfo to ContentKV in the
process, due to duplicate types and simply better name
for this object.
2023-08-24 18:19:29 +02:00
jangko 820525d78c
Fix outdated EIP-4844 constants 2023-08-24 12:11:19 +07:00
jangko 8773eb609d
Fix engine api client ambiguity in engine api sim 2023-08-24 11:34:12 +07:00
Kim De Mey 745ca026fe
Return false instead of exception on failed deleteEnr JSON-RPC (#1706) 2023-08-23 21:04:17 +02:00