Commit Graph

442 Commits

Author SHA1 Message Date
KonradStaniec f4cacdfc6a
bump nimbus-eth2 (#1345)
* bump nimbus-eth2
2022-12-02 16:09:31 +01:00
KonradStaniec e78110202b
bump nim-eth (#1342) 2022-11-30 12:05:41 +01:00
Etan Kissling bc3f164b97
bump `nim-eth` for `withdrawalsRoot` support (#1326)
The `BlockHeader` structure in `nim-eth` was updated with support for
EIP-4895 (withdrawals). To enable the `nim-eth` bump, the ingress of
`BlockHeader` structures has been hardened to reject headers that have
the new `withdrawalsRoot` field until proper withdrawals support exists.
https://github.com/status-im/nim-eth/pull/562
2022-11-26 15:59:19 +01:00
KonradStaniec 4b497ec3ca
bump nim-web3 (#1309) 2022-11-21 08:02:15 +01:00
Kim De Mey ced7db901b
Bump nim-eth module for bug fixes in RLPx (#1306)
- RLP Enum DisconnectionReason deserialization fixes
- RLPx auth RangeError fixes
2022-11-17 13:52:55 +01:00
Kim De Mey cbe40285cf
Bump nim-eth for discv4 assertion error fix (#1289) 2022-11-07 17:24:19 +01:00
Kim De Mey 36d430aaa2
Add headers with proof content type and use it for verification (#1281)
* Add headers with proof content type and use it for verification

- Add BlockHeaderWithProof content type & content
- Use BlockHeaderWithProof content to verify if chain data is
part of the canonical chain
- Adjust parser & seeder code to be able to seed these headers
with proof
- Adjust eth_data_exporter to be able to export custom header
ranges for which to build proofs (mostly for testing)

There is currently quite some ugliness & clean-up needed for which
a big part is due tos upporting both BlockHeader and
BlockHeaderWithProof on the network.

* Change accumulator proof to array / SSZ vector type

- Change accumulator proof to SSZ vector instead of SSZ list.
- Add and use general buildProof and buildHeaderWithProof func.

* Make the BlockHeaderWithProof an SSZ Union with None option

* Update portal-spec-tests to master commit
2022-11-04 09:27:01 +01:00
Kim De Mey 425b3ae322
Bump stint to (hopefully) resolve issue #1275 (#1276) 2022-10-21 08:54:12 +02:00
Kim De Mey 74a83c1229
Bump nim-eth and remove all now unneeded p2p related imports (#1273) 2022-10-20 10:34:59 +02:00
Kim De Mey 9430619272
Remove Accumulator as network content type and add it to binary (#1267)
Portal master accumulator was removed from the network specs as a
content type shared on the network, as since the merge this is
a finite accumulator (pre-merge only).
So in this PR the accumulator gets removed as network type and
gets instead baked into the library. Building it is done by
seperate tooling (eth_data_exporter).
Because of this a lot of extra code can be removed that was
located in history_network, content_db, portal_protocol, etc.

Also removed to option to build the accumulator at start-up
of fluffy as this takes several minutes making it not viable.
It can still be loaded from a provided file however.

The ssz accumulator file is for now stored in the recently
created portal-spec-tests repository.
2022-10-17 20:38:51 +02:00
jangko a813865788
bump nim-eth: move eth1 specific code to here 2022-10-15 19:48:22 +07:00
KonradStaniec 538efad325
Bump nimbuseth2 (#1255)
* bump nimbus-eth2

* Remove win i386 from CI due to the fact that it is not tested in dependencies, which can lead to CI failures with updates.
2022-10-10 07:58:12 +02:00
Kim De Mey 5b4643b8ff
Bump nim-eth for rlpx double future completion fix (#1254) 2022-10-06 19:20:39 +02:00
jangko ae20b92428
bump nim-evmc 2022-10-03 16:38:32 +07:00
KonradStaniec b84c59eef2
bump nim-web3 (#1244) 2022-09-28 08:11:23 +02:00
jangko 8dff89ecad
bump confutils, serialization family, and stew 2022-09-23 10:09:53 +07:00
KonradStaniec ef54204fdc
Bump nim-json-rpc (#1236) 2022-09-22 15:21:50 +02:00
KonradStaniec 5be329741a
Bump deps related to lc-proxy (#1230) 2022-09-19 16:25:25 +02:00
KonradStaniec 2503964149
Add improved mpt trie proof verification (#1223) 2022-09-15 13:04:41 +02:00
KonradStaniec b43709f473
Add missing nimbus-eth2 modules (#1224) 2022-09-14 15:10:05 +02:00
Zahary Karadjov 4b83208a19 Switch to the new style of importing vendor packages
In preparation of our migration to the new Nimble-based setup
and [nim-workspace][1], we switch to a new model where the vendor
packages are no longer imported through a locally generated Nimble
dir, but rather through an auto-generated `nimbus-build-system.paths`
file that features regular `--path:` statements.

This file will be imported only within the nimbus-build-system
environment in order to avoid any unwanted interference in working
copies based on the new Nimble setup.

[1]: https://github.com/status-im/nim-workspace
2022-09-12 22:56:08 +03:00
jangko 113de71252
bump nim-eth: fix discovery ping pong handler 2022-09-12 15:00:28 +07:00
KonradStaniec c138a4913c
Implement getStorageAt endpoint (#1216)
* Implement getStorageAt endpoint
2022-09-09 15:59:36 +02:00
KonradStaniec 9d10f8fbae
Add mpt proof verification to proxy (#1213)
* Add mpt proof verification to proxy
2022-09-06 18:14:50 +02:00
Jacek Sieka 8d6ec3dd1a bumps
cleanups mostly
2022-09-06 11:00:40 +07:00
andri lim ad4e25b27e
Fix eth66 and eth67 handshake (#1214)
* bump nim-eth

* fix eth66 and eth67 handshake
2022-09-05 23:37:58 +02:00
KonradStaniec d6322aab22
Bump nim-web3 (#1212) 2022-09-05 11:30:18 +02:00
Jacek Sieka c2ed731fa5
eth: adapt to smaller eth_types (#1210) 2022-09-03 20:15:35 +02:00
KonradStaniec 5aaf16049f
Use separeate client and server in lcproxy (#1208)
* Use separate client and server in lc proxy
2022-08-31 07:48:48 +02:00
jangko c6f35142a8
simple peer manager to handle static peers reconnection
fix #618
2022-08-26 22:07:56 +07:00
KonradStaniec caa5e009ff
Initial Binary for light client proxy (#1202)
* Initial Binary for light client proxy
2022-08-26 13:54:10 +02:00
Kim De Mey 91a3425731
Bump nim-secp256k1 module for signature recovery id check (#1196) 2022-08-23 16:28:24 +02:00
jangko fe54f93ab6
bump nim eth
better method not implemented message of AbstractChainDB
2022-08-22 20:11:36 +07:00
Zahary Karadjov daac75796f
Extract the EIP1559 gas fee calculation in nim-eth, so it can be reused in nimbus-eth2 2022-08-22 10:52:20 +07:00
Nikolay Mitev 1e4f138574 Add nimbus-eth2 as submodule. Add launch_local_testnet script to CI. 2022-08-18 18:06:38 +03:00
Jordan Hrycaj f07945d37b
Misc snap sync updates (#1192)
* Bump nim-stew

why:
  Need fixed interval set

* Keep track of accumulated account ranges over all state roots

* Added comments and explanations to unit tests

* typo
2022-08-17 08:30:11 +01:00
KonradStaniec c5101c16e9
Bump nim eth (#1188)
* Bump nim eth

* Bump actions ubuntu
2022-08-16 09:17:57 +02:00
Jordan Hrycaj 7d7e26d45f
Experimental bulk loader tests (#1187)
why:
  Rocksdb bulk loading might provide a slight advantage when loading
  larger data sets into the system
2022-08-12 16:42:07 +01:00
KonradStaniec 4f8d10f338
Bump nim-eth (#1185) 2022-08-06 10:11:36 +02:00
Jordan Hrycaj 5f0e89a41e
Snap accounts bulk import preparer (#1183)
* Provided common scheduler API, applied to `full` sync

* Use hexary trie as storage for proofs_db records

also:
 + Store metadata with account for keeping track of account state
 + add iterator over accounts

* Common scheduler API applied to `snap` sync

* Prepare for accounts bulk import

details:
+ Added some ad-hoc checks for proving accounts data received from the
  snap/1 (will be replaced by proper database version when ready)
+ Added code that dumps some of the received snap/1 data into a file
  (turned of by default, see `worker_desc.nim`)
2022-08-04 09:04:30 +01:00
jangko a5d4759bfd
enhance net-key command line option to accept random, hex, and path
- previously it only accept hex
- fix #587
2022-07-30 08:46:11 +07:00
Jordan Hrycaj 5d98f68c09
Sync update to work with sepolia reorgs (#1168)
* Error return in `persistBlocks()` on initial `VmState` roblem

why:
  previously threw an exception

* Updated sync mode option

why:
 using enum rather than bool => space for more

* Added sync mode `full`, re-factued legacy sync

also:
  rebased

* Fix typo (crashes `pesistBlocks()` otherwise)

also:
  rebase to master

* Reduce log ticker noise by suppressing duplicate messages

* Clarify staged queue overflow handling

why:
  backtrack/re-org mode in `stageItem()` should be detected by both,
  the global indicator or the work item where it might have moved into.

also:
  rebased
2022-07-21 13:14:41 +01:00
jangko 76e438da2a
bump submodule: json-rpc and graphql
this is an update to support jwt auth and cors hook
2022-07-18 16:56:44 +07:00
jangko 6cfaaf5b45
bump json-rpc, secp256k1, blscurve, and accompanying fixes 2022-07-12 20:01:02 +07:00
Kim De Mey 87ea6ffb92
Bump most of the submodules (#1156) 2022-07-08 20:28:31 +02:00
Kim De Mey fb12793482
Bump nim-eth and nim-bearssl and accompanying fixes (#1150) 2022-07-04 09:38:02 +02:00
Kim De Mey 0068e3b427
Bump some modules to fix breakage from #1145 (#1149) 2022-07-01 22:46:36 +02:00
Jordan Hrycaj 134fe26997
Store proved snap accounts (#1145)
* Relocated `IntervalSets` to nim-stew repo

* Accumulate accounts on temporary kv-DB

why:
  Explore the data as returned from snap/1. Will be converted to a
  `eth/db` next.

details:
  Verify and accumulate per/state-root accounts downloaded via snap.

also:
  Some unit tests

* Replace `Table` by `TrieDatabaseRef` for accounts accumulator

* update ticker statistics

details:
  mean/variance based counter update

* allow persistent db for proved accounts

* rebase, and globally activate unit test

* fix statistics
2022-07-01 12:42:17 +01:00
Kim De Mey d4d4e8c28f
Add accumulator buildProof, verifyProof and verifyHeader (#1132) 2022-06-23 21:00:59 +02:00
Jacek Sieka 9e8d2d0550
sqlite: bump 2022-06-21 16:18:41 +02:00
andri lim 5374874552
bump serialization family(serialization, json, toml) (#1135) 2022-06-20 11:32:41 +02:00
KonradStaniec 0776f35e0c
Use queries with custom xor function (#1113)
* Use queries with custom xor function
2022-06-03 13:44:42 +02:00
KonradStaniec 5c78fe64e4
Fix utp connection leak on cancel (#1107)
* Fix utp connection leak on cancel
2022-05-31 14:28:02 +02:00
KonradStaniec b975d35c84
Bump sqlite-abi (#1108) 2022-05-31 12:37:10 +02:00
jangko b03c2dca4f
bump nim-web3: engine api version alpha.9 2022-05-29 11:23:57 +07:00
Jordan Hrycaj ba940a5ce7
Snap sync simplify object inheritance (#1098)
* Reorg SnapPeerBase descriptor, notably start/stop flags

details:
  Instead of using three boolean flags startedFetch, stopped, and
  stopThisState a single enum type is used with values SyncRunningOk,
  SyncStopRequest, and SyncStopped.

* Restricting snap to eth66 and later

why:
  Id-tracked request/response wire protocol can handle overlapped
  responses when requests are sent in row.

* Align function names with source code file names

why:
  Easier to reconcile when following the implemented logic.

* Update trace logging (want file locations)

why:
  The macros previously used hid the relevant file location (when
  `chroniclesLineNumbers` turned on.) It rather printed the file
  location of the template that was wrapping `trace`.

* Use KeyedQueue table instead of sequence

why:
  Quick access, easy configuration as LRU or FIFO with max entries
  (currently LRU.)

* Dissolve `SnapPeerEx` object extension into `SnapPeer`

why;
  It is logically cleaner and more obvious not to inherit from
  `SnapPeerBase` but to specify opaque field object references of the
  merged `SnapPeer` object. These can then be locally inherited.

* Dissolve `SnapSyncEx` object extension into `SnapSync`

why;
  It is logically cleaner and more obvious not to inherit from
  `SnapSyncEx` but to specify opaque field object references of
  the `SnapPeer` object. These can then be locally inherited.

  Also, in the re-factored code here the interface descriptor
  `SnapSyncCtx` inherited `SnapSyncEx` which was sub-optimal (OO
  inheritance makes it easier to work with call back functions.)
2022-05-23 17:53:19 +01:00
jangko 41e9ea8d45
bump nim-confutils
why:
- fix Acc Quoted ident processsing bug in config-file
- fix Dot Expr processsing bug in config-file
2022-05-10 16:04:21 +07:00
Jordan Hrycaj 3b9f0e3980
Proper Uint256 parser for JSON genesis (#1071)
* Update exception tracinig

* Use lazy JSON parser

why:
  Uint246 type is not directly supported by the JSON serializer

* Json parser update for stringified UInt256 integers

why:
  Now available

* update sub-module branch reference
2022-05-06 09:02:28 +01:00
Kim De Mey 7bd9afa465
Bump nim-eth for max discv5 packet size (#1070) 2022-05-03 09:18:33 +02:00
jangko e2c9ec4f84
update submodules to accomodate styleCheck 2022-04-14 08:39:50 +07:00
Kim De Mey d337806301
Improve logs and loglevels, and general cleanup (#1059) 2022-04-13 07:56:01 +02:00
Kim De Mey 82aab2a404
Bump chronos and json-rpc and avoid http conn re-use for blockwalk (#1053)
Currently re-using http connections for the json-rpc requests
seems to slow down the requests considerably. Avoid this by
forcing a close after each requests in the blockwalk tool.
2022-04-11 19:39:45 +02:00
Jordan Hrycaj 8af5c33ef9
Facilitate http code response on websocket JWT authentication failure (#1043)
* Facilitate http code response on websocket JWT authentication failure

* Update JSON-RPC link
2022-04-07 10:37:35 +01:00
Jordan Hrycaj 737236fd6e
Enable JWT authentication for websockets (#1039)
* Enable JWT authentication for websockets

details:
  Currently, this is optional and only enabled when the jwtsecret option
  is set.

  There is a default mechanism to generate a JWT secret if it is not
  explicitly stated. This mechanism is currently unused.

* Make JWT authentication compulsory for websockets

* Fix unit test entry point + cosmetics

* Update JSON-RPC link

* Improvements as suggested by Mamy
2022-04-06 15:11:13 +01:00
Kim De Mey 16ecbeb902
Bump nim-eth for lower loglevel of rlpx and discv4 harmless errors (#1041) 2022-04-06 11:57:08 +02:00
Jacek Sieka 1d6a9951d6
move rocksdb support to eth1 (#927)
* move rocksdb support to eth1

only used here / causes unnecessary build deps
2022-04-06 07:28:19 +02:00
Jacek Sieka e9afa00f16 snappy revamp
not entirely sure why nimbus-eth1 has a copy of snappy but here's a new
version: https://github.com/status-im/nim-snappy/pull/10
2022-04-05 12:03:01 +07:00
Jacek Sieka 60032bf106
Bump chronos (#925)
Memory fixes, http fixes, fix for issue #963, etc
2022-03-31 14:55:01 +02:00
KonradStaniec 75f2c76ad0
Bump nim-eth module (#1017) 2022-03-29 21:09:49 +02:00
Jordan Hrycaj 4696a53302
Enable optional chunked RLPx messages (#1010)
* Enable optional chunked RLPx messages

why:
  Legacy feature used by Nethermind

details:
  Disable with make flag: ENABLE_CHUNKED_RLPX=0

* Rebase & bump nim-eth

* Fix default behaviour

why:
  Got lost somehow. Comments do not match GNU make code directives.
2022-03-29 10:19:32 +01:00
Ștefan Talpalaru 51bc1cf87f
dist: precompiled binaries and Docker images (#1015)
* dist: precompiled binaries and Docker images

The builds are reproducible, the binaries are portable and statically link librocksdb.

This took some patching. Upstream PR: https://github.com/facebook/rocksdb/pull/9752

32-bit ARM is missing as a target because two different GCC versions
fail with an ICE when trying to cross-compile RocksDB. Using Clang
instead is too much trouble for a platform that nobody should be using
anyway.

(Clang doesn't come with its own target headers and libraries, can't be
easily convinced to use the ones from GCC, so it needs an fs image from
a 32-bit ARM distro - at which point I stopped caring).

* CI: disable reproducibility test
2022-03-27 13:21:15 +02:00
jangko a0ba5a492c
bump some submodules
- graphql: bump version 0.2.20 to 0.2.22
- json-rpc: various bug fixes
- websock: various bug fixes
- http-utils: chronos dependancy
2022-03-25 17:16:21 +07:00
KonradStaniec ad51bd68fc
Bump nim-eth (#1004) 2022-03-21 20:18:43 +01:00
Kim De Mey b00ac490a9
Bump nim-eth and adjust API to support binding to specific address (#994)
- Fixes also a bug which would not start an EthereumNode properly
when no bootstrap nodes are provided
2022-03-16 09:01:35 +01:00
KonradStaniec 2c2bcae00e
Bump nim-eth (#986) 2022-03-04 14:36:32 +01:00
jangko 0d18b606f5
bump json-rpc 2022-02-28 10:36:35 +07:00
jangko 1f9ed449d9
bump nim-web3
update Engine API from v1-alpha6 to v1-alpha7
2022-02-27 14:37:35 +07:00
jangko b46d60f65d
bump nim-json-rpc
null value of optional param rejected by serializer
because of a bug in `jsonmarshal.nim` now fixed
2022-02-14 21:23:00 +07:00
Kim De Mey 38036966a6
Improve the tests of the local testnet (#953)
* Improve the tests of the local testnet

The local testnet test was rather flaky and would occasionally
fail. It has been made more robust by adding the ENRs directly
to the routing table instead of doing some random lookups.

Additionally, the amount of nodes were increased (=64), ip limits
configuration was added, and the bits-per-hop value was set to 1
in order to make the lookups more likely to hit the network
instead of only the local routing table.

Failure is obviously still possible to happen when sufficient
packets get lost. If this turns out to be the case with the current
amount of nodes, we might have to revise the testing strategy here.

* Disable lookup test for State network

Disable lookup test for State network due to issue with custom
distance function causing the lookup to not always converging
towards the target.
2022-02-02 22:48:33 +01:00
jangko 3e60948785 bump nim-eth
changes because of EIP-4399 preparation:
- `mixDigest` field of BlockHeader got alias `random`
2022-02-01 18:11:14 +02:00
Kim De Mey ac1196d915
Bump nim-ssz-serialization and clean-up fromSszBytes raises (#948) 2022-01-31 21:57:34 +01:00
KonradStaniec 296bf3156d
Bump nim-eth (#946)
implementation of utp fast resends
implementation of utp fast resends when processing selective acks
more logging for utp
2022-01-28 06:55:13 +01:00
Zahary Karadjov 3cbb920406
Migrate to Engine API spec version v1.0.0-alpha.5; More progress towards working M1 2022-01-25 12:52:50 +02:00
KonradStaniec 990846135a
Bump nim-eth (#941)
* Bump nim-eth
2022-01-25 10:19:16 +01:00
Zahary Karadjov 137eb97766 Initial implementation of the merge spec
Includes a simple test harness for the merge interop M1 milestone

This aims to enable connecting nimbus-eth2 to nimbus-eth1 within
the testing protocol described here:

https://github.com/status-im/nimbus-eth2/blob/amphora-merge-interop/docs/interop_merge.md

To execute the work-in-progress test, please run:

In terminal 1:
tests/amphora/launch-nimbus.sh

In terminal 2:
tests/amphora/check-merge-test-vectors.sh
2022-01-24 09:44:39 +02:00
Zahary Karadjov e7adc609f4
Bump nim-stew 2022-01-23 19:25:46 +02:00
Zahary Karadjov 8ef9c620cb
Bump nim-json-rpc 2022-01-23 19:19:24 +02:00
Jacek Sieka f47c710ada
stew: bump (#924)
`Result` changes and `endians2` performance fix
2022-01-13 21:11:54 +01:00
Kim De Mey 73b4a73423
Bump nim-eth and nim-ssz-serialization vendor modules (#926)
- nim-eth: uTP updates, ENR improvemens and extra discv5 metrics
- nim-ssz-serialization: require BitList pairs
- update github actions cache keys to invalidate cache
2022-01-12 21:41:23 +01:00
Jordan Hrycaj 1f774c01a2
Jordan/accounts cache scenario (#904)
* crash test scenario

details:
  Example code for inspecting nested block chain and accounts cache
  database transaction framework. There seems to be a pathological
  case where the system crashes after a rollback (as appeared in the
  tx-pool packer code.)

* simplified crash scenario

* Workable solution (as suggested by Andri)

details:
  Avoiding db.rollback() (db.commit() is OK) while vmState.stateDB is
  alive.

* Rename text_txcrash => test_accounts_cache

why:
  Unit tests covers part of accounts_cache handling

* comment update
2021-12-13 11:58:05 +00:00
Kim De Mey 70625bc02c
Use one bootstrap nodes argument for discv5 and Portal (#911)
Currently bootstrap nodes for discv5 and for the Portal nodes
were provided through separate cli arguments. This is however
confusing and cumbersome as typically when (currently) testing
a node will have both discv5 and the Portal networks enabled.
We merge them into one argument.

If a node happens not to support a Portal network, it will be
removed after message request failure.

Commit also contains additional clean-up and nim-eth bump.
2021-12-13 09:06:29 +01:00
jangko d5d0a44b0e
bump nim-confutils with addition features needed by nimbus-eth1
new features:
- multiple lines long description
- ignore property in addition to hidden
2021-12-12 16:38:10 +07:00
Kim De Mey 63b0945b52
Use json rpc client to run tests on portal testnet (#899)
Tests have also been added & adjusted. Original ones were actually
failing but unnoticed due to mistake in port passed in curl command.
2021-12-03 09:51:25 +01:00
jangko dcd86bb221
vendor: bump nim-json-rpc
bugfix and features:
- Switch to the Chronos HTTP client (adds support for HTTPS)
- Allow dynamic RPC method names in the 'rpc' macro
- Restore the support for using the news package
2021-11-30 11:47:11 +07:00
Kim De Mey 903350bdde
Add local testnet script and required json-rpc calls (#891)
- Add basic discv5 and portal json-rpc calls and activate them in
fluffy
- Renames in the rpc folder
- Add local testnet script and run this script in CI
- bump nim-eth
2021-11-24 08:45:55 +01:00
KonradStaniec f9ed83c221
bump nim eth (#890) 2021-11-22 12:35:08 +01:00
Kim De Mey 02afda1b45
Run everything fluffy with chronosStrictException (#889) 2021-11-18 17:52:44 +01:00
Kim De Mey 78f585399f
Bump submodules (#887)
* Bump submodules

* Fix missing import for state and history networks
2021-11-18 12:06:53 +01:00
Kim De Mey bdbd8e4c87
Add SSZ Unions through case objects (#882)
* Add SSZ Unions through case objects

* Add connection id content response test and improve other test vectors

* Implement content keys and ids for state network as per spec

Content keys case object is used so that it can be serialized and
deserialized as an SSZ Union.

* Let message Union in Portal wire protocol start at 0 as per new spec
2021-11-17 17:11:17 +01:00
kdeme 6de2be146a
Bump unittest2 and testutils 2021-11-14 16:02:22 +07:00