Commit Graph

3074 Commits

Author SHA1 Message Date
Jordan Hrycaj fe04b50fef
Slightly change the static peer manager lookup behaviour (#1484)
why:
  The peer manager runs concurrently to the discovery scheme. So the p2p
  peer observer will also present `peer` non-static entries. Previously,
  this peer manager throw an assert defect when this happened.
2023-03-06 09:22:07 +00:00
Jordan Hrycaj 10ad7867e4
Prepare snap server client test scenario cont1 (#1485)
* Renaming androgynous sub-object names according to where they belong

why:
  These objects are not explicitly dealt with. They give meaning to
  some generic wrapper objects. Naming them after their origin may
  help troubleshooting.

* Redefine proof nodes list data type for `snap/1` wire protocol

why:
  The current specification suffered from the fact that the basic data
  type for a proof node is an RLP encoded hexary node. This slightly
  confused the encoding/decoding magic.

details:
  This is the second attempt, now wrapping the `seq[Blob]` into a
  wrapper object of `seq[SnapProof]` for a distinct alias sequence.

  In the previous attempt, `SnapProof` was a wrapper object holding the
  `Blob` with magic applied to the `seq[]`. This needed the `append`
  mixin to strip the outer wrapper that was applied to the `Blob` already
  when it was passed as argument.

* Fix some prototype inconsistency

why:
  For easy reading, `getAccountRange()` handler return code should
  resemble the `accoundRange()` anruments prototype.
2023-03-03 20:01:59 +00:00
Jordan Hrycaj f20f20f962
Prepare snap server client test scenario (#1483)
* Enable `snap/1` accounts range service

* Allow to change the garbage collector to `boehm` as a Makefile option.

why:
  There is still an unsolved memory corruption problem that might be
  related to the standard `gc`. It seemingly goes away if the `gc` is
  changed to `boehm`.

  Specifying another `gc` on the make level simplifies debugging and
  development.

* Code cosmetics

details:
* updated exception annotations
* extracted `worker_desc.nim` from `full/worker.nim`
* etc.

* Implement option to state a sync modifier file

why:
  This allows to specify extra sync type specific options which might
  change over time. This file is regularly checked for updates.

* Implement a threshold when to suspend full syncing

why:
  For a test scenario, a full sync beep may work as a local snap server.
  There is no need to download the full block chain.

details:
  The file containing the pivot specs is specified by the
  `--sync-ctrl-file` option. It is regularly parsed for updates.
2023-03-02 09:57:58 +00:00
Kim De Mey 06512409ea
Fix, rework and re-enable Portal beacon light client (#1481) 2023-02-26 19:18:03 +01:00
Kim De Mey b64135b830
Bump nimbus-eth2 and fix beacon LC related changes (#1473)
* Bump nimbus-eth2 and fix beacon LC related changes

* Disable local_testnet test in CI
Latest version is no longer compatible with the old nightlies that are being downloaded for the test
2023-02-26 16:44:05 +01:00
jangko 155652360a
fix flaky simulators ci 2023-02-24 02:04:17 +07:00
jangko 55b4baa6ba bump submodules to reduce compiler warnings 2023-02-23 20:14:29 +07:00
Jordan Hrycaj bf53226c2c
Minor updates for testing and cosmetics (#1476)
* Fix locked database file annoyance with unit tests on Windows

why:
  Need to clean up old files first from previous session as files remain
  locked despite closing of database.

* Fix initialisation order

detail:
  Apparently this has no real effect as the ticker is only initialised
  here but started later.

  This possible bug has been in all for a while and was running with the
  previous compiler and libraries.

* Better naming of data fields for sync descriptors

details:
* BuddyRef[S,W]: buddy.data -> buddy.only
* CtxRef[S]: ctx.data -> ctx.pool
2023-02-23 13:13:02 +00:00
jangko 2f6b4de3e9 ci: fix nightly build 2023-02-23 18:34:04 +07:00
jangko a74bbef74c
evmstate tool: fix regression 2023-02-23 09:15:58 +07:00
jangko f5c12062bd
bump submodules to reduce compiler warnings 2023-02-22 12:21:41 +07:00
Zahary Karadjov e47387d41f Flake-based direnv setup for nixOS users 2023-02-20 18:01:49 +02:00
Adam Spitz fad3ed64cf
Time based forking (#1465)
* Refactoring in preparation for time-based forking.

* Timestamp-based hard-fork-transition.

* Workaround SideEffect issue / compiler bug for both failing locations in Portal history code

---------

Co-authored-by: kdeme <kim.demey@gmail.com>
2023-02-16 12:40:07 +01:00
jangko 54958d4731 bump submodules to reduce compiler warnings 2023-02-16 08:56:37 +07:00
Jordan Hrycaj b793f0de8d
Snap sync extractor and sub range proofs cont1 (#1468)
* Redefine `seq[Blob]` => `seq[SnapProof]` for `snap/1` protocol

why:
  Proof nodes are traded as `Blob` type items rather than Nim objects. So
  the RLP transcoder must not extra wrap proofs which are of type
  seq[Blob]. Without custom encoding one would produce a
  `list(blob(item1), blob(item2) ..)` instead of `list(item1, item2 ..)`.

* Limit leaf extractor by RLP size rather than number of items

why:
  To be used serving `snap/1` requests, the result of function
  `hexaryRangeLeafsProof()` is limited by the maximal space
  needed to serialise the result which will be part of the
  `snap/1` repsonse.

* Let the range extractor `hexaryRangeLeafsProof()` return RLP list sizes

why:
  When collecting accounts, the size oft the accounts list when encoded
  as RLP is continually updated. So the summed up value is available
  anyway. For the proof nodes list, there are not many (~ 10) so summing
  up is not expensive here.
2023-02-15 10:14:40 +00:00
Jordan Hrycaj 880313d7a4
Silence some compiler gossip -- part 8, sync (#1467)
details:
  Adding some missing exception annotation
2023-02-14 23:38:33 +00:00
Jordan Hrycaj df1217b7ca
Silence compiler gossip after nim upgrade cont3 (#1466)
* Removed some Windows specific unit test annoyances

details:
+ Short put()/get() cycles on persistent database have a race condition
  with vendor rocksdb. On a specific (and slow) qemu/win7 a 50ms `sleep()`
  in between will mostly do the job (i.e. unless heavy CPU load.) This
  issue was not observed on github/ci.
+ Removed annoyances when qemu/Win7 keeps the rocksdb database files
  locked even after closing the db. The problem is solved by strictly
  using fresh names for each test. No assumption made to be able to
  properly clean up. This issue was not observed on github/ci.

* Silence some compiler gossip -- part 7, misc/non(sync or graphql)

details:
  Adding some missing exception annotation
2023-02-14 20:27:17 +00:00
Jordan Hrycaj 7490f23124
Silence some compiler gossip -- part 6, evm (#1462)
* Silence some compiler gossip -- part 6, evm

details:
  Adding some missing exception annotation

* Update evmc cases

why:
  were previously missing

* Increase Windows stack needed to run EVMC unit tests

why:
  After annotating functions to trace exceptions some unit tests started
  to fail on Windows without clear error report.

  EVMC works recursively and now there seems to be a stack problem
  reported by the nim compiler. Increasing the NIM stack ass sugessted by
  NIM (using -d:nimCallDepthLimit=###) had some effect but no clear
  solution.

Note that this patch set unrolls some NIM compiler settings
2023-02-14 14:37:21 +00:00
jangko 28129d1df2
evm: fix premature position truncation of byte op 2023-02-13 21:20:52 +07:00
Kim De Mey 77135e7001
Avoid raw Exception raises by moving confutils config load (#1461) 2023-02-07 17:21:12 +01:00
Jordan Hrycaj c2fc46a99a
Snap sync extractor test sub range proofs (#1460)
* Unit tests to verify calculations based on hard coded constants

why:
  Sizes of RLP encoded objects are available at run time only.

* Changed argument order for `hexaryRangeLeafsProof()` prototype

why:
  Better to read as a stand-alone function (arguments were optimised
  for functional pipelines)

* Run sub-range proof tests for extracted ranges
2023-02-02 13:27:09 +00:00
Kim De Mey ba92c53624
Fix potential overflow error on toPC (#1458)
One might want to rework that toPC code to avoid this oddities.
Consider this a quick fix.
2023-02-02 09:30:14 +01:00
jangko 7cb146c037
fluffy ci: both testutp and linux test using ubuntu 22.04
to avoid conflict because if testutp using ubuntu-latest
cached Nim compiler will be used by linux test which is using
ubuntu 20.04, then missing glibc 2.33 error. ubuntu 20.04 still
using glibc 2.31
2023-02-02 11:11:01 +07:00
jangko 0f065e91b5
bump submodules to further reduce compiler warnings 2023-02-02 09:29:02 +07:00
Jordan Hrycaj 6ca6bcd96f
Snap sync fix trie interpolation fringe condition (#1457)
* Cosmetics

details:
+ Update doc generator
+ Fix key type representation in `hexary_desc` for debugging
+ Redefine `isImportOk()` as template for better `check()` line reporting

* Fix fringe condition when interpolating Merkle-Patricia tries

details:
  Small change with profound effect fixing some pathological condition
  that haunted the unit test set on large data sers. There is still one
  condition left which might well be due to an incomplete data set.

* Unit test proof nodes for node range extractor

* Unit tests to run on full extraction set

why:
  Left over from troubleshooting, range length was only 5
2023-02-01 18:56:06 +00:00
Kim De Mey 73e93f1f11
Reduce Nim 1.6 compiler warnings & hints
* Reduce Nim 1.6 compiler warnings/hints for Fluffy and Nimbus proxy

Mostly raises Defect removals, TaintedString removal and some
unnecessary imports.

Also updating the copyright years alongside.

* Further reduce Nim 1.6 compiler warnings/hints for Nimbus
2023-01-31 13:38:08 +01:00
Jordan Hrycaj 6b9f3c9ac5
Silence compiler gossip after nim upgrade cont1 (#1455)
* Silence some compiler gossip -- part 5, common

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Silence some compiler gossip -- part 6, db, rpc, utils

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Silence some compiler gossip -- part 7, randomly collected source files

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Silence some compiler gossip -- part 8, assorted tests

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Clique update

why:
  More impossible exceptions (undoes temporary fix from previous PR)
2023-01-31 01:32:17 +00:00
Jordan Hrycaj 89ae9621c4
Silence compiler gossip after nim upgrade (#1454)
* Silence some compiler gossip -- part 1, tx_pool

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Silence some compiler gossip -- part 2, clique

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Silence some compiler gossip -- part 3, misc core

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Silence some compiler gossip -- part 4, sync

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Clique update

why:
  Missing exception annotation
2023-01-30 22:10:23 +00:00
Jordan Hrycaj 197d2b16dd
Snap sync interval range extractor (#1449)
* Update comments and test noise

* Fix boundary proofs

why:
  Where neither used in production, nor unit tested. For production, other
  methods apply to test leaf range integrity directly based of the proof
  nodes.

* Added `hexary_range()`: interval range + proof extractor

details:
+ Will be used for `snap/1` protocol handler
+ Unit tests added (also for testing left boundary proof)

todo:
  Need to verify completeness of proof nodes

* Reduce some nim 1.6 compiler noise

* Stop unit test gossip for ci tests
2023-01-30 17:50:58 +00:00
Kim De Mey d65bb18ad2
Fix nimbus CI yml file introduced when disabling x86 (#1450) 2023-01-27 16:05:30 +01:00
Kim De Mey 8523d8b40e
Bump submodules and fix related issues (#1448)
* Updated to the latest nim-eth, nim-rocksdb, nim-web3

* Bump nimbus-eth2 module and fix related issues

Temporarily disabling Portal beacon light client network as it is
a lot of copy pasted code that did not yet take into account
forks. This will require a bigger rework and was not yet tested
in an actual network anyhow.

* More nimbus fixes after module bumps

---------

Co-authored-by: Adam Spitz <adamspitz@status.im>
Co-authored-by: jangko <jangko128@gmail.com>
2023-01-27 15:57:48 +01:00
Kim De Mey a669b51ec5
Bump Nim to 1.6 and resolve the related issues (#1445)
Two unresolved items currently:
- Three tests that are temporarily disabled as they fail in the
macro_assembler code, which seems to be due to an ambigious
identifier Stop (Ops and chronos ServerCommand enum).
- i386 CI disabled as it fails at Nim compilation already. Failed
tests where already ignored for this target.
2023-01-26 13:37:19 +01:00
Jordan Hrycaj 1b441a485b
Replace coded unit test PRNG for Clique test address generation (#1446)
why:
  Clique relies on the even/odd position of an address after sorting. For
  address generation, the Nim PRNG was used which seems to have changed
  with Nim 1.6.11 (Linux, Windoes only.)

  As a replace, the Posix.1-2001 example (two-liner calculation) generator
  is used.
2023-01-24 14:52:02 +00:00
Jordan Hrycaj e093fa452d
Declutter snap sync unit tests (#1444)
* Extracted RocksDB timing unit tests into separate file

why:
  make space for more in main module :)

* Extracted `inspectionRunner()` unit tests into separate file

why:
  make space for more in main module :)

* Extracted `storagesRunner()` unit tests into separate file

why:
  make space for more in main module :)

* Extracted pivot checkpoint store/retrieval unit tests into separate file

why:
  make space for more in main module :)

* Extract helper functions into separate source file

* Extracted account import unit tests into separate file

why:
  make space for more in main module :)

* Rename `test_decompose()` => `test_NodeRangeDecompose()`

why:
  There will be more functions with `test_NodeRange` prefix.
2023-01-23 16:09:12 +00:00
Kim De Mey 9b1193c402
Revert "Updated to the latest nim-eth, nim-rocksdb, nim-web3. (#1441)" (#1443)
This reverts commit 1c1a3352b3.
2023-01-23 10:50:17 +01:00
Adam Spitz 1c1a3352b3
Updated to the latest nim-eth, nim-rocksdb, nim-web3. (#1441) 2023-01-21 15:04:22 +01:00
Kim De Mey 12f66ae598
Move BlockHeaderWithProof content to content key selector 0 (#1379)
* Move BlockHeaderWithProof content to content key selector 0

- Remove as content type with content key selector 4
- Replace regular block header with BlockHeaderWithProof at
content key selector 0

* Apply blockHeader content key also to bridge

* Add tests for header with proof generation and verification
2023-01-20 23:04:58 +01:00
Jordan Hrycaj 6fb48517ba
Add snap protocol service stub (#1438)
* Cosmetics, update logger `topics`

* Clean up sync/start methods in nimbus

why:
* The `protocols` list selects served (as opposed to sync) protocols only.
* The `SyncMode.Default` object is allocated with the other possible sync
  mode objects.

* Add snap service stub to `nimbus`

* Provide full set of snap response handler stubs

* Bicarb for the latest CI hiccup

why:
  Might be a change in the CI engine for MacOS.
2023-01-20 15:01:29 +00:00
Kim De Mey 9efb40336b
Add possible historical_roots with proof structure (#1439)
This is an example of how the beacon state historical_roots could
be added with a proof to be able to provide on the network and
verify by means of proof against the state root.
2023-01-20 13:19:49 +01:00
Kim De Mey 3a079bf30e
Add first iteration of a fluffy beacon bridge for block data (#1437) 2023-01-19 09:33:11 +01:00
Jordan Hrycaj fda7971aaf
Reorganise eth handlers (#1436)
* Reorganise eth handlers

why:
  Make space for `snap` handlers in a similar fashion.

* fix typo
2023-01-18 15:00:14 +00:00
Jordan Hrycaj 30135ab1ef
Simplify beacon stream pivot update (#1435)
* Simplify pivot update

why:
  No need to fetch the pivot header from the network when it can be
  be made available in the ivot cache

also:
  Keep `txPool` update disabled while syncing

* Cosmetics, tune down some logging noise

* Support `snap/1` without `eth/6?`

why:
  Eth is not needed here.

* Snap is an (optional) extension of `eth`

so:
  It it must be supported somehow. Nevertheless it will be currently
  unused in the snap syncer.
2023-01-18 08:31:57 +00:00
Kim De Mey a1163b4ade
Do not allow pre-merge headers without an AccumulatorProof (#1426) 2023-01-17 14:47:22 +01:00
Kim De Mey 52a2257922
Adjust Portal JSON-RPC Offer call and add Gossip call (#1418) 2023-01-17 14:47:10 +01:00
Jordan Hrycaj 707e47ac38
External beacon stream tracker (#1433)
* Register external beacon stream header

why:
  This will be used to sync the peers against.

* Update total coverage book-keeping for 100% roll-over

details:
  Provide commonly available/used function

* Replace best pivot by beacon stream tracker

details:
  Beacon stream header cache will be updated by external chain monitor via
  RPC. This cached header will then be used to sync the pivot.
2023-01-17 09:28:14 +00:00
Jordan Hrycaj a6f45e341b
Fetch-reject-reconnect loop protection (#1432)
why:
  Some peers reconnect recurrently after dialogue was found useless. The
  reconnect loop protection was in place already, albeit insufficient.

also:
  Some updates to allow setting previously constant parameters at run
  time.
2023-01-16 14:51:32 +00:00
jangko be20e19991
fix EIP-3860 intrinsic gas once again 2023-01-15 14:37:19 +07:00
jangko 6814140c63
unify coinbase state clearing of gst, evmstate, and t8n
it is troublesome if we have to fix it in three places
in case of a bug, it also reduce code duplication.
2023-01-14 17:17:55 +07:00
jangko e2601e7c54
bump eth_tests 2023-01-14 15:56:08 +07:00
jangko 192646ca8a
fix test_blockchain_json lastBlockHash comparison 2023-01-14 15:56:08 +07:00