Commit Graph

3014 Commits

Author SHA1 Message Date
Kim De Mey 1cdbc1293d
Fix git branch name in Fluffy docker nightly build (#1404)
And change build hour temporarily
2023-01-03 10:08:24 +01:00
Kim De Mey b9ce04ecfc
Add simple nightly Docker image build for Fluffy (#1403) 2023-01-02 20:50:52 +01:00
Kim De Mey 624c87d8b3
Add experimental beacon chain block proofs for Portal network (#1376)
* Add experimental beacon chain block proofs for Portal network

* Add Caveat on proving recent blocks
2023-01-02 16:46:48 +01:00
Kim De Mey 1bfbf4f02f
Clarify Fluffy's docs and config (#1402) 2023-01-02 16:37:22 +01:00
jangko 74e76e5237
remove unused 'refundGas' from evm/state_transactions 2022-12-28 01:45:56 +07:00
KonradStaniec 2bce04afc7
Add light client mode to fluffy (#1396) 2022-12-27 15:57:10 +01:00
KonradStaniec 270ce41d5c
Add light client bridge (#1386)
* Add light client bridge binary
2022-12-27 15:25:20 +01:00
Jordan Hrycaj 5134bb5e04
Extract finding of missing nodes for healing into separate module (#1398)
why:
  Duplicate implementation of same functionality
2022-12-25 17:56:57 +00:00
Jordan Hrycaj 88b315bb41
Snap sync refactor healing (#1397)
* Simplify accounts healing threshold management

why:
  Was over-engineered.

details:
  Previously, healing was based on recursive hexary trie perusal.

  Due to "cheap" envelope decomposition of a range complement for the
  hexary trie, the cost of running extra laps have become time-affordable
  again and a simple trigger mechanism for healing will do.

* Control number of dangling result nodes in `hexaryInspectTrie()`

also:
+ Returns number of visited nodes available for logging so the maximum
  number of nodes can be tuned accordingly.
+ Some code and docu update

* Update names of constants

why:
  Declutter, more systematic naming

* Re-implemented `worker_desc.merge()` for storage slots

why:
  Provided as proper queue management in `storage_queue_helper`.

details:
+ Several append modes (replaces `merge()`)
+ Added third queue to record entries currently fetched by a worker. So
  another parallel running worker can safe the complete set of storage
  slots in as checkpoint. This was previously lost.

* Refactor healing

why:
  Simplify and remove deep hexary trie perusal for finding completeness.

   Due to "cheap" envelope decomposition of a range complement for the
   hexary trie, the cost of running extra laps have become time-affordable
   again and a simple trigger mechanism for healing will do.

* Docu update

* Run a storage job only once in download loop

why:
  Download failure or rejection (i.e. missing data) lead to repeated
  fetch requests until peer disconnects, otherwise.
2022-12-24 09:54:18 +00:00
jangko e36d2f432a
fix default --key-store location if --data-dir is custom 2022-12-22 11:17:04 +07:00
jangko d98b3bb32d
cleanup numeric utils and remove unstable rangeToPadded 2022-12-21 18:41:03 +07:00
Jordan Hrycaj 0f132c1d01
Snap sync fix ticker crash (#1393)
* Fix SEGFAULT showstopper

* Update logging
2022-12-20 15:38:57 +00:00
Jordan Hrycaj bd42ebb193
Snap sync refactor accounts healing (#1392)
* Relocated mothballing (i.e. swap-in preparation) logic

details:
  Mothballing was previously tested & started after downloading
  account ranges in `range_fetch_accounts`.

  Whenever current download or healing stops because of a pivot change,
  swap-in preparation is needed (otherwise some storage slots may get
  lost when swap-in takes place.)

  Also, `execSnapSyncAction()` has been moved back to `pivot_helper`.

* Reorganised source file directories

details:
  Grouped pivot focused modules into `pivot` directory

* Renamed `checkNodes`, `sickSubTries` as `nodes.check`, `nodes.missing`

why:
  Both lists are typically used together as pair. Renaming `sickSubTries`
  reflects moving away from a healing centric view towards a swap-in
  attitude.

* Multi times coverage recording

details:
  Per pivot account ranges are accumulated into coverage range set. This
  set fill eventually contain a singe range of account hashes [0..2^256]
  which amounts to 100% capacity.

  A counter has been added that is incremented whenever max capacity is
  reached. The accumulated range is then reset to empty.

  The effect of this setting is that the coverage can be evenly duplicated.
  So 200% would not accumulate on a particular region.

* Update range length comparisons (mod 2^256)

why:
  A range interval can have sizes 1..2^256 as it cannot be empty by
  definition. The number of points in a range intervals set can have
  0..2^256 points. As the scalar range is a residue class modulo 2^256,
  the residue class 0 means length 2^256 for a range interval, but can
  be 0 or 2^256 for the number of points in a range intervals set.

* Generalised `hexaryEnvelopeDecompose()`

details:
  Compile the complement of the union of some (processed) intervals and
  express this complement as a list of envelopes of sub-tries.

  This facility is directly applicable to swap-in book-keeping.

* Re-factor `swapIn()`

why:
  Good idea but baloney implementation. The main algorithm is based on
  the generalised version of `hexaryEnvelopeDecompose()` which has been
  derived from this implementation.

* Refactor `healAccounts()` using `hexaryEnvelopeDecompose()` as main driver

why:
  Previously, the hexary trie was searched recursively for dangling nodes
  which has a poor worst case performance already when the trie  is
  reasonably populated.

  The function `hexaryEnvelopeDecompose()` is a magnitude faster because
  it does not peruse existing sub-tries in order to find missing nodes
  although result is not fully compatible with the previous function.

  So recursive search is used in a limited mode only when the decomposer
  will not deliver a useful result.

* Logging & maintenance fixes

details:
  Preparation for abandoning buddy-global healing variables `node`,
  `resumeCtx`, and `lockTriePerusal`. These variable are trie-perusal
  centric which will be run on the back burner in favour of
  `hexaryEnvelopeDecompose()` which is used for accounts healing already.
2022-12-19 21:22:09 +00:00
Kim De Mey 05ac755e84
Set disableMarchNative for Fluffy Dockerfile (#1391) 2022-12-19 13:19:50 +01:00
Jordan Hrycaj d55a72ae49
Full sync peer negotiation control (#1390)
* Additional logging for scheduler

* Fix duplicate occurrence of `bestNumber`

why:
  Happened when the `block_queue` module was separated out of
  the `worker` module. Somehow testing was insufficient or skipped,
  at all.

* Update `runPool()` mixin for scheduler

details:
  Could be simplified

* Dynamically adapt pivot header negotiation mode

details:
  After accepting one peer and some timeout, do not search for more
  peers for start syncing but rather continue in relaxed mode with a
  single peer.
2022-12-18 16:06:43 +00:00
Kim De Mey 69204fffe9
Update two of the Fluffy bootstrap nodes for testnet (#1389) 2022-12-16 21:11:34 +01:00
Kim De Mey 591a1e7062
Add content verifier tool check data availability on the network (#1388) 2022-12-16 21:11:09 +01:00
Kim De Mey 733d7e5ceb
Add new portal JSON-RPC OfferReal as Offer is doing gossip now (#1387)
The portal_*Offer call was changed in the specs to actually do
gossip. Make it no longer possible to test purely an offer with
one node. Add OfferReal call for now until spec potentially gets
adjusted.

Also Add some Node information logging for FindContent and Offer
to be able to better debug failures and interoperability.
2022-12-16 17:47:52 +01:00
Kim De Mey e234a73b7e
Remove libpcre3-dev and librocksdb-dev deps from Fluffy Dockerfile (#1384) 2022-12-16 14:53:31 +01:00
Kim De Mey 3f74f084c3
Add LocalContent call to Portal JSON-RPC API (#1383) 2022-12-16 11:00:10 +01:00
Kim De Mey 727b2445b7
Fix Portal history/state JSON-RPC API and add missing calls (#1380) 2022-12-16 08:49:18 +01:00
jangko eb701fd3d7
fix addKnownToPeer in wire protocol handler 2022-12-16 07:55:38 +07:00
jangko 35e4607f87
add more test cases to t8n tool 2022-12-16 07:54:38 +07:00
Kim De Mey 095854107c
Fix Portal discv5 JSON-RPC API where it deviates from spec (#1378) 2022-12-15 16:24:23 +01:00
Kim De Mey 706effea2a
Disable canonical verification for blockheaders post-merge (#1377) 2022-12-15 14:13:26 +01:00
jangko e07898f949
add more test cases to txparse tool 2022-12-15 13:30:37 +07:00
jangko 3a7d3311e5
add more test cases to evmstate tool 2022-12-15 10:57:10 +07:00
jangko 9256c741a0
fix evmstate,t8n,txparse build instructions 2022-12-14 21:53:41 +07:00
jangko 2b5195c526
add geth compatibility flag to emvstate tool and transaction tracer 2022-12-14 21:52:43 +07:00
Kim De Mey abafc3e91f
Bump nim-eth for rlpx disconnect rlp decode fix (#1370) 2022-12-14 14:00:04 +01:00
Etan Kissling bab49dcd7e update for `excessDataGas` type chaneg 2022-12-14 11:04:13 +02:00
Etan Kissling 22338b7870 bump `nim-eth` for `eip4844` support
The `BlockHeader` structure in `nim-eth` was updated with support for
EIP-4844 (danksharding). To enable the `nim-eth` bump, the ingress of
`BlockHeader` structures has been hardened to reject headers that have
the new `excessDataGas` field until proper EIP4844 support exists.
https://github.com/status-im/nim-eth/pull/570
2022-12-14 11:04:13 +02:00
Kim De Mey c962bafd5a
Fix Portal Hive fails by correcting Portal history JSON RPC API (#1365)
* Fix Portal Hive fails by correcting Portal history JSON RPC API

- Field naming in discv5_nodeInfo
- Call naming of portal_historyStore
- Other: some proc to func adjustements
2022-12-13 19:22:36 +01:00
Jordan Hrycaj 52517d598f
Fix typo (#1364)
details:
  Accessing wrong result (out of two) leads to an exception
2022-12-13 11:13:13 +00:00
Jordan Hrycaj cc2c888a63
Snap sync swap in other pivots (#1363)
* Provide index to reconstruct missing storage slots

why;
  Pivots will be changed anymore once they are officially archived. The
  account of the archived pivots are ready to be swapped into the active
  pivot. This leaves open how to treat storage slots not fetched yet.

  Solution: when mothballing, an `account->storage-root` index is
  compiled that can be used when swapping in accounts.

* Implement swap-in from earlier pivots

details;
  When most accounts are covered by the current and previous pivot
  sessions, swapping inthe accounts and storage slots  (i.e. registering
  account ranges done) from earlier pivots takes place if there is a
  common sub-trie.

* Throttle pivot change when healing state has bean reached

why:
  There is a hope to complete the current pivot, so pivot update can be
  throttled. This is achieved by setting another minimum block number
  distance for the pivot headers. This feature is still experimental
2022-12-12 22:00:24 +00:00
jangko dcd1225724
implement txparse tool and add make target for txparse 2022-12-10 20:38:47 +07:00
jangko 1cd4a3fd54
tools: add missing copyright header 2022-12-10 19:53:24 +07:00
jangko 13e5b1a76b
fix persistHeaderToDb, allow POS block header to be canonical 2022-12-10 09:02:30 +07:00
KonradStaniec 1cc6fc5cff
bump nimbus-eth2 and web3 (#1360)
- nimbus-eth2 exposes new callbacks in LightClient
- web3 exposes createAccessList call
2022-12-09 18:00:45 +01:00
Kim De Mey c5ecba83cd
Refactor and clean-up of history network and eth rpc (#1357)
- Switch from using Option to Opt which allows for smoother
usage with already existing Result types
- With all moved to Opt, make more use of valueOr to avoid
too many if else clause indentation and unstead have a more
clear error path at each step
- Remove dead code, char limits, style guide, etc.
- Replace getEncodedKeyForContent with ContentKey.init
and use ContentKey.init for each type
2022-12-09 17:59:36 +01:00
Jordan Hrycaj 179b4adac3
Snap sync tweaks n fixes (#1359)
* Miscellaneous tweaks & fixes

details:
+ Catch `TransportError` exception in `legacy.nim` module
+ Fix self-calling wrapper `hexaryEnvelopeTouchedBy()`

* Update documentation, logging etc.

* Changed `checkNode` batch list `seq[Blob]` => `seq[NodeSpecs]`

why:
  The `NodeSpecs` type as used here is a tuple `(partial-path,node-key)`.

  When `checkNode` partial paths are collected, also the node key is
  available so it should be registered and not repeatedly recovered from
  the database.

* Add optional begin/end trace statement in snap scheduler

why:
  Allows to trace invoked entity and scheduler state variables
2022-12-09 13:43:55 +00:00
jangko 7bab58731b
simplify test_blockchain_json a bit 2022-12-09 15:26:53 +07:00
jangko 93725bdc02
remove clique unused code 2022-12-09 11:26:14 +07:00
jangko b16b35caea
cleanup unused txpool unused code 2022-12-09 11:26:14 +07:00
jangko 718f4991cf
eliminate github action warning by upgrading to v3
also change deprecated `set-output` to $GITHUB_OUTPUT
2022-12-08 16:21:43 +07:00
jangko d2381b6536
eliminate simulators ci warning 2022-12-08 13:21:15 +07:00
jangko 4287a31961
fix t8n compiler switch when evmc_enabled 2022-12-08 13:20:23 +07:00
jangko a57471197f
update t8n tool testdata 2022-12-08 13:20:23 +07:00
jangko 490b2f8023
fix t8n tool tx and env parser 2022-12-08 13:20:23 +07:00
jangko ed518c760f
fix t8n does not support BLOCKHASH opcode 2022-12-08 13:20:22 +07:00