Commit Graph

3277 Commits

Author SHA1 Message Date
jangko 6814140c63
unify coinbase state clearing of gst, evmstate, and t8n
it is troublesome if we have to fix it in three places
in case of a bug, it also reduce code duplication.
2023-01-14 17:17:55 +07:00
jangko e2601e7c54
bump eth_tests 2023-01-14 15:56:08 +07:00
jangko 192646ca8a
fix test_blockchain_json lastBlockHash comparison 2023-01-14 15:56:08 +07:00
jangko eff5de3046
fix EIP-3860 intrinsic gas 2023-01-14 15:56:07 +07:00
Jordan Hrycaj 8da4002df3
Update eth/6? messages Get/PooledTransactions (#1415)
why:
  There non-existent txs must be skipped.
2023-01-13 19:55:16 +00:00
Jordan Hrycaj b1b1bb734c
update `mainNetTTD` (#1428)
why:
  Was accidentially set as hex value from eip-3675 specs (was meant
  decimal as-is)
2023-01-13 18:50:16 +00:00
jangko a232c7eb1d
reduce code duplication in multiple test runners 2023-01-13 13:04:55 +07:00
jangko 735d780424
fix EIP-3860 bugs 2023-01-13 13:04:23 +07:00
jangko cdf40b2d37
emergency fix windows ci 2023-01-13 11:18:30 +07:00
Kim De Mey 96776deed6
Update Fluffy docs on how to seed data (#1408) 2023-01-09 17:57:55 +01:00
Kim De Mey 90f91e5ebd
Reorganize eth data files parsing code (#1407) 2023-01-05 15:26:58 +01:00
Adam Spitz 4bf4aeba94
Some of Shanghai: EIP-3651, EIP-3855, EIP-3860 (#1406)
* EIP-3651: Warm COINBASE

* EIP-3855: PUSH0 instruction

* EIP-3860: Limit and meter initcode
2023-01-04 08:11:33 -05:00
Kim De Mey 63c6000bad
Change Fluffy Docker hub repo and cron schedule for Docker build (#1405) 2023-01-03 22:38:41 +01:00
Kim De Mey 1cdbc1293d
Fix git branch name in Fluffy docker nightly build (#1404)
And change build hour temporarily
2023-01-03 10:08:24 +01:00
Kim De Mey b9ce04ecfc
Add simple nightly Docker image build for Fluffy (#1403) 2023-01-02 20:50:52 +01:00
Kim De Mey 624c87d8b3
Add experimental beacon chain block proofs for Portal network (#1376)
* Add experimental beacon chain block proofs for Portal network

* Add Caveat on proving recent blocks
2023-01-02 16:46:48 +01:00
Kim De Mey 1bfbf4f02f
Clarify Fluffy's docs and config (#1402) 2023-01-02 16:37:22 +01:00
jangko 74e76e5237
remove unused 'refundGas' from evm/state_transactions 2022-12-28 01:45:56 +07:00
KonradStaniec 2bce04afc7
Add light client mode to fluffy (#1396) 2022-12-27 15:57:10 +01:00
KonradStaniec 270ce41d5c
Add light client bridge (#1386)
* Add light client bridge binary
2022-12-27 15:25:20 +01:00
Jordan Hrycaj 5134bb5e04
Extract finding of missing nodes for healing into separate module (#1398)
why:
  Duplicate implementation of same functionality
2022-12-25 17:56:57 +00:00
Jordan Hrycaj 88b315bb41
Snap sync refactor healing (#1397)
* Simplify accounts healing threshold management

why:
  Was over-engineered.

details:
  Previously, healing was based on recursive hexary trie perusal.

  Due to "cheap" envelope decomposition of a range complement for the
  hexary trie, the cost of running extra laps have become time-affordable
  again and a simple trigger mechanism for healing will do.

* Control number of dangling result nodes in `hexaryInspectTrie()`

also:
+ Returns number of visited nodes available for logging so the maximum
  number of nodes can be tuned accordingly.
+ Some code and docu update

* Update names of constants

why:
  Declutter, more systematic naming

* Re-implemented `worker_desc.merge()` for storage slots

why:
  Provided as proper queue management in `storage_queue_helper`.

details:
+ Several append modes (replaces `merge()`)
+ Added third queue to record entries currently fetched by a worker. So
  another parallel running worker can safe the complete set of storage
  slots in as checkpoint. This was previously lost.

* Refactor healing

why:
  Simplify and remove deep hexary trie perusal for finding completeness.

   Due to "cheap" envelope decomposition of a range complement for the
   hexary trie, the cost of running extra laps have become time-affordable
   again and a simple trigger mechanism for healing will do.

* Docu update

* Run a storage job only once in download loop

why:
  Download failure or rejection (i.e. missing data) lead to repeated
  fetch requests until peer disconnects, otherwise.
2022-12-24 09:54:18 +00:00
jangko e36d2f432a
fix default --key-store location if --data-dir is custom 2022-12-22 11:17:04 +07:00
jangko d98b3bb32d
cleanup numeric utils and remove unstable rangeToPadded 2022-12-21 18:41:03 +07:00
Jordan Hrycaj 0f132c1d01
Snap sync fix ticker crash (#1393)
* Fix SEGFAULT showstopper

* Update logging
2022-12-20 15:38:57 +00:00
Jordan Hrycaj bd42ebb193
Snap sync refactor accounts healing (#1392)
* Relocated mothballing (i.e. swap-in preparation) logic

details:
  Mothballing was previously tested & started after downloading
  account ranges in `range_fetch_accounts`.

  Whenever current download or healing stops because of a pivot change,
  swap-in preparation is needed (otherwise some storage slots may get
  lost when swap-in takes place.)

  Also, `execSnapSyncAction()` has been moved back to `pivot_helper`.

* Reorganised source file directories

details:
  Grouped pivot focused modules into `pivot` directory

* Renamed `checkNodes`, `sickSubTries` as `nodes.check`, `nodes.missing`

why:
  Both lists are typically used together as pair. Renaming `sickSubTries`
  reflects moving away from a healing centric view towards a swap-in
  attitude.

* Multi times coverage recording

details:
  Per pivot account ranges are accumulated into coverage range set. This
  set fill eventually contain a singe range of account hashes [0..2^256]
  which amounts to 100% capacity.

  A counter has been added that is incremented whenever max capacity is
  reached. The accumulated range is then reset to empty.

  The effect of this setting is that the coverage can be evenly duplicated.
  So 200% would not accumulate on a particular region.

* Update range length comparisons (mod 2^256)

why:
  A range interval can have sizes 1..2^256 as it cannot be empty by
  definition. The number of points in a range intervals set can have
  0..2^256 points. As the scalar range is a residue class modulo 2^256,
  the residue class 0 means length 2^256 for a range interval, but can
  be 0 or 2^256 for the number of points in a range intervals set.

* Generalised `hexaryEnvelopeDecompose()`

details:
  Compile the complement of the union of some (processed) intervals and
  express this complement as a list of envelopes of sub-tries.

  This facility is directly applicable to swap-in book-keeping.

* Re-factor `swapIn()`

why:
  Good idea but baloney implementation. The main algorithm is based on
  the generalised version of `hexaryEnvelopeDecompose()` which has been
  derived from this implementation.

* Refactor `healAccounts()` using `hexaryEnvelopeDecompose()` as main driver

why:
  Previously, the hexary trie was searched recursively for dangling nodes
  which has a poor worst case performance already when the trie  is
  reasonably populated.

  The function `hexaryEnvelopeDecompose()` is a magnitude faster because
  it does not peruse existing sub-tries in order to find missing nodes
  although result is not fully compatible with the previous function.

  So recursive search is used in a limited mode only when the decomposer
  will not deliver a useful result.

* Logging & maintenance fixes

details:
  Preparation for abandoning buddy-global healing variables `node`,
  `resumeCtx`, and `lockTriePerusal`. These variable are trie-perusal
  centric which will be run on the back burner in favour of
  `hexaryEnvelopeDecompose()` which is used for accounts healing already.
2022-12-19 21:22:09 +00:00
Kim De Mey 05ac755e84
Set disableMarchNative for Fluffy Dockerfile (#1391) 2022-12-19 13:19:50 +01:00
Jordan Hrycaj d55a72ae49
Full sync peer negotiation control (#1390)
* Additional logging for scheduler

* Fix duplicate occurrence of `bestNumber`

why:
  Happened when the `block_queue` module was separated out of
  the `worker` module. Somehow testing was insufficient or skipped,
  at all.

* Update `runPool()` mixin for scheduler

details:
  Could be simplified

* Dynamically adapt pivot header negotiation mode

details:
  After accepting one peer and some timeout, do not search for more
  peers for start syncing but rather continue in relaxed mode with a
  single peer.
2022-12-18 16:06:43 +00:00
Kim De Mey 69204fffe9
Update two of the Fluffy bootstrap nodes for testnet (#1389) 2022-12-16 21:11:34 +01:00
Kim De Mey 591a1e7062
Add content verifier tool check data availability on the network (#1388) 2022-12-16 21:11:09 +01:00
Kim De Mey 733d7e5ceb
Add new portal JSON-RPC OfferReal as Offer is doing gossip now (#1387)
The portal_*Offer call was changed in the specs to actually do
gossip. Make it no longer possible to test purely an offer with
one node. Add OfferReal call for now until spec potentially gets
adjusted.

Also Add some Node information logging for FindContent and Offer
to be able to better debug failures and interoperability.
2022-12-16 17:47:52 +01:00
Kim De Mey e234a73b7e
Remove libpcre3-dev and librocksdb-dev deps from Fluffy Dockerfile (#1384) 2022-12-16 14:53:31 +01:00
Kim De Mey 3f74f084c3
Add LocalContent call to Portal JSON-RPC API (#1383) 2022-12-16 11:00:10 +01:00
Kim De Mey 727b2445b7
Fix Portal history/state JSON-RPC API and add missing calls (#1380) 2022-12-16 08:49:18 +01:00
jangko eb701fd3d7
fix addKnownToPeer in wire protocol handler 2022-12-16 07:55:38 +07:00
jangko 35e4607f87
add more test cases to t8n tool 2022-12-16 07:54:38 +07:00
Kim De Mey 095854107c
Fix Portal discv5 JSON-RPC API where it deviates from spec (#1378) 2022-12-15 16:24:23 +01:00
Kim De Mey 706effea2a
Disable canonical verification for blockheaders post-merge (#1377) 2022-12-15 14:13:26 +01:00
jangko e07898f949
add more test cases to txparse tool 2022-12-15 13:30:37 +07:00
jangko 3a7d3311e5
add more test cases to evmstate tool 2022-12-15 10:57:10 +07:00
jangko 9256c741a0
fix evmstate,t8n,txparse build instructions 2022-12-14 21:53:41 +07:00
jangko 2b5195c526
add geth compatibility flag to emvstate tool and transaction tracer 2022-12-14 21:52:43 +07:00
Kim De Mey abafc3e91f
Bump nim-eth for rlpx disconnect rlp decode fix (#1370) 2022-12-14 14:00:04 +01:00
Etan Kissling bab49dcd7e update for `excessDataGas` type chaneg 2022-12-14 11:04:13 +02:00
Etan Kissling 22338b7870 bump `nim-eth` for `eip4844` support
The `BlockHeader` structure in `nim-eth` was updated with support for
EIP-4844 (danksharding). To enable the `nim-eth` bump, the ingress of
`BlockHeader` structures has been hardened to reject headers that have
the new `excessDataGas` field until proper EIP4844 support exists.
https://github.com/status-im/nim-eth/pull/570
2022-12-14 11:04:13 +02:00
Kim De Mey c962bafd5a
Fix Portal Hive fails by correcting Portal history JSON RPC API (#1365)
* Fix Portal Hive fails by correcting Portal history JSON RPC API

- Field naming in discv5_nodeInfo
- Call naming of portal_historyStore
- Other: some proc to func adjustements
2022-12-13 19:22:36 +01:00
Jordan Hrycaj 52517d598f
Fix typo (#1364)
details:
  Accessing wrong result (out of two) leads to an exception
2022-12-13 11:13:13 +00:00
Jordan Hrycaj cc2c888a63
Snap sync swap in other pivots (#1363)
* Provide index to reconstruct missing storage slots

why;
  Pivots will be changed anymore once they are officially archived. The
  account of the archived pivots are ready to be swapped into the active
  pivot. This leaves open how to treat storage slots not fetched yet.

  Solution: when mothballing, an `account->storage-root` index is
  compiled that can be used when swapping in accounts.

* Implement swap-in from earlier pivots

details;
  When most accounts are covered by the current and previous pivot
  sessions, swapping inthe accounts and storage slots  (i.e. registering
  account ranges done) from earlier pivots takes place if there is a
  common sub-trie.

* Throttle pivot change when healing state has bean reached

why:
  There is a hope to complete the current pivot, so pivot update can be
  throttled. This is achieved by setting another minimum block number
  distance for the pivot headers. This feature is still experimental
2022-12-12 22:00:24 +00:00
jangko dcd1225724
implement txparse tool and add make target for txparse 2022-12-10 20:38:47 +07:00
jangko 1cd4a3fd54
tools: add missing copyright header 2022-12-10 19:53:24 +07:00