Commit Graph

262 Commits

Author SHA1 Message Date
Etan Kissling cf9dc4a79b
Merge branch 'dev/etan/df-forkedblobs' into feat_eip-7688 2024-12-04 14:18:18 +01:00
Etan Kissling 13df492133
Merge branch 'stable' into dev/etan/df-forkedblobs 2024-12-03 14:38:06 +01:00
Eugene Kabanov a461bb102f
Fix block proposer verification failure for light forward syncing algorithm. (#6726) 2024-11-24 23:51:21 +01:00
tersec 5579c5234f
automated consensus spec URL updating to v1.5.0-alpha.9 (#6725) 2024-11-24 13:16:09 +01:00
Eugene Kabanov bd04dcc3dc
Fix syncing problem introduced by light forward syncing PR. (#6714)
* Fix blobs check response algorithm and add more logging information.

* Add MAX_BLOBS_PER_BLOCK check.
Add tests.

* Adopt AllTests.

* Address review comments.

* One more missed lenu64.
2024-11-19 06:35:09 +01:00
Eugene Kabanov 18409a69e1
Light forward sync mechanism (#6515)
* Initial commit.

* Add hybrid syncing.

* Compilation fixes.

* Cast custom event for our purposes.

* Instantiate AsyncEventQueue properly.

* Fix mistype.

* Further research on optimistic updates.

* Fixing circular deps.

* Add backfilling.

* Add block download feature.

* Add block store.

* Update backfill information before storing block.

* Use custom block verifier for backfilling sync.

* Skip signature verification in backfilling.

* Add one more generic reload to storeBackfillBlock().

* Add block verification debugging statements.

* Add more debugging

* Do not use database for backfilling, part 1.

* Fix for stash.

* Stash fixes part 2.

* Prepare for testing.

* Fix assertion.

* Fix post-restart syncing process.

* Update backfill loading log statement.
Use proper backfill slot callback for sync manager.

* Add handling of Duplicates.

* Fix store duration and block backfilled log statements.

* Add proper syncing state log statement.

* Add snappy compression to beaconchain_file.
Format syncing speed properly.

* Add blobs verification.

* Add `slot` number to file structure for easy navigation over stream of compressed objects.

* Change database filename.

* Fix structure size.

* Add more consistency properties.

* Fix checkRepair() issues.

* Preparation to state rebuild process.

* Add plain & compressed size.

* Debugging snappy encode process.

* Add one more debugging line.

* Dump blocks.

* One more filedump.

* Fix chunk corruption code.

* Fix detection issue.

* Some fixes in state rebuilding process.

* Add more clearance steps.

* Move updateHead() back to block_processor.

* Fix compilation issues.

* Make code more async friendly.

* Fix async issues.
Add more information when proposer verification failed.

* Fix 8192 slots issue.

* Fix Future double completion issue.

* Pass updateFlags to some of the core procedures.

* Fix tests.

* Improve initial sync handling mechanism.

* Fix checkStateTransition() performance improvements.

* Add some performance tuning and meters.

* Light client performance tuning.

* Remove debugging statement.

* Use single file descriptor for blockchain file.

* Attempt to fix LC.

* Fix timeleft calculation when untrusted sync backfilling started right after LC block received.

* Workaround for `chronicles` + `results` `error` issue.
Remove some compilation warnings.
Fix `CatchableError` leaks on Windows.

* Address review comments.

* Address review comments part 2.

* Address review comments part 1.

* Rebase and fix the issues.

* Address review comments part 3.

* Add tests and fix some issues in auto-repair mechanism.

* Add tests to all_tests.

* Rename binary test file to pass restrictions.

* Add `bin` extension to excluded list.
Recover binary test data.

* Rename fixture file to .bin again.

* Update AllTests.

* Address review comments part 4.

* Address review comments part 5 and fix tests.

* Address review comments part 6.

* Eliminate foldl and combine from blobs processing.
Add some tests to ensure that checkResponse() also checks for correct order.

* Fix forgotten place.

* Post rebase fixes.

* Add unique slots tests.

* Optimize updateHead() code.

* Add forgotten changes.

* Address review comments on state as argument.
2024-10-30 05:38:53 +00:00
Etan Kissling 91f47a382d
Merge branch 'dev/etan/df-forkedblobs' into feat_eip-7688 2024-10-11 15:52:10 +02:00
Etan Kissling a6311884d1
Merge branch 'unstable' into dev/etan/df-forkedblobs 2024-10-10 00:55:37 +02:00
tersec 3868365ac0
automated consensus spec URL updating to v1.5.0-alpha.8 (#6617) 2024-10-09 08:37:35 +02:00
tersec 17d66118e8
automated consensus spec URL updating to v1.5.0-alpha.7 (#6601) 2024-10-06 14:03:56 +00:00
Etan Kissling d976a9313e
Merge branch 'stable' into dev/etan/df-forkedblobs 2024-10-04 16:14:58 +02:00
Etan Kissling af75741f2a
Merge branch 'unstable' into feat_eip-7688 2024-09-21 14:44:53 +02:00
tersec 256f6b8b89
automated consensus spec URL updating to v1.5.0-alpha.6 (#6571) 2024-09-21 03:22:41 +02:00
Etan Kissling 7ac4756e73
Merge branch 'dev/etan/df-forkedblobs' into feat_eip-7688 2024-09-01 15:48:31 +02:00
Etan Kissling 18af8dcd9f
Merge branch 'stable' into dev/etan/df-forkedblobs 2024-08-30 14:54:33 +02:00
Etan Kissling bd09e4d864
inhibit LC sync while DAG is synced (#6505)
Normally, running LC and DAG sync at same time is fine, but on tiny
devnet where some peer may not support the LC data, we can end up in
situation where peer gets disconnected when DAG is in sync, because
DAG sync never uses any req/resp on local devnet (perfect nw conditions)
so the LC sync over minutes removes the peer as sync is stuck.

We don't need to actively sync LC from network if DAG is already synced,
preventing this specific low peer devnet issue (there are others still).
LC is still locally updated when DAG finalized checkpoint advances.
2024-08-22 06:13:47 +00:00
tersec 21aeeaf561
automated consensus spec URL updating to v1.5.0-alpha.5 (#6504) 2024-08-21 14:25:19 +02:00
tersec f258cba816
some v1.5.0-alpha.4 consensus spec URL updates (#6485) 2024-08-10 07:09:37 +02:00
tersec 904318cf83
automated consensus spec URL updating to v1.5.0-alpha.4 (#6483) 2024-08-09 18:24:49 +02:00
Etan Kissling 2abfee3488
Merge branch 'dev/etan/df-forkedblobs' into feat_eip-7688 2024-07-25 21:01:33 +02:00
Etan Kissling d028baea2a
introduce `ForkedBlobSidecar` for EIP-7688 Electra period before PeerDAS
On `ELECTRA_FORK_EPOCH`, PeerDAS is not yet activated, hence the current
mechanism based on `BlobSidecar` is still in use. With EIP-7688, the
generalized indices of `BeaconBlockBody` get reindexed, changing the
length of the inclusion proof within the `BlobSidecar`. Because network
Req/Resp operations allow responses across fork boundaries, this creates
the need for a `ForkedBlobSidecar` in that layer, same as already done
for `ForkedSignedBeaconBock` for similar reasons.

Note: This PR is only needed if PeerDAS is adopted _after_ EIP-7688.
If PeerDAS is adopted _before_ EIP-7688, a similar PR may be needed for
forked columns. Coincidental `Forked` jank can only be fully avoided if
both features activate at the same epoch, actual changes to blobs aside.
Delaying EIP-7688 for sole purpose of epoch alignemnt is not worth it.
2024-07-25 18:54:38 +02:00
Etan Kissling ae0a1488b5
Fix blob syncing for Electra (#6438)
* Fix blob syncing for Electra

`BlobSidecar` requests on libp2p have a context prefix based on:

> The `<context-bytes>` field is calculated as context =
>     `compute_fork_digest(fork_version, genesis_validators_root)`

We currently only process blobs if that indicates Deneb, meaning that
on Electra we incorrectly report `InvalidContextBytes` and refuse to
process the blob response data.

Fix this, and also ensure that the code no longer needs maintenance
with every fork unrelated to blobs.

* fix
2024-07-23 03:10:41 +00:00
Etan Kissling 754e9c3b10
Merge branch 'dev/etan/lc-electra' into feat_eip-7688 2024-06-25 14:57:39 +02:00
tersec 9b6b42c8f9
some consensus spec URL updates to v1.5.0-alpha.3 (#6382) 2024-06-22 07:28:19 +02:00
Eugene Kabanov 61610fd243
BN: Disable genesis sync via long-range-sync argument. (#6361)
* Initial commit.

* Update options.md.

* Add pre-database initialization weak subjectivity period check.

* Add proper log message.
2024-06-20 18:57:08 +00:00
tersec 88fa8b17ea
automated consensus spec URL updating to v1.5.0-alpha.3 (#6364) 2024-06-16 02:59:25 +02:00
Etan Kissling 2915833ce8
Merge branch 'dev/etan/cc-fieldname' into feat_eip-7688 2024-06-13 12:43:49 +02:00
tersec 77d926525f
stop using deprecated stew/results (#6343)
* stop using deprecated stew/results

* exception tracking linting
2024-06-11 02:34:56 +00:00
Etan Kissling 031036f0bc
Merge branch 'feat/eip-7495' into feat_eip-7688 2024-05-24 17:17:40 +02:00
Etan Kissling 44d8a8bde0
add EIP-7688 support
- https://eips.ethereum.org/EIPS/eip-7688
2024-05-22 16:27:22 +02:00
tersec b56a671122
fix most ConvFromXtoItselfNotNeeded hints and unhide remaining ones (#6307) 2024-05-22 13:56:37 +02:00
tersec 6b8061b5d6
automated consensus spec URL updating to v1.5.0-alpha.2 (#6279) 2024-05-09 05:03:10 +00:00
tersec 41f8400f97
fix Electra light client objects; use version-2-0 for Nim again (#6222) 2024-04-19 19:20:45 +00:00
tersec 0132f5d689
some consensus spec v1.4.0 spec URL updates (#6215) 2024-04-18 03:00:04 +02:00
tersec e51c5ec783
add Electra blob support to block/blob quarantines, block processor, and request manager (#6201) 2024-04-11 09:31:39 +00:00
Etan Kissling 96e27d9d35
avoid modifying blobless collection during iteration in edge case (#6168)
The fallback when blobless quarantine contains a block with all blobs
modifies collection while iterating, potentially asserting if reachable.
Using a second loop to process this situation resolves that.
2024-04-03 23:02:24 +02:00
Etan Kissling 2dbe24c740
move split view catchup to research branch (#6133)
Using a dedicated branch for researching the effectiveness of split view
scenario handling simplifies testing and avoids having partial work on
`unstable`. If we want, we can reintroduce it under a `--debug` flag at
a later time. But for now, Goerli is a rare opoprtunity to test this,
maybe just for another week or so.

- https://github.com/status-im/infra-nimbus/pull/179
2024-03-25 19:09:31 +01:00
Etan Kissling fc9bc1da3a
add branch discovery module for supporting chain stall situation (#6125)
In split view situation, the canonical chain may only be served by a
tiny amount of peers, and branches may span long durations. Minority
branches may still have a large weight from attestations and should
be discovered. To assist with that, add a branch discovery module that
assists in such a situation by specifically targeting peers with unknown
histories and downloading from them, in addition to sync manager work
which handles popular branches.
2024-03-24 08:41:47 +00:00
Etan Kissling 2d9586a5a8
enqueue missing parent block if stored in local DB (#6122)
When checking for `MissingParent`, it may be that the parent block was
already discovered as part of a prior run. In that case, it can be
loaded from storage and processed without having to rediscover the
entire branch from the network. This is similar to #6112 but for blocks
that are discovered via gossip / sync mgr instead of via request mgr.
2024-03-22 14:35:46 +01:00
Etan Kissling 9d5643240b
only request blobs if a sync response actually provided blocks (#6121)
During sync, we can skip the `blobSidecarsByRange` request when there
are no blocks with `kzg_commitments` in the blocks data. Avoids running
into throttling from peers during long periods of non-finality.
2024-03-22 03:27:02 +01:00
Etan Kissling 17ee40b39b
make blobs use less quota when other nodes sync from us (#6120)
Each individual blob currently uses as much quota from the network limit
as an entire block does, 128 items per second shared across all peers.
Blobs are 128 KB each instead of up to several MB and are simpler to
encode. There can be multiple per block (6 currently), so allow 2000
blobs per second across all peers. That decreases the cost per block
from `3125 + 3125 * blobs.len` quota (= `[3125, 21875]`) to a lower
`3125 + 200 * blobs.len` quota (= `[3125, 4325]`), accounting for the
slight increase in data transfer and encoding time.
2024-03-22 02:36:08 +01:00
Etan Kissling 2a45bb3c7c
add error information when sync requests fail (#6119)
During sync it may be interesting to know why requests are failing.
Extend debug logging accordingly.
2024-03-22 00:26:50 +00:00
Etan Kissling 6f466894ab
answer `RequestManager` queries from disk if possible (#6109)
When restarting beacon node, orphaned blocks remain in the database but
on startup, only the canonical chain as selected by fork choice loads.
When a new block is discovered that builds on top of an orphaned block,
the orphaned block is re-downloaded using sync/request manager, despite
it already being present on disk. Such queries can be answered locally
to improve discovery speed of alternate forks.
2024-03-21 18:37:31 +01:00
tersec 0a6d189161
automated consensus spec URL updating to v1.4.0 (#6074) 2024-03-14 07:26:36 +01:00
tersec e865817d44
add some push raises, ElectraBeaconStateNoImmutableValidators, and ELECTRA_FORK_VERSION/EPOCH (#5949) 2024-02-24 05:08:22 +00:00
tersec c73d7c6f6f
automated consensus spec URL updating to v1.4.0-beta.7 (#5942) 2024-02-21 19:44:48 +00:00
tersec 642774e596
unrevert rest of https://github.com/status-im/nimbus-eth2/pull/5765 (#5867)
* unrevert rest of https://github.com/status-im/nimbus-eth2/pull/5765

* rm stray e2store docs changes

* reduce diff

* fix indent

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2024-02-09 09:35:41 +01:00
tersec 225ef5e69a
partially revert https://github.com/status-im/nimbus-eth2/pull/5765 (#5833) 2024-01-28 23:45:52 +01:00
Jacek Sieka 6328c77778
raises for gossip (#5808)
* raises for gossip

* fix light client
2024-01-22 17:34:54 +01:00
tersec 6c53dc1e11
automated consensus spec URL updating to v1.4.0-beta.6 (#5804) 2024-01-20 11:19:47 +00:00