Commit Graph

649 Commits

Author SHA1 Message Date
Etan Kissling 13df492133
Merge branch 'stable' into dev/etan/df-forkedblobs 2024-12-03 14:38:06 +01:00
tersec 5579c5234f
automated consensus spec URL updating to v1.5.0-alpha.9 (#6725) 2024-11-24 13:16:09 +01:00
Agnish Ghosh a157760643
Init Fulu fork (#6677)
* fulu

* fix ci

* fix ncli

* attempt to fix ci

* fix wss_sim build

* fix vc api

* updated tests

* resolve merge conflicts

* fix tests

* fix check

* attempt to fix minimal test in CI

* addressed reviews

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
2024-11-13 10:29:14 +07:00
Eugene Kabanov 18409a69e1
Light forward sync mechanism (#6515)
* Initial commit.

* Add hybrid syncing.

* Compilation fixes.

* Cast custom event for our purposes.

* Instantiate AsyncEventQueue properly.

* Fix mistype.

* Further research on optimistic updates.

* Fixing circular deps.

* Add backfilling.

* Add block download feature.

* Add block store.

* Update backfill information before storing block.

* Use custom block verifier for backfilling sync.

* Skip signature verification in backfilling.

* Add one more generic reload to storeBackfillBlock().

* Add block verification debugging statements.

* Add more debugging

* Do not use database for backfilling, part 1.

* Fix for stash.

* Stash fixes part 2.

* Prepare for testing.

* Fix assertion.

* Fix post-restart syncing process.

* Update backfill loading log statement.
Use proper backfill slot callback for sync manager.

* Add handling of Duplicates.

* Fix store duration and block backfilled log statements.

* Add proper syncing state log statement.

* Add snappy compression to beaconchain_file.
Format syncing speed properly.

* Add blobs verification.

* Add `slot` number to file structure for easy navigation over stream of compressed objects.

* Change database filename.

* Fix structure size.

* Add more consistency properties.

* Fix checkRepair() issues.

* Preparation to state rebuild process.

* Add plain & compressed size.

* Debugging snappy encode process.

* Add one more debugging line.

* Dump blocks.

* One more filedump.

* Fix chunk corruption code.

* Fix detection issue.

* Some fixes in state rebuilding process.

* Add more clearance steps.

* Move updateHead() back to block_processor.

* Fix compilation issues.

* Make code more async friendly.

* Fix async issues.
Add more information when proposer verification failed.

* Fix 8192 slots issue.

* Fix Future double completion issue.

* Pass updateFlags to some of the core procedures.

* Fix tests.

* Improve initial sync handling mechanism.

* Fix checkStateTransition() performance improvements.

* Add some performance tuning and meters.

* Light client performance tuning.

* Remove debugging statement.

* Use single file descriptor for blockchain file.

* Attempt to fix LC.

* Fix timeleft calculation when untrusted sync backfilling started right after LC block received.

* Workaround for `chronicles` + `results` `error` issue.
Remove some compilation warnings.
Fix `CatchableError` leaks on Windows.

* Address review comments.

* Address review comments part 2.

* Address review comments part 1.

* Rebase and fix the issues.

* Address review comments part 3.

* Add tests and fix some issues in auto-repair mechanism.

* Add tests to all_tests.

* Rename binary test file to pass restrictions.

* Add `bin` extension to excluded list.
Recover binary test data.

* Rename fixture file to .bin again.

* Update AllTests.

* Address review comments part 4.

* Address review comments part 5 and fix tests.

* Address review comments part 6.

* Eliminate foldl and combine from blobs processing.
Add some tests to ensure that checkResponse() also checks for correct order.

* Fix forgotten place.

* Post rebase fixes.

* Add unique slots tests.

* Optimize updateHead() code.

* Add forgotten changes.

* Address review comments on state as argument.
2024-10-30 05:38:53 +00:00
Agnish Ghosh 250a80eb0c
add gossip validation for dc, and data column quarantine strategy (#6581)
* add gossip validation for dc

* review 1

* rm callback

* review 2

* added custody columns as a global entity

* alpha 8

* few typosA
2024-10-22 05:19:34 +00:00
tersec 84eaa57f82
don't use {.noinit.} with case objects (#6662) 2024-10-21 01:22:16 +02:00
tersec c01cf6601a
attestation and aggregated attestation BN fixes for Electra (#6641)
* attestation and aggregated attestation BN fixes for Electra

* disable electra transition for this PR

* use cleaned-up verification function everywhere
2024-10-14 14:20:26 +00:00
Etan Kissling a6311884d1
Merge branch 'unstable' into dev/etan/df-forkedblobs 2024-10-10 00:55:37 +02:00
tersec 3868365ac0
automated consensus spec URL updating to v1.5.0-alpha.8 (#6617) 2024-10-09 08:37:35 +02:00
tersec 50eed81dcf
electra BN attestation aggregating (#6596) 2024-10-07 07:53:38 +00:00
tersec 17d66118e8
automated consensus spec URL updating to v1.5.0-alpha.7 (#6601) 2024-10-06 14:03:56 +00:00
Etan Kissling d976a9313e
Merge branch 'stable' into dev/etan/df-forkedblobs 2024-10-04 16:14:58 +02:00
Pedro Miranda ff0cc8e62e
Electra - Filter aggregated attestations by root/committee (#6594)
* Filter aggregated attestations by root/committee

* fixed validator api documentation link

---------

Co-authored-by: Pedro Miranda <pedro.miranda@nimbus.team>
2024-10-04 05:15:48 +00:00
Jacek Sieka 8f0d910422
web: bump to new nim-eth types (#6590) 2024-10-02 15:26:11 +02:00
tersec a441695c51
plumb through Electra attestation callbacks (#6586) 2024-09-27 13:30:26 +02:00
tersec fa8b7e380f
add Electra beacon chain database state tests (#6584) 2024-09-27 07:53:58 +02:00
Pedro Miranda daf7f899c2
Attestation API updates for Electra (#6557)
* new V2 endpoint for beacon getBlockAttestations

* nnew GET endpoint version (V2) for getPoolAttestations

* new POST endpoint version (V2) for submitPoolAttestations

* remove premature ncli tests

* review improvements

* review comments and increased test coverage

* small improvements

* documentation typos

---------

Co-authored-by: Pedro Miranda <pedro.miranda@nimbus.team>
2024-09-25 12:33:58 +00:00
tersec d4e441e694
add Electra attester slashing pool (#6579) 2024-09-24 11:01:40 +02:00
tersec ff9d32d30d
some v1.5.0-alpha.6 consensus spec URL updates (#6572) 2024-09-21 06:53:34 +02:00
tersec 256f6b8b89
automated consensus spec URL updating to v1.5.0-alpha.6 (#6571) 2024-09-21 03:22:41 +02:00
Pedro Miranda 1ac9b851b9
Add GetAggregateAttestation V2 endpoint version (#6511)
* attestation pool support and tests

* REST endpoints changes

* initial ncli and validator client support

* updated tests file

* fixed typos

* review improvements

* remove V1 endpoint

* revert v1 removal

* V2 endpoint version available to pre electra

---------

Co-authored-by: Pedro Miranda <pedro.miranda@nimbus.team>
2024-09-06 12:15:30 +00:00
Etan Kissling 18af8dcd9f
Merge branch 'stable' into dev/etan/df-forkedblobs 2024-08-30 14:54:33 +02:00
Pedro Miranda f9e44b2a3b
Improved electra attestations packing (#6498)
* test disjoint comittee bits

* test for aggregated attestations with disjoint bits

* improved attestation pool on-chain collecting and packing

* addressed format issues

* improved sorting method

* updated AllTests-mainnet file

* updated AllTests-mainnet file

* review corrections

* review corrections

* removed old phase0 artifacts

---------

Co-authored-by: Pedro Miranda <pedro.miranda@nimbus.team>
2024-08-23 13:26:35 +02:00
tersec 8c2e8f2b0f
some v1.5.0-alpha.5 consensus spec URL updates (#6509) 2024-08-23 12:43:54 +02:00
tersec e4f18cba09
stop tracking phase0 spec artifact of current/prev target epoch for attestation block packing (#6508) 2024-08-22 20:28:21 +02:00
tersec 44cc72c104
refactor engine API conversions out of EL manager (#6507) 2024-08-22 16:12:03 +02:00
tersec 21aeeaf561
automated consensus spec URL updating to v1.5.0-alpha.5 (#6504) 2024-08-21 14:25:19 +02:00
tersec 904318cf83
automated consensus spec URL updating to v1.5.0-alpha.4 (#6483) 2024-08-09 18:24:49 +02:00
Etan Kissling 3375875e05
remove option to select Capella fork choice algo (#6478)
* remove option to select Capella fork choice algo

With Deneb having run stable for quite a while now, it's time to remove
the option to select the prior fork choice algo from Capella.

* also remove usage from test
2024-08-07 14:48:05 +00:00
tersec 535819263d
clear old electra attestations from attestation pool 2024-08-07 11:14:57 +00:00
Etan Kissling d028baea2a
introduce `ForkedBlobSidecar` for EIP-7688 Electra period before PeerDAS
On `ELECTRA_FORK_EPOCH`, PeerDAS is not yet activated, hence the current
mechanism based on `BlobSidecar` is still in use. With EIP-7688, the
generalized indices of `BeaconBlockBody` get reindexed, changing the
length of the inclusion proof within the `BlobSidecar`. Because network
Req/Resp operations allow responses across fork boundaries, this creates
the need for a `ForkedBlobSidecar` in that layer, same as already done
for `ForkedSignedBeaconBock` for similar reasons.

Note: This PR is only needed if PeerDAS is adopted _after_ EIP-7688.
If PeerDAS is adopted _before_ EIP-7688, a similar PR may be needed for
forked columns. Coincidental `Forked` jank can only be fully avoided if
both features activate at the same epoch, actual changes to blobs aside.
Delaying EIP-7688 for sole purpose of epoch alignemnt is not worth it.
2024-07-25 18:54:38 +02:00
Etan Kissling 9f21182646
construct info object for `BlobSidecar` earlier (#6445)
To avoid "forked" types creeping into `BlobSidecar`, move the reduction
to `BlobSidecarInfoObject` to the sole caller. The info object is fork
agnostic, so does not need "forked" if `BlobSidecar` ever updates.
2024-07-23 20:22:29 +00:00
Etan Kissling a769550db5
rename LC gindex constants to match spec (#6444)
Use `_ELECTRA` suffix for gindex constants to match consensus-specs.
2024-07-23 20:20:15 +00:00
Etan Kissling 9924aec1e3
extend light client protocol for Electra (#6375)
* extend light client protocol for Electra

Add missing Electra support for light client protocol:

- https://github.com/ethereum/consensus-specs/pull/3811

Tested against PR consensus-spec-tests, the test runner automatically
picks up the new tests once available.

* workaround `version-2-0`: `Error: cannot instantiate: 'SomeUnsignedInt'`

* fix initialization when Electra not scheduled

* try reduce stack size in test

* put correct sync committee branch version into DB

* adjust fork schedule in light client data tests

* further reduce stack size

* split function into multiple parts

* rename variable

* regenerate test reports to cover new Electra tests

* add Nim bug reference
2024-06-26 19:02:03 +00:00
tersec 288540e794
fix UnnamedBreak deprecation build warnings (#6388) 2024-06-26 10:48:49 +02:00
Etan Kissling cd4de1357a
during proposal duties, perform sync aggregation duties if not received (#6384)
Including sync contributions into a block affects validator rewards.
When we have not received aggregate sync contributions, but have seen
individual messages, we can produce the contributions locally, improving
validator rewards when subscribing to all subnets or when having a
non-aggregating attached validator in the sync committee.
2024-06-24 09:13:15 +00:00
tersec 9b6b42c8f9
some consensus spec URL updates to v1.5.0-alpha.3 (#6382) 2024-06-22 07:28:19 +02:00
tersec 88fa8b17ea
automated consensus spec URL updating to v1.5.0-alpha.3 (#6364) 2024-06-16 02:59:25 +02:00
andri lim 238589868c
Bump nim-web3 and nim-eth (#6345)
In nim-web3 all std.Option are replaced by results.Opt. The same goes in nim-eth, with additional fields name changes and GasInt changed from int64 to uint64.
2024-06-12 08:46:39 +07:00
Etan Kissling cccd81a47b
add ANSI art placeholder for Electra 🦒 (#6338)
Trigger ANSI art on upgrade to Electra

Art created by http://beatscribe.com (beatscribe#1008 on Discord)
2024-06-10 16:45:40 +00:00
tersec c7bf6fb542
rm debugRaiseAssert; clean up several debugComments (#6308)
* rm debugRaiseAssert; clean up several debugComments

* exception linting
2024-05-23 23:51:09 +02:00
tersec b56a671122
fix most ConvFromXtoItselfNotNeeded hints and unhide remaining ones (#6307) 2024-05-22 13:56:37 +02:00
Jacek Sieka 045c4cf185
electra attestation updates (#6295)
* electra attestation updates

In Electra, we have two attestation formats: on-chain and on-network -
the former combines all committees of a slot in a single committee bit
list.

This PR makes a number of cleanups to move towards fixing this -
attestation packing however still needs to be fixed as it currently
creates attestations with a single committee only which is very
inefficient.

* more attestations in the blocks

* signing and aggregation fixes

* tool fix

* test, import
2024-05-17 15:37:41 +03:00
tersec c1b9e82502
electra attestation gossip plumbing (#6287) 2024-05-14 19:01:26 +03:00
tersec 1c3aaa7be2
add (Signed)AggregateAndProof SSZ tests (#6285) 2024-05-14 13:51:06 +02:00
tersec 6b8061b5d6
automated consensus spec URL updating to v1.5.0-alpha.2 (#6279) 2024-05-09 05:03:10 +00:00
tersec 1b30dcc165
initial electra attestation pool changes; electra block_sim (#6255) 2024-05-07 15:01:51 +00:00
tersec 6119389c3a
add EF consensus spec test Electra attestation operations fixture (#6248) 2024-04-28 00:52:14 +00:00
tersec 87452374e4
add Electra SSZ object test fixture (#6225) 2024-04-22 09:00:38 +00:00
tersec d139c92df9
explicitly scope AttesterSlashing and IndexedAttestation types to phase0 (#6224) 2024-04-21 05:49:11 +00:00