Commit Graph

1859 Commits

Author SHA1 Message Date
Eugene Kabanov 18409a69e1
Light forward sync mechanism ()
* Initial commit.

* Add hybrid syncing.

* Compilation fixes.

* Cast custom event for our purposes.

* Instantiate AsyncEventQueue properly.

* Fix mistype.

* Further research on optimistic updates.

* Fixing circular deps.

* Add backfilling.

* Add block download feature.

* Add block store.

* Update backfill information before storing block.

* Use custom block verifier for backfilling sync.

* Skip signature verification in backfilling.

* Add one more generic reload to storeBackfillBlock().

* Add block verification debugging statements.

* Add more debugging

* Do not use database for backfilling, part 1.

* Fix for stash.

* Stash fixes part 2.

* Prepare for testing.

* Fix assertion.

* Fix post-restart syncing process.

* Update backfill loading log statement.
Use proper backfill slot callback for sync manager.

* Add handling of Duplicates.

* Fix store duration and block backfilled log statements.

* Add proper syncing state log statement.

* Add snappy compression to beaconchain_file.
Format syncing speed properly.

* Add blobs verification.

* Add `slot` number to file structure for easy navigation over stream of compressed objects.

* Change database filename.

* Fix structure size.

* Add more consistency properties.

* Fix checkRepair() issues.

* Preparation to state rebuild process.

* Add plain & compressed size.

* Debugging snappy encode process.

* Add one more debugging line.

* Dump blocks.

* One more filedump.

* Fix chunk corruption code.

* Fix detection issue.

* Some fixes in state rebuilding process.

* Add more clearance steps.

* Move updateHead() back to block_processor.

* Fix compilation issues.

* Make code more async friendly.

* Fix async issues.
Add more information when proposer verification failed.

* Fix 8192 slots issue.

* Fix Future double completion issue.

* Pass updateFlags to some of the core procedures.

* Fix tests.

* Improve initial sync handling mechanism.

* Fix checkStateTransition() performance improvements.

* Add some performance tuning and meters.

* Light client performance tuning.

* Remove debugging statement.

* Use single file descriptor for blockchain file.

* Attempt to fix LC.

* Fix timeleft calculation when untrusted sync backfilling started right after LC block received.

* Workaround for `chronicles` + `results` `error` issue.
Remove some compilation warnings.
Fix `CatchableError` leaks on Windows.

* Address review comments.

* Address review comments part 2.

* Address review comments part 1.

* Rebase and fix the issues.

* Address review comments part 3.

* Add tests and fix some issues in auto-repair mechanism.

* Add tests to all_tests.

* Rename binary test file to pass restrictions.

* Add `bin` extension to excluded list.
Recover binary test data.

* Rename fixture file to .bin again.

* Update AllTests.

* Address review comments part 4.

* Address review comments part 5 and fix tests.

* Address review comments part 6.

* Eliminate foldl and combine from blobs processing.
Add some tests to ensure that checkResponse() also checks for correct order.

* Fix forgotten place.

* Post rebase fixes.

* Add unique slots tests.

* Optimize updateHead() code.

* Add forgotten changes.

* Address review comments on state as argument.
2024-10-30 05:38:53 +00:00
tersec f85521230f
stop checking for irrelevant unsigned wraparound in Electra deposit codepath () 2024-10-28 17:18:01 +01:00
tersec 4565c029e1
make Eth2Digest isZero 8x faster ()
* make Eth2Digest isZero 8x faster

* use staticFor and copyMem

* branchless with or
2024-10-28 05:21:18 +00:00
tersec 58a34e00a1
fix inconsistent aggregation bits len in Electra () 2024-10-25 16:04:35 +02:00
andri lim 5d940b4c1e
Bump nim-eth to 66297c5c0a8c22ec3f16c899e902d79aa00df575 () 2024-10-22 14:15:33 +02:00
Agnish Ghosh 250a80eb0c
add gossip validation for dc, and data column quarantine strategy ()
* add gossip validation for dc

* review 1

* rm callback

* review 2

* added custody columns as a global entity

* alpha 8

* few typosA
2024-10-22 05:19:34 +00:00
Etan Kissling 6f6de8d9dc
Implement latest EIP-7685 `requestsHash` method ()
It's now this weird one-off hashing method so that EL can partially
forget in some modules what the data schema is.
2024-10-21 17:09:26 +02:00
tersec 2d3614ff69
update to devnet-4-compatible engine API () 2024-10-17 16:53:37 +02:00
Etan Kissling bd94c20325
fix migration of `blob_gas_used`/`excess_blob_gas` from Deneb to Electra ()
`blob_gas_used` and `excess_blob_gas` were reset on fork transition
from Deneb to Electra, which is not according to spec. Fix it.
2024-10-17 00:15:34 +00:00
Eugene Kabanov fcf72a6e8e
VC: Electra fixes. ()
* Initial commit.

* Add aggregated attestation processing.

* Add missing presets file.

* Fix compilation error.

* Fix post-rebase compilation error.

* Satisfy push raises requirement.

* Fix sync committee duties retrieval process.

* Fix forks configuration management.

* Fix deposits to use new fork configuration scheme.

* Fix /eth/v2/validator/aggregate_attestation implementation.

* Fix RANDAO preparation loop to handle blocks at epoch boundary properly.

* Simplification of RANDAO fix.

* Fix typo.

* Address review comments and fix tests.

* Fix incorrect status codes in REST test.

* Rework attestation and aggregated attestations processing code.

* Address review comments.

* Fill committee_index in RegisteredAttestation construction code.

* Address review comments part 2.

* Address review comments part 3.

* use Deneb fork epoch

* Add transition from Deneb to Electra into CI finalization test.

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
2024-10-16 17:20:39 +00:00
Etan Kissling c4be5f677a
bump nim-eth to `171531fbf1a308c1bf4d2f6b88c6ee6924657437` ()
- rm Goerli bootnodes
- update Sepolia and Holesky bootnodes
- Fix Authorization fields name to v, r, s
- Add helper for recovering authority
2024-10-15 21:47:17 +00:00
tersec acad28d05b
avoid combining multiple aggregates from same committee () 2024-10-15 06:02:36 +02:00
Miran 585deb8f90
use types from `eth/common/eth_types` ()
* bump nim-eth

* use types from `eth/common/eth_types`
2024-10-14 14:55:56 +00:00
tersec c01cf6601a
attestation and aggregated attestation BN fixes for Electra ()
* attestation and aggregated attestation BN fixes for Electra

* disable electra transition for this PR

* use cleaned-up verification function everywhere
2024-10-14 14:20:26 +00:00
tersec bc51ba3077
styleCheck tweaks () 2024-10-12 00:51:28 +02:00
tersec 5608a803d7
update beacon_validators to be more Nim 2.2-compatible () 2024-10-11 15:39:45 +02:00
tersec 181f5f984d
use Deneb genesis for local testnet; add Electra forking support to local tesnet ()
* use Deneb genesis for local testnet; add Electra forking support to local tesnet

* deneb -> electra in ncli_testnet help
2024-10-10 15:48:05 +00:00
tersec 9ba31fb5e6
adjust isomorphicCast to be Nim 2.2-compatible () 2024-10-10 12:23:23 +00:00
tersec 3868365ac0
automated consensus spec URL updating to v1.5.0-alpha.8 () 2024-10-09 08:37:35 +02:00
tersec b8a424991d
use EF consensus spec v1.5.0-alpha.8 test vectors () 2024-10-09 04:33:17 +00:00
Jacek Sieka 7990cc2d1c
Ordered trie for trie root computations ()
Gives a 100x speed boost for this operation which happens during
optimistic sync when computing the block hash.

Co-authored-by: Etan Kissling <etan@nimbus.team>
2024-10-08 22:10:50 +02:00
Miran 3f37caa0da
fix deprecated warnings () 2024-10-08 01:54:02 +00:00
Eugene Kabanov 5d11c5229b
Add Web3signer electra support ()
* Add Electra support to nimbus_signing_node.
Add tests.
Annotate nimbus_signing_node with asyncraises.

* Update AllTests.
2024-10-07 23:31:16 +00:00
tersec b370112fe0
some manual updates to EF consensus-specs v1.5.0-alpha.7 URLs () 2024-10-07 12:30:09 +02:00
tersec 50eed81dcf
electra BN attestation aggregating () 2024-10-07 07:53:38 +00:00
tersec 17d66118e8
automated consensus spec URL updating to v1.5.0-alpha.7 () 2024-10-06 14:03:56 +00:00
tersec 9c9f3707a1
use EF consensus spec v1.5.0-alpha.7 test vectors () 2024-10-06 13:15:56 +02:00
Jacek Sieka 8f0d910422
web: bump to new nim-eth types () 2024-10-02 15:26:11 +02:00
Jacek Sieka b20f1ba8b3
Update nim-eth types ()
Minimal changes needed for compatiblity with
https://github.com/status-im/nim-eth/pull/733 which aligns core types
with execution spec.
2024-09-29 13:32:55 +02:00
tersec 2e64fdfa67
centralize ConsensusFork <=> string handling in spec/forks () 2024-09-28 09:23:45 +02:00
Pedro Miranda 4beb890385
new pool attester slashings endpoint version (V2) for electra ()
* new pool attester slashings endpoint version (V2) for electra

* formatting

---------

Co-authored-by: Pedro Miranda <pedro.miranda@nimbus.team>
2024-09-28 00:00:26 +00:00
Pedro Miranda 744cc00984
replace default word usage ()
Co-authored-by: Pedro Miranda <pedro.miranda@nimbus.team>
2024-09-26 19:49:03 +00:00
tersec 28b20936e9
add Electra attester slashing gossip support () 2024-09-26 08:11:58 +02:00
Pedro Miranda daf7f899c2
Attestation API updates for Electra ()
* new V2 endpoint for beacon getBlockAttestations

* nnew GET endpoint version (V2) for getPoolAttestations

* new POST endpoint version (V2) for submitPoolAttestations

* remove premature ncli tests

* review improvements

* review comments and increased test coverage

* small improvements

* documentation typos

---------

Co-authored-by: Pedro Miranda <pedro.miranda@nimbus.team>
2024-09-25 12:33:58 +00:00
tersec f2d6166099
remove block fork guessing/inference from REST JSON decoding ()
* remove block fork guessing/inference from REST JSON decoding

* use template to avoid repetitive per-fork code

* consolidate RestPublishedSignedBeaconBlock and RestPublishedSignedBlockContents parsing fork handling
2024-09-25 02:58:53 +00:00
Agnish Ghosh 31b5c3e30e
add: inclusion proof and dc sidecar verification ()
* add: inclusion proof and dc sidecar verification

* review 1
2024-09-24 14:46:05 +02:00
tersec d4e441e694
add Electra attester slashing pool () 2024-09-24 11:01:40 +02:00
Eugene Kabanov 77e835153c
Bump nimcrypto to fix nim22 issues. ()
* Bump nimcrypto to fix nim22 issues.

* Fix compilation issue.
2024-09-24 02:55:16 +00:00
Agnish Ghosh 85d7109065
add: `get_data_column_sidecars` ()
* add: get_data_column_sidecars

* remove redundant type conversions

* remove redundant for loop

* review 2

* review 1
2024-09-22 23:35:57 +02:00
tersec ff9d32d30d
some v1.5.0-alpha.6 consensus spec URL updates () 2024-09-21 06:53:34 +02:00
tersec 256f6b8b89
automated consensus spec URL updating to v1.5.0-alpha.6 () 2024-09-21 03:22:41 +02:00
tersec 193a982d24
use EF consensus spec v1.5.0-alpha.6 test vectors ()
* use EF consensus spec v1.5.0-alpha.6 test vectors

* LC changes are good as is, no need for the debug comment

* Fix indentation

* Use `execution_requests` for requests root

* fill in spec reference URLs for ExecutionPayload(Header)

* copy `execution_requests` when blidning blocks for Electra and add test

* fix `test_engine_api_conversions`

---------

Co-authored-by: Etan Kissling <etan@nimbus.team>
2024-09-20 21:32:16 +02:00
Etan Kissling 9df3d075de
fix blinded block conversion for Deneb and Electra ()
`blob_gas_used` and `excess_blob_gas` were not copied on blinding
signed beacon block.
2024-09-20 16:02:24 +02:00
Eugene Kabanov e3fcd8b031
REST: getBlockRewards() and getSyncCommitteeRewards() implementation ()
* Initial commit.

* Use temporary state instead of clearance.

* Attempt to fix `finalized`.

* Fix `genesis` response.

* Pre-calculate genesis block rewards response.

* Add implementation for sync committee rewards.

* Add total active balance calculation.

* Add genesis special case.

* Fix negative reward values.

* Address review comments.

* Fix isGenesis implementation and add REST test rules for both calls.
2024-09-18 23:34:32 +00:00
Agnish Ghosh 3f0d58492e
add: ssz consensus objs for peerdas () 2024-09-17 22:57:48 +00:00
Agnish Ghosh 1feeff4f92
add: recover matrix () 2024-09-17 10:21:55 +00:00
Agnish Ghosh b3c8c71776
add: compute matrix ()
* add: compute matrix

* rm unsused imports

* rm unused imports 2

* rm bin

* rm unused line

* improve doc

* rid space on proc sig

* review 1

* added seeded PRNG

* randomize blob count as well, with smaller seed

* review 2
2024-09-16 11:19:34 +00:00
tersec 72b5c2bd6e
add some electra support to REST JSON decoding; rm unused code () 2024-09-15 13:33:13 +00:00
Pedro Miranda 7b6caebf6e
Add publishAggregateAndProofsV2 ()
* added new endpoint version(v2) for validator aggregate and proof

* review improvements

* introduced nim template for proof executiion

---------

Co-authored-by: Pedro Miranda <pedro.miranda@nimbus.team>
2024-09-13 21:11:09 +00:00
tersec ed2422112c
resolve several outstanding items for Electra support; add debug-foo to light client yamux () 2024-09-13 10:07:06 +02:00