Commit Graph

590 Commits

Author SHA1 Message Date
tersec 5579c5234f
automated consensus spec URL updating to v1.5.0-alpha.9 (#6725) 2024-11-24 13:16:09 +01:00
Agnish Ghosh a157760643
Init Fulu fork (#6677)
* fulu

* fix ci

* fix ncli

* attempt to fix ci

* fix wss_sim build

* fix vc api

* updated tests

* resolve merge conflicts

* fix tests

* fix check

* attempt to fix minimal test in CI

* addressed reviews

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
2024-11-13 10:29:14 +07:00
Eugene Kabanov 18409a69e1
Light forward sync mechanism (#6515)
* Initial commit.

* Add hybrid syncing.

* Compilation fixes.

* Cast custom event for our purposes.

* Instantiate AsyncEventQueue properly.

* Fix mistype.

* Further research on optimistic updates.

* Fixing circular deps.

* Add backfilling.

* Add block download feature.

* Add block store.

* Update backfill information before storing block.

* Use custom block verifier for backfilling sync.

* Skip signature verification in backfilling.

* Add one more generic reload to storeBackfillBlock().

* Add block verification debugging statements.

* Add more debugging

* Do not use database for backfilling, part 1.

* Fix for stash.

* Stash fixes part 2.

* Prepare for testing.

* Fix assertion.

* Fix post-restart syncing process.

* Update backfill loading log statement.
Use proper backfill slot callback for sync manager.

* Add handling of Duplicates.

* Fix store duration and block backfilled log statements.

* Add proper syncing state log statement.

* Add snappy compression to beaconchain_file.
Format syncing speed properly.

* Add blobs verification.

* Add `slot` number to file structure for easy navigation over stream of compressed objects.

* Change database filename.

* Fix structure size.

* Add more consistency properties.

* Fix checkRepair() issues.

* Preparation to state rebuild process.

* Add plain & compressed size.

* Debugging snappy encode process.

* Add one more debugging line.

* Dump blocks.

* One more filedump.

* Fix chunk corruption code.

* Fix detection issue.

* Some fixes in state rebuilding process.

* Add more clearance steps.

* Move updateHead() back to block_processor.

* Fix compilation issues.

* Make code more async friendly.

* Fix async issues.
Add more information when proposer verification failed.

* Fix 8192 slots issue.

* Fix Future double completion issue.

* Pass updateFlags to some of the core procedures.

* Fix tests.

* Improve initial sync handling mechanism.

* Fix checkStateTransition() performance improvements.

* Add some performance tuning and meters.

* Light client performance tuning.

* Remove debugging statement.

* Use single file descriptor for blockchain file.

* Attempt to fix LC.

* Fix timeleft calculation when untrusted sync backfilling started right after LC block received.

* Workaround for `chronicles` + `results` `error` issue.
Remove some compilation warnings.
Fix `CatchableError` leaks on Windows.

* Address review comments.

* Address review comments part 2.

* Address review comments part 1.

* Rebase and fix the issues.

* Address review comments part 3.

* Add tests and fix some issues in auto-repair mechanism.

* Add tests to all_tests.

* Rename binary test file to pass restrictions.

* Add `bin` extension to excluded list.
Recover binary test data.

* Rename fixture file to .bin again.

* Update AllTests.

* Address review comments part 4.

* Address review comments part 5 and fix tests.

* Address review comments part 6.

* Eliminate foldl and combine from blobs processing.
Add some tests to ensure that checkResponse() also checks for correct order.

* Fix forgotten place.

* Post rebase fixes.

* Add unique slots tests.

* Optimize updateHead() code.

* Add forgotten changes.

* Address review comments on state as argument.
2024-10-30 05:38:53 +00:00
Agnish Ghosh 250a80eb0c
add gossip validation for dc, and data column quarantine strategy (#6581)
* add gossip validation for dc

* review 1

* rm callback

* review 2

* added custody columns as a global entity

* alpha 8

* few typosA
2024-10-22 05:19:34 +00:00
tersec 3868365ac0
automated consensus spec URL updating to v1.5.0-alpha.8 (#6617) 2024-10-09 08:37:35 +02:00
tersec b8a424991d
use EF consensus spec v1.5.0-alpha.8 test vectors (#6616) 2024-10-09 04:33:17 +00:00
tersec 17d66118e8
automated consensus spec URL updating to v1.5.0-alpha.7 (#6601) 2024-10-06 14:03:56 +00:00
tersec a441695c51
plumb through Electra attestation callbacks (#6586) 2024-09-27 13:30:26 +02:00
tersec 28b20936e9
add Electra attester slashing gossip support (#6582) 2024-09-26 08:11:58 +02:00
tersec d4e441e694
add Electra attester slashing pool (#6579) 2024-09-24 11:01:40 +02:00
tersec ff9d32d30d
some v1.5.0-alpha.6 consensus spec URL updates (#6572) 2024-09-21 06:53:34 +02:00
tersec 256f6b8b89
automated consensus spec URL updating to v1.5.0-alpha.6 (#6571) 2024-09-21 03:22:41 +02:00
Eugene Kabanov e3fcd8b031
REST: getBlockRewards() and getSyncCommitteeRewards() implementation (#6556)
* Initial commit.

* Use temporary state instead of clearance.

* Attempt to fix `finalized`.

* Fix `genesis` response.

* Pre-calculate genesis block rewards response.

* Add implementation for sync committee rewards.

* Add total active balance calculation.

* Add genesis special case.

* Fix negative reward values.

* Address review comments.

* Fix isGenesis implementation and add REST test rules for both calls.
2024-09-18 23:34:32 +00:00
tersec 16c21e1c1e
rm unused code/type conversoins; re-enable Electra block gossip verification (#6545) 2024-09-12 11:11:08 +02:00
tersec 21aeeaf561
automated consensus spec URL updating to v1.5.0-alpha.5 (#6504) 2024-08-21 14:25:19 +02:00
tersec f258cba816
some v1.5.0-alpha.4 consensus spec URL updates (#6485) 2024-08-10 07:09:37 +02:00
tersec 904318cf83
automated consensus spec URL updating to v1.5.0-alpha.4 (#6483) 2024-08-09 18:24:49 +02:00
Etan Kissling 3375875e05
remove option to select Capella fork choice algo (#6478)
* remove option to select Capella fork choice algo

With Deneb having run stable for quite a while now, it's time to remove
the option to select the prior fork choice algo from Capella.

* also remove usage from test
2024-08-07 14:48:05 +00:00
tersec c0fc0f41dd
calculate next slot's withdrawals properly even across epoch boundary (#6470) 2024-08-06 13:33:16 +02:00
Eugene Kabanov b345c4fa72
Fix: Avoid downloading genesis multiple times, for holesky testnet. (#6452) 2024-07-26 14:54:03 +00:00
Etan Kissling 9f21182646
construct info object for `BlobSidecar` earlier (#6445)
To avoid "forked" types creeping into `BlobSidecar`, move the reduction
to `BlobSidecarInfoObject` to the sole caller. The info object is fork
agnostic, so does not need "forked" if `BlobSidecar` ever updates.
2024-07-23 20:22:29 +00:00
Eugene Kabanov f751fde417
Use Nimbus specific agent string instead of nim-presto defaults. (#6424)
* Use Nimbus specific agent string instead of nim-presto defaults.
Use response(code) overload.

* Fix REST API test.
2024-07-15 02:27:23 +00:00
Eugene Kabanov 61610fd243
BN: Disable genesis sync via long-range-sync argument. (#6361)
* Initial commit.

* Update options.md.

* Add pre-database initialization weak subjectivity period check.

* Add proper log message.
2024-06-20 18:57:08 +00:00
Kim De Mey 7a8c1d818a
Bump nim-eth for Opt changes and make Opt related adjustments (#6369) 2024-06-18 23:46:51 +02:00
tersec 88fa8b17ea
automated consensus spec URL updating to v1.5.0-alpha.3 (#6364) 2024-06-16 02:59:25 +02:00
Etan Kissling 9fc870777c
remove Bellatrix EL syncing support from light client (#6352)
Bellatrix light client data does not contain the EL block hash, so we
had to follow blocks gossip to learn the EL `block_hash` of such blocks.
Now that Bellatrix is obsolete, we can simplify EL syncing logic under
light client scenarios. Bellatrix light client data can still be used
to advance the light client sync itself, but will no longer result in
`engine_forkchoiceUpdated` calls until the sync reaches Capella. This
also frees up some memory as we no longer have to retain blocks.
2024-06-14 01:23:17 +00:00
Eugene Kabanov 27664291c4
VC/NIMBUS-BN validators protection. (#6329)
* Beacon node side implementation.

* Validator client side implementation.

* Address review comments and fix the test.

* Only 400 errors could be IndexedErrorMessage, 500 errors are always ErrorMessage.

* Remove VC shutdown functionality.

* Remove magic constants.

* Make arguments more visible and disable default values.

* Address review comments.
2024-06-11 19:38:16 +00:00
Etan Kissling cccd81a47b
add ANSI art placeholder for Electra 🦒 (#6338)
Trigger ANSI art on upgrade to Electra

Art created by http://beatscribe.com (beatscribe#1008 on Discord)
2024-06-10 16:45:40 +00:00
tersec 69f13eaa51
add electra gossip transition 2024-05-15 13:30:49 +00:00
tersec 0ca5211f6a
enable nbn electra block processing (#6288) 2024-05-14 20:29:03 +02:00
tersec c1b9e82502
electra attestation gossip plumbing (#6287) 2024-05-14 19:01:26 +03:00
tersec 9c669cf57a
some debugRaiseAssert to debugComment; unblock some spec tests (#6284) 2024-05-14 12:19:24 +03:00
tersec 3f972a2ca5
add some support for electra aggregated attestations (#6283) 2024-05-14 06:12:35 +02:00
tersec 6b8061b5d6
automated consensus spec URL updating to v1.5.0-alpha.2 (#6279) 2024-05-09 05:03:10 +00:00
tersec abc8bbbf23
add EF consensus spec test Electra fork and transition fixtures (#6251) 2024-04-28 16:13:17 +02:00
tersec 87452374e4
add Electra SSZ object test fixture (#6225) 2024-04-22 09:00:38 +00:00
tersec d139c92df9
explicitly scope AttesterSlashing and IndexedAttestation types to phase0 (#6224) 2024-04-21 05:49:11 +00:00
tersec 603c83522e
explicitly refer to phase0.{Attestation,TrustedAttestation} rather than sans module name (#6214) 2024-04-17 20:44:29 +00:00
tersec 1bf4058e22
add Electra support to beacon API produceBlindedBlock and produceBlockV3 endpoints; add further VC publish(Blinded)Block Electra support (#6190) 2024-04-10 10:54:00 +02:00
tersec 7fa32b7f02
add Electra to ConsensusFork enum (#6169)
* add Electra to ConsensusFork enum

* fix gnosis check
2024-04-03 16:43:43 +02:00
Etan Kissling 2dbe24c740
move split view catchup to research branch (#6133)
Using a dedicated branch for researching the effectiveness of split view
scenario handling simplifies testing and avoids having partial work on
`unstable`. If we want, we can reintroduce it under a `--debug` flag at
a later time. But for now, Goerli is a rare opoprtunity to test this,
maybe just for another week or so.

- https://github.com/status-im/infra-nimbus/pull/179
2024-03-25 19:09:31 +01:00
Etan Kissling fc9bc1da3a
add branch discovery module for supporting chain stall situation (#6125)
In split view situation, the canonical chain may only be served by a
tiny amount of peers, and branches may span long durations. Minority
branches may still have a large weight from attestations and should
be discovered. To assist with that, add a branch discovery module that
assists in such a situation by specifically targeting peers with unknown
histories and downloading from them, in addition to sync manager work
which handles popular branches.
2024-03-24 08:41:47 +00:00
Etan Kissling 66a9304fea
use separate state when catching up to perform validator duties (#6131)
There are situations where all states in the `blockchain_dag` are
occupied and cannot be borrowed.

- headState: Many assumptions in the code that it cannot be advanced
- clearanceState: Resets every time a new block gets imported, including
  blocks from non-canonical branches
- epochRefState: Used even more frequently than clearanceState

This means that during the catch-up mechanic where the head state is
slowly advanced to wall clock to catch up on validator duties in the
situation where the canonical head is way behind non-canonical heads,
we cannot use any of the three existing states. In that situation,
Nimbus already consumes an increased amount of memory due to all the
`BlockRef`, fork choice states and so on, so experience is degraded.
It seems reasonable to allocate a fourth state temporarily during that
mechanic, until a new proposal could be made on the canonical chain.

Note that currently, on `unstable`, proposals _do_ happen every couple
hours because sync manager doesn't manage to discover additional heads
in a split-view scenario on Goerli. However, with the branch discovery
module, new blocks are discovered all the time, and the clearanceState
may no longer be borrowed as it is reset to different branch too often.

The extra state could also find other uses in the future, e.g., for
incremental computations as in reindexing the database, or online
collection of historical light client data.
2024-03-24 07:18:33 +01:00
Etan Kissling 6f466894ab
answer `RequestManager` queries from disk if possible (#6109)
When restarting beacon node, orphaned blocks remain in the database but
on startup, only the canonical chain as selected by fork choice loads.
When a new block is discovered that builds on top of an orphaned block,
the orphaned block is re-downloaded using sync/request manager, despite
it already being present on disk. Such queries can be answered locally
to improve discovery speed of alternate forks.
2024-03-21 18:37:31 +01:00
Etan Kissling de2d205f61
use PR-3431 style fork choice on all networks (#6110)
To start phasing out Capella fork choice logic, set default to PR 3431.
A subsequent release can remove the fallback option.
2024-03-20 19:12:33 +01:00
Etan Kissling 035ca015e6
continue validator duties if chain does not progress for a long time (#6101)
Nimbus currently stops performing validator duties if the blockchain
does not progress for `node.config.syncHorizon` slots. This means that
the chain won't recover because no new blocks are proposed. To fix that,
continue performing validator duties if no progress is registered for a
long time, and none of our peers is indicating any progress.
2024-03-20 03:23:53 +01:00
Etan Kissling 5d42859176
make `Gwei` `distinct` (#6090)
#6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be
serialized differently than before. Using a `distinct` type for `Gwei`
improves type safety and avoids such problems in the future.
2024-03-19 14:22:07 +01:00
tersec 0a6d189161
automated consensus spec URL updating to v1.4.0 (#6074) 2024-03-14 07:26:36 +01:00
Eugene Kabanov 72c844534f
Add Keymanager API graffiti endpoints. (#6054)
* Initial commit.

* Add more tests.

* Fix API mistypes.

* Fix mistypes in tests.

* Fix one more mistype.

* Fix affected tests because of error code 401.

* Add GetGraffitiResponse object.

* Add more tests.

* Fix compilation errors.

* Recover old behavior.

* Recover old behavior.

* Fix mistype.

* Test could not know default graffiti value.

* Make VC use adopted graffiti settings.

* Make BN use adopted graffiti settings.

* Update Alltests.

* Fix test.

* Revert "Fix test."

This reverts commit c735f855d3cb9c4a1c8e8af29d3f4438d068e31f.

* Workaround {.push raises.} requirement.

* Fix comment.

* Update Alltests.
2024-03-14 03:44:00 +00:00
Etan Kissling 984d5e4631
fix regression when fetching genesis data in trustedNodeSync mode (#6047)
Corrects a regression from #5998 that led to crashes in #6046.
In `trustedNodeSync` mode, the config does not contain genesis keys,
so attempting to load from them is a `Defect`.
2024-03-08 14:52:54 +01:00