Commit Graph

1570 Commits

Author SHA1 Message Date
tersec 34ba05f4d3
add EF consensus spec test Electra deposits operations fixture (#6243) 2024-04-26 07:18:44 +00:00
tersec b0f58a58b6
add EF consensus spec test Electra consolidation operations fixture (#6235) 2024-04-25 18:50:54 +00:00
tersec 8c4ddd64c0
add EF consensus spec test attester slashing, proposer slashing, voluntary exit, and execution layer withdrawal request tests (#6234) 2024-04-25 08:41:17 +00:00
tersec f53271eaaa
add rest of EF consensus spec test Electra epoch transition fixture(s) (#6232) 2024-04-24 12:28:47 +00:00
tersec 9ba6b8d8a7
add Electra epoch rewards test fixture (#6231)
* add Electra epoch rewards test fixture

* version-2-0 still has another issue
2024-04-23 15:47:39 +00:00
tersec e6e95cbab7
add Electra slot sanity tests (#6230) 2024-04-23 07:58:32 +00:00
tersec eb997f4e1f
add 10 sub-test-fixtures for electra epoch transitions; use v2.0.4 for Nim 2.x (#6229) 2024-04-23 01:15:55 +00:00
tersec 67a22b3887
add Electra BLS to Execution Change, Block Heaer, Sync Aggregate, and Withdrawals operations test fixtures (#6226) 2024-04-22 12:37:38 +00:00
tersec 87452374e4
add Electra SSZ object test fixture (#6225) 2024-04-22 09:00:38 +00:00
tersec d139c92df9
explicitly scope AttesterSlashing and IndexedAttestation types to phase0 (#6224) 2024-04-21 05:49:11 +00:00
careworry 2ec82fe3ce
chore: remove repetitive words (#6219)
Signed-off-by: careworry <worrycare@outlook.com>
2024-04-19 11:31:15 +00:00
tersec 73738c702c
use consensus spec v1.5.0-alpha.0 test vectors; use Nim v2.0.4 specifically for 2.0.x (#6221) 2024-04-19 02:55:58 +00:00
tersec 645e627b03
add new Electra beacon chain containers and presets (#6216)
* add new Electra beacon chain containers and presets

* re-enable attestations in electra state transition
2024-04-18 07:30:01 +00:00
tersec 0132f5d689
some consensus spec v1.4.0 spec URL updates (#6215) 2024-04-18 03:00:04 +02:00
tersec 603c83522e
explicitly refer to phase0.{Attestation,TrustedAttestation} rather than sans module name (#6214) 2024-04-17 20:44:29 +00:00
tersec 867995acd1
some consensus spec v1.4.0 spec URL updates (#6208) 2024-04-17 05:51:16 +02:00
tersec bd3c9af0f8
implement EIP-7002 (#6206)
* implement EIP-7002

* fix linting
2024-04-16 18:09:39 +00:00
tersec 361be082ae
add Electra toSignedBlindedBlock tests (#6182) 2024-04-08 10:34:15 +02:00
tersec 937cc62b85
block_sim runs electra (#6181) 2024-04-07 09:58:11 +02:00
tersec 0d534224b3
remove some debugRaiseAsserts and fill in Electra functionality (#6178) 2024-04-06 07:46:02 +00:00
tersec 06894dd2f6
remove some debugRaiseAsserts and fill in Electra functionality (#6177) 2024-04-05 21:30:06 +02:00
tersec 0b6f75b88b
remove some debugRaiseAsserts and fill in actual Electra functionality (#6172) 2024-04-04 05:17:31 +02:00
tersec 13a70e9d69
rm built-in Goerli beacon chain support (#6057) 2024-04-03 23:25:36 +00:00
tersec 7fa32b7f02
add Electra to ConsensusFork enum (#6169)
* add Electra to ConsensusFork enum

* fix gnosis check
2024-04-03 16:43:43 +02:00
tersec 27ec2893ff
rm unused parameters of get_state_exit_queue_info() (#6167) 2024-04-03 01:45:57 +00:00
tersec 109007dc93
avoid quadratic behavior exiting validators (#6161)
* avoid quadratic behavior exiting validators

* fix libnfuzz callers
2024-04-02 12:18:40 +00:00
tersec 4457334dd0
Add support for Pectra ExecutionPayload representations (#6158) 2024-03-29 13:37:27 +01:00
Etan Kissling 12a2f8c026
when adding duplicates to quarantine, schedule deepest missing parent (#6112)
During sync, sometimes the same block gets encountered and added to
quarantine multiple times. If its parent is already known, quarantine
incorrectly registers it as missing, leading to re-download. This can
be fixed by registering the parent's deepest missing parent recursively.

Also increase the stickiness of `missing`. We only perform 4 attempts
within ~16 seconds before giving up. Very frequently, this is not enough
and there is no progress until sync manager kicks in even on holesky.
2024-03-21 18:41:05 +00:00
Etan Kissling 5d42859176
make `Gwei` `distinct` (#6090)
#6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be
serialized differently than before. Using a `distinct` type for `Gwei`
improves type safety and avoids such problems in the future.
2024-03-19 14:22:07 +01:00
tersec 0a6d189161
automated consensus spec URL updating to v1.4.0 (#6074) 2024-03-14 07:26:36 +01:00
Eugene Kabanov 72c844534f
Add Keymanager API graffiti endpoints. (#6054)
* Initial commit.

* Add more tests.

* Fix API mistypes.

* Fix mistypes in tests.

* Fix one more mistype.

* Fix affected tests because of error code 401.

* Add GetGraffitiResponse object.

* Add more tests.

* Fix compilation errors.

* Recover old behavior.

* Recover old behavior.

* Fix mistype.

* Test could not know default graffiti value.

* Make VC use adopted graffiti settings.

* Make BN use adopted graffiti settings.

* Update Alltests.

* Fix test.

* Revert "Fix test."

This reverts commit c735f855d3cb9c4a1c8e8af29d3f4438d068e31f.

* Workaround {.push raises.} requirement.

* Fix comment.

* Update Alltests.
2024-03-14 03:44:00 +00:00
tersec fd8429da74
rm Web3Signer Bellatrix block support (#6061) 2024-03-11 22:09:31 +00:00
Etan Kissling a0bc3fff86
fix `/eth/v1/beacon/deposit_snapshot` for EIP-4881 (#6038)
Fix the `/eth/v1/beacon/deposit_snapshot` API to produce proper EIP-4881
compatible `DepositTreeSnapshot` responses. The endpoint used to expose
a Nimbus-specific database internal format.

Also fix trusted node sync to consume properly formatted EIP-4881 data
with `--with-deposit-snapshot`, and `--finalized-deposit-tree-snapshot`
beacon node launch option to use the EIP-4881 data. Further ensure that
`ncli_testnet` produces EIP-4881 formatted data for interoperability.
2024-03-08 14:22:03 +01:00
tersec 816361ed8c
Bloom filter acceleration for deposit processing (#5982) 2024-03-07 23:38:24 +00:00
Etan Kissling 50a43f397f
rename `DepositTreeSnapshot` -> `DepositContractSnapshot` (#6036)
EIP-4881 was never correctly implemented, the `DepositTreeSnapshot`
structure has nothing to do with its actual definition. Reflect that
by renaming the type to a Nimbus-specific `DepositContractSnapshot`,
so that an actual EIP-4881 implementation can use the correct names.

- https://eips.ethereum.org/EIPS/eip-4881#specification

Notably, `DepositTreeSnapshot` contains a compressed sequence in
`finalized`, only containing the minimally required intermediate roots.

That also explains the incorrect REST response reported in #5508.

The non-canonical representation was introduced in #4303 and is also
persisted in the database. We'll have to maintain it for a while.
2024-03-07 18:42:52 +01:00
tersec e6253c0793
don't gate test execution on not defined(i386) (#6035) 2024-03-06 17:24:05 +00:00
tersec 39fafc1248
unconditionally run test_keystore_management (#6034) 2024-03-06 15:55:31 +01:00
tersec 9e4a70e0c4
unconditionally run test_keystore (#6033) 2024-03-06 13:14:52 +01:00
tersec 9fda162e99
unconditionally run test_blockchain_dag (#6032) 2024-03-06 06:10:34 +00:00
Etan Kissling 2c924bcc8c
add test runner for LC data collection tests (#6021)
Introduce a test runner for upcoming EF test suites related to canonical
light client data collection.

- https://github.com/ethereum/consensus-specs/pull/3553
2024-03-05 15:41:09 +01:00
tersec 8222d62500
rm unused phase0 and altair remote signing test code (#6027) 2024-03-05 02:38:34 +01:00
tersec 2a13c09615
add proposer reward accounting to block transitions (#6022)
* add proposer reward accounting to block transitions

* Update beacon_chain/spec/state_transition_block.nim

Co-authored-by: Etan Kissling <etan@status.im>

---------

Co-authored-by: Etan Kissling <etan@status.im>
2024-03-04 17:00:46 +00:00
tersec 9d56f3920b
fix some styleCheck hints (#6016) 2024-03-03 02:04:45 +01:00
Etan Kissling f5e9d3ffe4
add trivial `{.push raises: [].}` to `research` and `test` folders (#5988)
Annotate the `research` and `test` files for which no further changes
are needed to successfully compile them, to not interfere with periodic
tasks such as spec reference bumps.
2024-02-29 13:24:08 +00:00
Etan Kissling 23fa8d0c70
handle `scanf` exceptions in `test_fixture_ssz_generic_types` (#5993)
`scanf` apparently has both a `bool` return as well as raising random
exceptions depending on what functions get called by the `macro`.
To make this explicit, catch the `ValueError` from the generated
`parseInt` call, to separate `scanf` behaviour from the actual SSZ
test logic. In the end, it mostly doesn't matter as there are some
`doAssert wasMatched` on the next line (not everywhere though).
But it still makes the `scanf` internals explicit, so is clearer.
2024-02-29 10:47:00 +00:00
Etan Kissling f4ff438aab
explicitly mention `{.raises.}` in `tests` folder (#5990)
Add `{.raises.}` annotations to `tests` files where needed to enable
`{.push raises: [].}`. Avoids interfering with periodic changes such as
spec version bumps, and avoids special casing folders when editing.
The effort to maintain `{.raises.}` is trivial after the initial round.
2024-02-29 10:28:32 +00:00
Etan Kissling 794ec9ed19
use `closureScope` with `{.raises.}` in `test_peer_pool` (#5992)
`test_peer_pool` is a bit different from the other tests as it uses
`closureScope` which doesn't play well with `{.push raises: [].}`.
Define an overload instead that allows passing `{.raises.}` to the
`template`. This allows using `unittest2`'s exception handler without
having to refactor the test.
2024-02-29 00:57:19 +00:00
Etan Kissling 078f4ac347
assert exceptions that statically cannot occur in test code (#5989)
There are some spurious exceptions that statically cannot occur.
Add `raiseAssert` for these.
2024-02-28 20:15:49 +00:00
Etan Kissling 57f4886d39
improve cleanup in error situation in slashing protection tests (#5991)
If removing a file files during `slashing_protection/test_fixtures.nim`,
still continue to try to remove the other ephemeral test files.
2024-02-28 18:28:04 +01:00
Etan Kissling 80d532abd7
avoid forgetting to quit in `os_ops` tools if log fails (#5986)
`stderr.write` may fail, e.g., if no tty is connected, which may happen
in some CI configurations. Discard such failures and continue quitting
instead of raising the error.
2024-02-28 16:04:34 +00:00
Etan Kissling 4a74ef9cba
add `{.raises.}` annotations to `test_fixture_operations` (#5984)
Some more progress in making exception flow explicit.
2024-02-28 11:54:29 +00:00
tersec f65c1121d2
add Electra overloads for spec functions; add Electra block processing (#5963) 2024-02-26 02:38:21 +00:00
tersec a4f4a35845
Revert "initial Electra support skeleton" (#5955)
* Revert "initial Electra support skeleton (#5946)"

This reverts commit d09bf3b587.

* Update test_signing_node.nim
2024-02-25 19:42:44 +00:00
tersec d09bf3b587
initial Electra support skeleton (#5946) 2024-02-24 13:44:15 +00:00
tersec bf6b5a316f
add push raises to tests; rm unused test code (#5948) 2024-02-24 04:41:58 +01:00
Etan Kissling 702f4c6a45
add `{.push raises: [].}` to more files (#5943)
Extend `{.push raises: [].}` to all files with spec references that are
periodically updated.

- https://github.com/status-im/nimbus-eth2/pull/5942
2024-02-22 11:03:09 +01:00
tersec c73d7c6f6f
automated consensus spec URL updating to v1.4.0-beta.7 (#5942) 2024-02-21 19:44:48 +00:00
Jacek Sieka 1ef7d237cc
Shared validator pubkey (#5883)
This PR allows sharing the pubkey data between validators by using a
thread-local cache for pubkey data, netting about a 400mb mem usage
reduction on holesky due to us keeping 3 permanent + several ephemeral
state copies in memory at all times and each state copy holding a full
validator.

The PR also introduces a hash cache for the key which gives ~14% speedup
for a full state `hash_tree_root` - the key makes up for a large part of
the `Validator` htr time.

Finally, the time it takes to copy a state goes down as well from ~80m
ms to ~60, for reasons similar to htr.

We use a `ptr` even if a `ref` could in theory have been used - there is
not much practical benefit to a `ref` (given it's mutable) while a `ptr`
is cheaper and easier to copy (when copying temporary states).

We could go further and cache a cooked pubkey but it turns out this is
quite intrusive - in all the relevant places, we're already using a
cooked key from the immutable validator data so there are no immediate
performance gains of doing so while managing the compressed -> cooked
key mapping would become more difficult - something for a future PR
perhaps.

Co-authored-by: Etan Kissling <etan@status.im>
2024-02-21 20:06:19 +01:00
Etan Kissling 4fc1550d0f
add `{.push raises: [].}` to recently modified files (#5908)
Status Nim style mandates `{.push raises: []}.` at start of modules.
Ensure that's the case so that exceptions are properly tracked.

- https://status-im.github.io/nim-style-guide/errors.exceptions.html
- https://github.com/status-im/nim-eth/pull/614#discussion_r1220906149
2024-02-18 01:16:49 +00:00
Etan Kissling bf882887bd
`noCancel` in `test_gossip_validation` (#5907)
To avoid spurious warnings, mark futures that are not cancelled by test
as such when waiting for them. Variable rename is for max line length.
2024-02-18 00:40:15 +00:00
Etan Kissling 71444a371a
annotate `test_fixture_fork_choice` with `{.raises.}` (#5903)
The fork choice test fixture is one of the more complex ones. Annotate
its functions with `{.raises.}` to make exception flow explicit.
2024-02-17 22:52:11 +01:00
tersec e410fe0052
https://github.com/ethereum/consensus-specs/pull/3600 (#5896) 2024-02-17 09:02:50 +00:00
tersec 384d40fc0b
remove requirement for REST JSON serialization tests to hash signedbeaconblock (#5887) 2024-02-13 17:30:03 +00:00
Jacek Sieka afdfe302f3
state loading optimizations (#5881)
* compute post-merge randao mix without loading state
* avoid copying state on shuffling computation and compute epochref
* speed up state copy for block production
2024-02-12 15:58:55 +01:00
tersec a4680cb7fa
refactor addHeadBlock() to research/ and tests/ helper (#5874)
* refactor addHeadBlock() to research/ and tests/ helper

* rm now-dead code
2024-02-09 23:46:51 +00:00
Etan Kissling 7c53841cd8
Revert "Revert "fix checkpoint block potentially not getting backfilled into DB (#5863)" (#5871)" (#5875)
This reverts commit 1575478b72.
2024-02-09 20:44:54 +01:00
tersec 1575478b72
Revert "fix checkpoint block potentially not getting backfilled into DB (#5863)" (#5871)
This reverts commit 65e6f892de.
2024-02-09 12:49:07 +00:00
Etan Kissling 65e6f892de
fix checkpoint block potentially not getting backfilled into DB (#5863)
When using checkpoint sync, only checkpoint state is available, block is
not downloaded and backfilled later.

`dag.backfill` tracks latest filled `slot`, and latest `parent_root` for
which no block has been synced yet.

In checkpoint sync, this assumption is broken, because there, the start
`dag.backfill.slot` is set based on checkpoint state slot, and the block
is also not available.

However, sync manager in backward mode also requests `dag.backfill.slot`
and `block_clearance` then backfills the checkpoint block once it is
synced. But, there is no guarantee that a peer ever sends us that block.
They could send us all parent blocks and solely omit the checkpoint
block itself. In that situation, we would accept the parent blocks and
advance `dag.backfill`, and subsequently never request the checkpoint
block again, resulting in gap inside blocks DB that is never filled.

To mitigate that, the assumption is restored that `dag.backfill.slot`
is the latest filled `slot`, and `dag.backfill.parent_root` is the next
block that needs to be synced. By setting `slot` to `tail.slot + 1` and
`parent_root` to `tail.root`, we put a fake summary into `dag.backfill`
so that `block_clearance` only proceeds once checkpoint block exists.
2024-02-09 11:20:36 +01:00
Etan Kissling 4266e16835
allow `getBlockIdAtSlot` to answer queries from available states (#5869)
After checkpoint sync, historical block IDs cannot yet be queried.
However, they are needed to compute dependent roots of `ShufflingRef`.
To allow lookup, enable `getBlockIdAtSlot` to answer from compatible
states in memory; as long as they descend from the finalized checkpoint
and the requested slot is sufficiently recent, `block_roots` contains
everything to recover `BlockSlotId` up to `SLOTS_PER_HISTORICAL_ROOT`.
This is similar to how `attester_dependent_root` etc. are computed.

This accelerates the first couple minutes of checkpoint sync on Mainnet,
especially the time until finality advances past the synced checkpoint.
2024-02-09 11:13:00 +01:00
tersec 642774e596
unrevert rest of https://github.com/status-im/nimbus-eth2/pull/5765 (#5867)
* unrevert rest of https://github.com/status-im/nimbus-eth2/pull/5765

* rm stray e2store docs changes

* reduce diff

* fix indent

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2024-02-09 09:35:41 +01:00
Etan Kissling e398078abc
`...ExecutionPayloadHash` --> `...ExecutionBlockHash` (#5864)
Finish the rename started in #4809 to have a consistent naming.
`ExecutionPayloadHash` suggests hash over payload instead of block.
`BlockHash` is also the canonical name in engine API.
2024-02-08 01:24:55 +01:00
tersec 87052eba4e
implement getBlindedBlock REST API (#5829) 2024-01-31 03:18:55 +00:00
xiaolou86 af6affb5b3
Fix typos (#5835)
* Fix typos

* Update copyright year
2024-01-29 09:37:38 +00:00
tersec 225ef5e69a
partially revert https://github.com/status-im/nimbus-eth2/pull/5765 (#5833) 2024-01-28 23:45:52 +01:00
Jacek Sieka 6328c77778
raises for gossip (#5808)
* raises for gossip

* fix light client
2024-01-22 17:34:54 +01:00
tersec 7fd8beb418
rm unused code in {ncli,research,tests}/ (#5809) 2024-01-21 07:55:03 +01:00
tersec d669eef97b
rm unused code; fix a Deprecated warning; proc to func (#5807) 2024-01-20 21:36:01 +00:00
tersec 3b025adda1
rm some unnecessary stew/shims/net imports to reduce deprecation warnings (#5805) 2024-01-20 12:32:38 +00:00
tersec 6c53dc1e11
automated consensus spec URL updating to v1.4.0-beta.6 (#5804) 2024-01-20 11:19:47 +00:00
Jacek Sieka 3ff9b69bf1
simplify eth2_network error handling (#5765)
This PR gets rid of a bunch of redundant exception handling through
async raises guarantees.

More can be removed once libp2p gets properly annotated.
2024-01-19 21:05:52 +00:00
tersec d99347afaa
add all remaining KZG test cases (#5793) 2024-01-19 10:59:07 +00:00
Etan Kissling 006fa59ec2
use consensus spec v1.4.0-beta.6 test vectors (#5783) 2024-01-18 18:14:44 +00:00
Etan Kissling 62ee92a094
bolster `BlobSidecar` syncing on incomplete responses (#5766)
Avoid marking blocks invalid when corresponding `blobSidecarsByRange`
returns an incomplete / incorrect response while syncing. The block
itself may still be valid in that scenario.
2024-01-18 15:45:10 +01:00
Jacek Sieka d5785677a8
split out eth1chain into its own module (#5768)
reduces import junk in some places - more could be done here
2024-01-17 15:26:16 +01:00
tersec 52ed0d6aa1
rm kzg test fixture code duplication and add test vector coverage checks (#5742) 2024-01-15 07:14:38 +01:00
tersec c815e71af0
rest of EF KZG test harnesses (#5734) 2024-01-13 01:31:29 +00:00
tersec aef3879652
some EF KZG test harnesses (#5733) 2024-01-12 22:29:43 +00:00
Etan Kissling 95e9740506
use `forks` sugar in `loadForkedState` testing function (#5720)
Adjust `loadForkedState` to use `withState` sugar to reduce code
duplication.
2024-01-09 17:27:32 +00:00
Jacek Sieka 62cbdeefc5
verify `genesis_time` more strictly (fixes #1667) (#5694)
Bogus values lead to crashes down the line when timers overflow
2024-01-06 15:26:56 +01:00
Etan Kissling 508f3b6368
add missing `std/` prefix to more imports (#5696)
Bumping some `std` imports to explicitly include the prefix.
Also add explicit `./` prefix for local directory imports.
2024-01-06 07:18:28 +01:00
Jacek Sieka 3f525acb87
chronos: bump (#5684)
This PR causes a few new warnings to appear - these are harmless but
will need addressing separately as they span several libraries.

* new asyncraises syntax
* asyncraises support in several modules
* `sink` usage reduces spurious copying
* `?` compatiblity for `async` + `results`
* remove `-d:chronosStrictException` (obsolete)
2024-01-05 10:08:38 +01:00
Etan Kissling 030226148d
rename `exit_pool` > `validator_change_pool` (#5679)
The `ExitPool` was renamed to `ValidatorChangePool` with Capella, but
the files were still using the previous name. Rename for consistency.
2023-12-23 06:55:47 +01:00
zah 29b29e1945
Use the new strict Json flavors mechanism for RestJson (#5674)
This requires all object types to be explicitly white-listed for
default serialization. The PR makes the minimal changes, although
a number of similar mechanisms in eth2_rest_serialization can now
be removed.
2023-12-19 17:44:27 +02:00
tersec 9c6ba7d142
consensus spec v1.4.0-beta.5 URL updates (#5672) 2023-12-16 03:27:06 +01:00
andri lim 15147cccb1
Bump nim-web3 to dcabb8f29ee55afedefdf93cd3e102bb1daee354 (#5664)
* bump nim-web3 to dcabb8f29ee55afedefdf93cd3e102bb1daee354

also bump json-rpc to a8731e91bc336d930ac66f985d3b88ed7cf2a7d7
2023-12-12 22:15:00 +07:00
tersec 9efb2958ec
automated consensus spec URL updating to v1.4.0-beta.5 (#5647) 2023-12-05 03:34:45 +01:00
tersec 0e5c44b529
use v5.3.0 slashing interchange test vectors (#5640) 2023-12-04 12:55:56 +01:00
Eugene Kabanov e2e4912645
REST API produceBlockV3 implementation (#5474)
Co-authored-by: Etan Kissling <etan@status.im>
Co-authored-by: Jacek Sieka <jacek@status.im>
2023-11-29 00:30:14 +01:00
tersec 6dee4d59fe
switch engine authentication from `Option` to `Opt` (#5624)
* switch engine authentication from Option to Opt

* el_conf updates

* LC and el_conf tests
2023-11-27 14:48:29 +00:00
Etan Kissling 8cea8af620
fix startup after BN exited between head and finalized blocks updates (#5617)
When the BN exits after writing new `head` to database, but before
completing the `updateFinalizedBlocks` call, the database is slightly
inconsistent due to the partial write. We currently fail to start up
after that. Fix that by catching up on partial `updateFinalizedBlocks`
tasks on start up, and add a test for this edge case.
2023-11-23 00:44:20 +01:00