1584 Commits

Author SHA1 Message Date
tersec
39fafc1248
unconditionally run test_keystore_management (#6034) 2024-03-06 15:55:31 +01:00
tersec
9e4a70e0c4
unconditionally run test_keystore (#6033) 2024-03-06 13:14:52 +01:00
tersec
9fda162e99
unconditionally run test_blockchain_dag (#6032) 2024-03-06 06:10:34 +00:00
Etan Kissling
2c924bcc8c
add test runner for LC data collection tests (#6021)
Introduce a test runner for upcoming EF test suites related to canonical
light client data collection.

- https://github.com/ethereum/consensus-specs/pull/3553
2024-03-05 15:41:09 +01:00
tersec
8222d62500
rm unused phase0 and altair remote signing test code (#6027) 2024-03-05 02:38:34 +01:00
tersec
2a13c09615
add proposer reward accounting to block transitions (#6022)
* add proposer reward accounting to block transitions

* Update beacon_chain/spec/state_transition_block.nim

Co-authored-by: Etan Kissling <etan@status.im>

---------

Co-authored-by: Etan Kissling <etan@status.im>
2024-03-04 17:00:46 +00:00
tersec
9d56f3920b
fix some styleCheck hints (#6016) 2024-03-03 02:04:45 +01:00
Etan Kissling
f5e9d3ffe4
add trivial {.push raises: [].} to research and test folders (#5988)
Annotate the `research` and `test` files for which no further changes
are needed to successfully compile them, to not interfere with periodic
tasks such as spec reference bumps.
2024-02-29 13:24:08 +00:00
Etan Kissling
23fa8d0c70
handle scanf exceptions in test_fixture_ssz_generic_types (#5993)
`scanf` apparently has both a `bool` return as well as raising random
exceptions depending on what functions get called by the `macro`.
To make this explicit, catch the `ValueError` from the generated
`parseInt` call, to separate `scanf` behaviour from the actual SSZ
test logic. In the end, it mostly doesn't matter as there are some
`doAssert wasMatched` on the next line (not everywhere though).
But it still makes the `scanf` internals explicit, so is clearer.
2024-02-29 10:47:00 +00:00
Etan Kissling
f4ff438aab
explicitly mention {.raises.} in tests folder (#5990)
Add `{.raises.}` annotations to `tests` files where needed to enable
`{.push raises: [].}`. Avoids interfering with periodic changes such as
spec version bumps, and avoids special casing folders when editing.
The effort to maintain `{.raises.}` is trivial after the initial round.
2024-02-29 10:28:32 +00:00
Etan Kissling
794ec9ed19
use closureScope with {.raises.} in test_peer_pool (#5992)
`test_peer_pool` is a bit different from the other tests as it uses
`closureScope` which doesn't play well with `{.push raises: [].}`.
Define an overload instead that allows passing `{.raises.}` to the
`template`. This allows using `unittest2`'s exception handler without
having to refactor the test.
2024-02-29 00:57:19 +00:00
Etan Kissling
078f4ac347
assert exceptions that statically cannot occur in test code (#5989)
There are some spurious exceptions that statically cannot occur.
Add `raiseAssert` for these.
2024-02-28 20:15:49 +00:00
Etan Kissling
57f4886d39
improve cleanup in error situation in slashing protection tests (#5991)
If removing a file files during `slashing_protection/test_fixtures.nim`,
still continue to try to remove the other ephemeral test files.
2024-02-28 18:28:04 +01:00
Etan Kissling
80d532abd7
avoid forgetting to quit in os_ops tools if log fails (#5986)
`stderr.write` may fail, e.g., if no tty is connected, which may happen
in some CI configurations. Discard such failures and continue quitting
instead of raising the error.
2024-02-28 16:04:34 +00:00
Etan Kissling
4a74ef9cba
add {.raises.} annotations to test_fixture_operations (#5984)
Some more progress in making exception flow explicit.
2024-02-28 11:54:29 +00:00
tersec
f65c1121d2
add Electra overloads for spec functions; add Electra block processing (#5963) 2024-02-26 02:38:21 +00:00
tersec
a4f4a35845
Revert "initial Electra support skeleton" (#5955)
* Revert "initial Electra support skeleton (#5946)"

This reverts commit d09bf3b587bc4f0c91b8e2f58884665a0ae80e34.

* Update test_signing_node.nim
2024-02-25 19:42:44 +00:00
tersec
d09bf3b587
initial Electra support skeleton (#5946) 2024-02-24 13:44:15 +00:00
tersec
bf6b5a316f
add push raises to tests; rm unused test code (#5948) 2024-02-24 04:41:58 +01:00
Etan Kissling
702f4c6a45
add {.push raises: [].} to more files (#5943)
Extend `{.push raises: [].}` to all files with spec references that are
periodically updated.

- https://github.com/status-im/nimbus-eth2/pull/5942
2024-02-22 11:03:09 +01:00
tersec
c73d7c6f6f
automated consensus spec URL updating to v1.4.0-beta.7 (#5942) 2024-02-21 19:44:48 +00:00
Jacek Sieka
1ef7d237cc
Shared validator pubkey (#5883)
This PR allows sharing the pubkey data between validators by using a
thread-local cache for pubkey data, netting about a 400mb mem usage
reduction on holesky due to us keeping 3 permanent + several ephemeral
state copies in memory at all times and each state copy holding a full
validator.

The PR also introduces a hash cache for the key which gives ~14% speedup
for a full state `hash_tree_root` - the key makes up for a large part of
the `Validator` htr time.

Finally, the time it takes to copy a state goes down as well from ~80m
ms to ~60, for reasons similar to htr.

We use a `ptr` even if a `ref` could in theory have been used - there is
not much practical benefit to a `ref` (given it's mutable) while a `ptr`
is cheaper and easier to copy (when copying temporary states).

We could go further and cache a cooked pubkey but it turns out this is
quite intrusive - in all the relevant places, we're already using a
cooked key from the immutable validator data so there are no immediate
performance gains of doing so while managing the compressed -> cooked
key mapping would become more difficult - something for a future PR
perhaps.

Co-authored-by: Etan Kissling <etan@status.im>
2024-02-21 20:06:19 +01:00
Etan Kissling
4fc1550d0f
add {.push raises: [].} to recently modified files (#5908)
Status Nim style mandates `{.push raises: []}.` at start of modules.
Ensure that's the case so that exceptions are properly tracked.

- https://status-im.github.io/nim-style-guide/errors.exceptions.html
- https://github.com/status-im/nim-eth/pull/614#discussion_r1220906149
2024-02-18 01:16:49 +00:00
Etan Kissling
bf882887bd
noCancel in test_gossip_validation (#5907)
To avoid spurious warnings, mark futures that are not cancelled by test
as such when waiting for them. Variable rename is for max line length.
2024-02-18 00:40:15 +00:00
Etan Kissling
71444a371a
annotate test_fixture_fork_choice with {.raises.} (#5903)
The fork choice test fixture is one of the more complex ones. Annotate
its functions with `{.raises.}` to make exception flow explicit.
2024-02-17 22:52:11 +01:00
tersec
e410fe0052
https://github.com/ethereum/consensus-specs/pull/3600 (#5896) 2024-02-17 09:02:50 +00:00
tersec
384d40fc0b
remove requirement for REST JSON serialization tests to hash signedbeaconblock (#5887) 2024-02-13 17:30:03 +00:00
Jacek Sieka
afdfe302f3
state loading optimizations (#5881)
* compute post-merge randao mix without loading state
* avoid copying state on shuffling computation and compute epochref
* speed up state copy for block production
2024-02-12 15:58:55 +01:00
tersec
a4680cb7fa
refactor addHeadBlock() to research/ and tests/ helper (#5874)
* refactor addHeadBlock() to research/ and tests/ helper

* rm now-dead code
2024-02-09 23:46:51 +00:00
Etan Kissling
7c53841cd8
Revert "Revert "fix checkpoint block potentially not getting backfilled into DB (#5863)" (#5871)" (#5875)
This reverts commit 1575478b721b6f62d6466a226b91188bdf97e00b.
2024-02-09 20:44:54 +01:00
tersec
1575478b72
Revert "fix checkpoint block potentially not getting backfilled into DB (#5863)" (#5871)
This reverts commit 65e6f892deb5d9ff4399a0840a90788726024008.
2024-02-09 12:49:07 +00:00
Etan Kissling
65e6f892de
fix checkpoint block potentially not getting backfilled into DB (#5863)
When using checkpoint sync, only checkpoint state is available, block is
not downloaded and backfilled later.

`dag.backfill` tracks latest filled `slot`, and latest `parent_root` for
which no block has been synced yet.

In checkpoint sync, this assumption is broken, because there, the start
`dag.backfill.slot` is set based on checkpoint state slot, and the block
is also not available.

However, sync manager in backward mode also requests `dag.backfill.slot`
and `block_clearance` then backfills the checkpoint block once it is
synced. But, there is no guarantee that a peer ever sends us that block.
They could send us all parent blocks and solely omit the checkpoint
block itself. In that situation, we would accept the parent blocks and
advance `dag.backfill`, and subsequently never request the checkpoint
block again, resulting in gap inside blocks DB that is never filled.

To mitigate that, the assumption is restored that `dag.backfill.slot`
is the latest filled `slot`, and `dag.backfill.parent_root` is the next
block that needs to be synced. By setting `slot` to `tail.slot + 1` and
`parent_root` to `tail.root`, we put a fake summary into `dag.backfill`
so that `block_clearance` only proceeds once checkpoint block exists.
2024-02-09 11:20:36 +01:00
Etan Kissling
4266e16835
allow getBlockIdAtSlot to answer queries from available states (#5869)
After checkpoint sync, historical block IDs cannot yet be queried.
However, they are needed to compute dependent roots of `ShufflingRef`.
To allow lookup, enable `getBlockIdAtSlot` to answer from compatible
states in memory; as long as they descend from the finalized checkpoint
and the requested slot is sufficiently recent, `block_roots` contains
everything to recover `BlockSlotId` up to `SLOTS_PER_HISTORICAL_ROOT`.
This is similar to how `attester_dependent_root` etc. are computed.

This accelerates the first couple minutes of checkpoint sync on Mainnet,
especially the time until finality advances past the synced checkpoint.
2024-02-09 11:13:00 +01:00
tersec
642774e596
unrevert rest of https://github.com/status-im/nimbus-eth2/pull/5765 (#5867)
* unrevert rest of https://github.com/status-im/nimbus-eth2/pull/5765

* rm stray e2store docs changes

* reduce diff

* fix indent

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2024-02-09 09:35:41 +01:00
Etan Kissling
e398078abc
...ExecutionPayloadHash --> ...ExecutionBlockHash (#5864)
Finish the rename started in #4809 to have a consistent naming.
`ExecutionPayloadHash` suggests hash over payload instead of block.
`BlockHash` is also the canonical name in engine API.
2024-02-08 01:24:55 +01:00
tersec
87052eba4e
implement getBlindedBlock REST API (#5829) 2024-01-31 03:18:55 +00:00
xiaolou86
af6affb5b3
Fix typos (#5835)
* Fix typos

* Update copyright year
2024-01-29 09:37:38 +00:00
tersec
225ef5e69a
partially revert https://github.com/status-im/nimbus-eth2/pull/5765 (#5833) 2024-01-28 23:45:52 +01:00
Jacek Sieka
6328c77778
raises for gossip (#5808)
* raises for gossip

* fix light client
2024-01-22 17:34:54 +01:00
tersec
7fd8beb418
rm unused code in {ncli,research,tests}/ (#5809) 2024-01-21 07:55:03 +01:00
tersec
d669eef97b
rm unused code; fix a Deprecated warning; proc to func (#5807) 2024-01-20 21:36:01 +00:00
tersec
3b025adda1
rm some unnecessary stew/shims/net imports to reduce deprecation warnings (#5805) 2024-01-20 12:32:38 +00:00
tersec
6c53dc1e11
automated consensus spec URL updating to v1.4.0-beta.6 (#5804) 2024-01-20 11:19:47 +00:00
Jacek Sieka
3ff9b69bf1
simplify eth2_network error handling (#5765)
This PR gets rid of a bunch of redundant exception handling through
async raises guarantees.

More can be removed once libp2p gets properly annotated.
2024-01-19 21:05:52 +00:00
tersec
d99347afaa
add all remaining KZG test cases (#5793) 2024-01-19 10:59:07 +00:00
Etan Kissling
006fa59ec2
use consensus spec v1.4.0-beta.6 test vectors (#5783) 2024-01-18 18:14:44 +00:00
Etan Kissling
62ee92a094
bolster BlobSidecar syncing on incomplete responses (#5766)
Avoid marking blocks invalid when corresponding `blobSidecarsByRange`
returns an incomplete / incorrect response while syncing. The block
itself may still be valid in that scenario.
2024-01-18 15:45:10 +01:00
Jacek Sieka
d5785677a8
split out eth1chain into its own module (#5768)
reduces import junk in some places - more could be done here
2024-01-17 15:26:16 +01:00
tersec
52ed0d6aa1
rm kzg test fixture code duplication and add test vector coverage checks (#5742) 2024-01-15 07:14:38 +01:00
tersec
c815e71af0
rest of EF KZG test harnesses (#5734) 2024-01-13 01:31:29 +00:00