280 Commits

Author SHA1 Message Date
Etan Kissling
6af70e2f80
wrap applyExecutionPayload in factories for Nim 2.0 (#5395)
In Nim 2.0, the `test_fixture_operations` files fail to compile with:

```
Error: 'result' is of type <Result[system.void, system.cstring]> which cannot be captured as it would violate memory safety, declared here: /nimbus-eth2/tests/consensus_spec/bellatrix/test_fixture_operations.nim(130, 5); using '-d:nimNoLentIterators' helps in some cases. Consider using a <ref Result[system.void, system.cstring]> which can be captured.
```

Wrapping the `applyExecutionPayload` in a factory that takes `path`
avoids this problem.
2023-09-06 10:35:37 +02:00
Etan Kissling
81397342d0
fix relative import paths for Nim 2.0 (#5397)
In Nim 2.0, relative `import` paths are validated more strictly;
this fixes the incorrect paths used in some tests.
2023-09-06 08:17:59 +00:00
Jacek Sieka
6fab826487
bump unittest2, clean up suite names (#5384)
Suite names were not being used because `test` has to have access to it
during instantiation - this PR cleans things up a little while at the
same time upgrading unittest2.
2023-09-04 13:05:11 +00:00
tersec
85e1976ac3
automated consensus spec URL updating to v1.4.0-beta.1 (#5280) 2023-08-09 03:58:47 +00:00
Jacek Sieka
b8a32419b8
async batch verification (+40% sig verification throughput) (#5176)
* async batch verification

When batch verification is done, the main thread is blocked reducing
concurrency.

With this PR, the new thread signalling primitive in chronos is used to
offload the full batch verification process to a separate thread
allowing the main threads to continue async operations while the other
threads verify signatures.

Similar to previous behavior, the number of ongoing batch verifications
is capped to prevent runaway resource usage.

In addition to the asynchronous processing, 3 addition changes help
drive throughput:

* A loop is used for batch accumulation: this prevents a stampede of
small batches in eager mode where both the eager and the scheduled batch
runner would pick batches off the queue, prematurely picking "fresh"
batches off the queue
* An additional small wait is introduced for small batches - this helps
create slightly larger batches which make better used of the increased
concurrency
* Up to 2 batches are scheduled to the threadpool during high pressure,
reducing startup latency for the threads

Together, these changes increase attestation verification throughput
under load up to 30%.

* fixup

* Update submodules

* fix blst build issues (and a PIC warning)

* bump

---------

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2023-08-03 11:36:45 +03:00
Jacek Sieka
5bc48acc36
reduce memory allocations during state transition (#5235)
This PR removes a few hundred thousand temporary seq allocations during
state transition - in particular, the flag seq was allocated per
validator while committees are computed per attestation.
2023-08-03 01:03:40 +02:00
tersec
614202e30d
automated consensus spec URL updating to v1.4.0-beta.0 (#5121) 2023-06-24 15:43:30 +00:00
Etan Kissling
2722778ce5
reduce nim-eth dependencies just for RNG (#5099)
We have several modules that import `nim-eth` for the sole purpose of
its `keys.newRng` function. This function is meanwhile a simple wrapper
around `nim-bearssl`'s `HmacDrbgContext.new()`, so the import doesn't
really serve a use anymore. Replace `keys.newRng` with the direct call
to reduce `nim-eth` imports.
2023-06-19 22:43:50 +00:00
Etan Kissling
3a6b50cf21
require sync committee supermajority in CI (#5085)
* require sync committee supermajority in CI

To better catch problems with sync committee messages in CI, extend
local testnet simulation to also verify that each block is signed
by a supermajority of the sync committee.

Requires #5083 and #5084

* lint
2023-06-17 06:00:01 +00:00
tersec
591c2246d5
update consensus spec URLs to v1.4.0-alpha.3 (#5088) 2023-06-16 16:45:09 +00:00
tersec
788cdb7133
automated v1.4.0-alpha.2 to v1.4.0-alpha.3 consensus spec URL updates (#5065) 2023-06-13 14:03:49 +00:00
tersec
22208836b1
automated v1.4.0-alpha.1 to v1.4.0-alpha.2 consensus spec URL updates (#5056) 2023-06-10 09:56:54 +00:00
tersec
f86febc111
update consensus spec URLs to v1.4.0-alpha.1 (#5027) 2023-06-04 10:32:50 +00:00
tersec
b25ca0833b
use v1.4.0-alpha.1 consensus spec test vectors (#5026) 2023-06-03 21:55:08 +00:00
tersec
1f535336da
automated v1.3.0 to v1.4.0-alpha.0 consensus spec URL updates (#4996)
* automated v1.3.0 to v1.4.0-alpha.0 consensus spec URL updates

* add copyright year headers
2023-05-26 00:14:28 +02:00
tersec
10569ff2e9
use v1.4.0-alpha.0 consensus spec test vectors (#4990) 2023-05-25 07:55:00 +00:00
Etan Kissling
22c298c07b
final portion of trivial v1.3.0 bumps (#4926)
* final portion of trivial v1.3.0 bumps

Updates unchanged logic to latest v1.3.0 consensus-specs refs.

```
https://github.com/ethereum/consensus-specs/(blob|tree)/(?!v1\.3\.0/)
```

* lint
2023-05-11 09:54:29 +00:00
Etan Kissling
e6e4ba9de6
clean up redundant tests and config (#4836)
The consensus-spec-tests already cover the scenarios of our custom test
runner, so the custom tests can be removed. Also cleans up unused config
flags and related unreachable logic.
2023-04-18 21:26:36 +02:00
tersec
2f634c10a4
automated consensus spec URL updating from v1.3.0-rc.4 to rc.5 (#4756) 2023-03-21 00:42:22 +00:00
tersec
ec77116414
automated consensus spec URL updating from v1.3.0-rc.3 to rc.4 (#4742) 2023-03-17 01:10:31 +00:00
Etan Kissling
fb26e19de9
use v1.3.0-rc.4 consensus spec test vectors (#4740) 2023-03-16 21:35:20 +01:00
Etan Kissling
bc74c35283
rename _state_fork > _consensus_fork (#4730) 2023-03-12 02:34:38 +00:00
Etan Kissling
fd8e86972d
rename lcDataForkAtStateFork > lcDataForkAtConsensusFork (#4726) 2023-03-11 20:09:21 +00:00
Etan Kissling
d8f39dd24f
rename stateForkForDigest > consensusForkForDigest (#4727) 2023-03-11 16:58:48 +00:00
Etan Kissling
b0cccae7f4
rename (with|at)StateFork > (with|at)ConsensusFork (#4725) 2023-03-11 14:39:29 +00:00
Etan Kissling
eba5f9af26
rename upgrade_to_eip4844 > upgrade_to_deneb (#4723) 2023-03-11 14:28:55 +00:00
Etan Kissling
ad118cd354
rename stateFork > consensusFork (#4718)
Just the variable, not yet `lcDataForkAtStateFork` / `atStateFork`.

- Shorten comment in `light_client.nim` to keep line width
- Do not rename `stateFork` mention in `runProposalForkchoiceUpdated`.
- Do not rename `stateFork` in `getStateField(dag.headState, fork)`

Rest is just a mechanical mass replace
2023-03-11 00:35:52 +00:00
Etan Kissling
bc37a2d1a6
update transition test category from EIP4844 to Deneb (#4716) 2023-03-10 17:12:46 +00:00
tersec
e3d96ef147
rename most eip4844Data to denebData (#4693) 2023-03-04 22:23:52 +00:00
tersec
3b41e6a0e7
rename ConsensusFork.EIP4844 to ConsensusFork.Deneb (#4692) 2023-03-04 13:35:39 +00:00
tersec
ea060de6d4
more eip4844 -> deneb module references (#4690) 2023-03-02 21:09:24 +01:00
tersec
d058aa09c8
more withdrowls (#4674) 2023-03-02 17:13:35 +01:00
henridf
1de3cf5246
Remove SignedBeaconBlockAndBlobsSidecar (#4683)
This commit removes SignedBeaconBlockAndBlobsSidecar and all remaining
references.
2023-03-02 15:12:04 +01:00
tersec
982d79f9a2
more eip4844 -> deneb changes (#4666) 2023-02-25 03:03:34 +02:00
tersec
2dd2a782d9
disable BareExcept warnings (#4662)
* disable BareExcept warnings

* BareExcept only exists in >= 1.6.11
2023-02-24 19:50:46 +00:00
tersec
069d973fc6
use 'deneb' in 4 more modules (#4658) 2023-02-23 19:06:57 +01:00
tersec
29fb65a9db
automated update of v1.3.0-rc.2 to v1.3.0-rc.3 consensus spec URLs (#4647) 2023-02-21 16:43:21 +00:00
tersec
68cb9fe7b1
use consensus spec v1.3.0-rc.3 test vectors (#4645) 2023-02-21 12:58:08 +00:00
tersec
4597486dcc
rename directory tests/consensus_specs/{eip4844,deneb}/ (#4628) 2023-02-15 21:07:31 +00:00
zah
067ba13c52
Capella metadata for Sepolia (#4615)
Other changes:

Renamed the `EIP_4844_FORK_*` config constants to `DENEB_FORK_*` as
this matches the latest spec and it's already used in the official
Sepolia config.
2023-02-15 14:44:09 +00:00
tersec
d12c32e988
EIP4844 -> Deneb in EF consensus spec test output (#4618) 2023-02-14 13:32:43 +00:00
Eugene Kabanov
e51095e2f8
Fix Windows MAX_PATH constraint issue in CI. (#4576)
* Fix MAX_PATH limitation in tests.

* Fix posix issues.

* Fix compilation issue.
2023-02-10 21:59:38 +01:00
tersec
58ed9308d2
automated v1.3.0-rc.1 to v1.3.0-rc.2 consensus spec URL updates (#4568) 2023-01-31 00:26:57 +01:00
tersec
f5e302f973
use 1.3.0-rc.2-hotfix consensus spec test vectors (#4566) 2023-01-30 22:54:16 +00:00
tersec
29ea42ce1a
use v1.3.0-rc.2 consensus spec test vectors (#4564) 2023-01-29 11:28:04 +00:00
Etan Kissling
258f151594
prepare LC update ranking test runner for Capella (#4563)
* prepare LC update ranking test runner for Capella

Adds missing Capella/EIP4844 support to LC update ranking runner.

* fix uncommitted typo

* skip capella / eip4844 tests

* cleanup test skip
2023-01-29 03:34:26 +00:00
tersec
0fb726c420
BeaconStateFork/BeaconBlockFork -> ConsensusFork (#4560)
* `BeaconStateFork/BeaconBlockFork` -> `ConsensusFork`

* revert unrelated change

* revert unrelated changes

* update test summaries
2023-01-28 19:53:41 +00:00
Etan Kissling
65ca523482
support EIP4844 LC data format (#4520)
Implements the proposed light client data format for EIP4844:
- https://github.com/ethereum/consensus-specs/pull/3151
2023-01-19 23:53:37 +01:00
Etan Kissling
58f4c13e14
support Capella LC data format (#4514)
Implements the proposed light client data format for Capella:
- https://github.com/ethereum/consensus-specs/pull/3151
2023-01-18 15:32:57 +01:00
tersec
073c544f0c
automated update from v1.3.0-rc.0 to v1.3.0-rc.1 consensus spec URLs (#4517) 2023-01-17 16:10:52 +00:00