Commit Graph

6420 Commits

Author SHA1 Message Date
Etan Kissling d7aaf1b809
only request `nextPeriod` sync duties close to end of `currentPeriod` (#5409)
Currently we always request duties for current and next sync period.
As sync periods are quite long (~27 hrs on Mainnet), having access to
the duties so early doesn't help too much. To avoid running into errors
when the BN does not have the duties available around period boundary,
delay requesting them until the current period is close to finish.

`SYNC_COMMITTEE_SUBNET_COUNT` epochs are what the spec says should be
the lookahead timing of starting to subscribe to sync committee gossip.
Reusing the constant here for consistency.

This fixes these warning messages in the first slot of a new period.

```
rocketpool_validator   | WRN 2023-09-07 20:19:35.439+00:00 Beacon node has incompatible configuration reason="Epoch value is far from the future;400;getSyncCommitteeDuties(first);invalid-request" node=http://eth2:5052[Nimbus/v23.8.0-872b19-stateofus] node_index=0 node_roles=AGBSDT
rocketpool_validator   | WRN 2023-09-07 20:19:35.440+00:00 Unable to get sync committee duties        period=889 epoch=227584 reason="Epoch value is far from the future;400;getSyncCommitteeDuties(first);invalid-request" service=duties_service
rocketpool_validator   | NOT 2023-09-07 20:19:35.441+00:00 Beacon node is in sync                     head_slot=7274495 sync_distance=1 is_optimistic=false node=http://eth2:5052[Nimbus/v23.8.0-872b19-stateofus] node_index=0 node_roles=AGBSDT
```
2023-09-13 08:58:28 +02:00
Etan Kissling cd68c71c6c
add gossip tests for period boundary (#5423)
Test `validateSyncCommitteeMessage` and `validateContribution`
around sync committee period boundary to cover edge cases.
2023-09-13 08:32:11 +02:00
Etan Kissling 4700030231
fix sync contribution validation for first slot per period (#5408)
Sync committee duties are performed by the sync committee as determined
by slot + 1. We did it correctly for individual messages, but selected
the incorrect participants for aggregate contributions for the very
first slot per period (roughly 1 per ~27 hrs on Mainnet). The faulty
participants selection code was originally introduced in #2925.
2023-09-13 03:23:18 +02:00
andri lim 5c88e74c08
Bump stint to v2.0: new array backend (#5113)
* bump stint to v2.0: new array backend

* Fix missing isEven in libnimbus_lc.nim

* bump nim-stint: compiles with arm64
2023-09-13 01:16:04 +07:00
tersec d4543239d7
rm state_sim WASM wrapper/demo (#5421) 2023-09-12 15:03:18 +00:00
Miran 73ab2e5f39
styling improvements for holesky.md (#5420)
- compact syntax for different options (as in the rest of the guide)
- correct numeration of the list (continuation vs starting from 1)
- add links to useful resources
- a note stating that Prater is deprecated
2023-09-12 13:53:55 +03:00
Etan Kissling 8fa5580c50
add infrastructure to select fork choice version (#5387)
To allow testing https://github.com/ethereum/consensus-specs/issues/3466
add support for selecting fork choice version at launch. This means we
can deploy a different logic when `DENEB_FORK_EPOCH != FAR_FUTURE_EPOCH`
that won't be used on Mainnet.
2023-09-12 09:52:51 +02:00
tersec a06c0c550f
enable Windows GitHub Actions with Nim 20 (#5413) 2023-09-12 04:57:20 +00:00
Zahary Karadjov de8d52eb1a
Add Holesky section to the guide 2023-09-12 03:50:29 +03:00
Zahary Karadjov 568e1fbb8c
Final Release Notes for v23.9.0 2023-09-12 02:00:38 +03:00
Zahary Karadjov 6b1ab81046
Support both snappy-encoded and pure SSZ genesis states 2023-09-12 01:52:15 +03:00
Etan Kissling 672c69b0bf
increase Windows test stack size for Nim 2.0 (#5419)
Nim 2.0 has trouble running the `test_libnimbus_lc` test with the MinGW
default main thread stack size of 2 MB. Bump test stack size to 8 MB.
2023-09-11 20:59:07 +00:00
tersec 2b4f987c80
remove pre-v1.4.0 attestation stability subnets (#5402)
* remove pre-v1.4.0 attestation stability subnets

* re-add most of should register stability subnets on attester duties test
2023-09-11 16:03:34 +00:00
Etan Kissling ff176a1569
fix `syncCommitteeDuties` log in `Slot end` message (#5416)
In `Slot end`, sync duties are reported one slot too late,
because sync committee duties are determined by the next slot
instead of the current one. Address that by taking account of this.
2023-09-11 10:51:55 +00:00
Jacek Sieka 4d8e972e70
snappy: bump (#5417) 2023-09-11 12:49:54 +02:00
Eugene Kabanov 3c3c4e8edf
Fix /eth/v1/validator/liveness/{epoch} call names and links in code (#5392)
* Fix getLiveness() call links and names.

* Add getLiveness() decoding test.

* Remove BN state updates from getLiveness() call.

* Fix AllTests.

* Reduce number of tests.
2023-09-11 12:07:34 +02:00
tersec 91a53e0cb6
capella genesis in local testnet (#5403) 2023-09-11 11:26:21 +02:00
tersec 10ec7be686
optimize epoch registry processing (#5412) 2023-09-11 11:21:50 +02:00
tersec cc13e0b7e0
restore full test coverage for process_rewards_and_penalties (#5407)
* restore full test coverage for process_rewards_and_penalties

* adjust ncli_db to use new iterator
2023-09-08 16:42:18 +00:00
tersec 59ccd25d8d
disable GitHub Actions Windows CI harder 2023-09-08 16:35:39 +00:00
tersec b424d214c2
Windows GitHub Action builds OOM; disable (#5410) 2023-09-08 16:13:25 +00:00
Zahary Karadjov 88e8a2818a
Add the new options to the Nimbus guide 2023-09-08 14:31:57 +03:00
Zahary Karadjov f0bb1cbd4e
Version 23.9.0 2023-09-08 13:33:51 +03:00
Zahary Karadjov c7db7d0f2c
Add CLI options `--genesis-state` and `--genesis-state-url` 2023-09-08 13:10:18 +03:00
zah 53589b5a7d
Add metadata for the Holesky network (#5337)
* Add metadata for the Holesky network

* Add copyright banner to the new Nim module

* Working version

* Bump Chronos to fix downloading from Github
* Add checksum check of the downloaded file
* Clean up debugging code and obsolete imports
2023-09-08 08:53:27 +03:00
Tanguy c5a79a6f8f
Add IDontWant metrics (#5405)
* Add IDontWant metrics

* update .gitmodules
2023-09-07 15:13:04 +02:00
tersec f53422c181
optimize epoch transition via get_flag_index_deltas() and get_inactivity_penalty_deltas() (#5404)
* optimize epoch transition via get_flag_index_deltas() and get_inactivity_penalty_deltas()

* directly mutate Altair info in {get,update}_flag_and_inactivity_deltas
2023-09-07 11:14:52 +00:00
tersec fbcb476ff9
allow archiving blobs (#5391) 2023-09-07 08:19:33 +02:00
zah 2b5bd74e15
Rename --validator-source to --web3-signer-url and document it (#5389)
Also allows multiple instances to be configured
2023-09-06 22:04:10 +03:00
Etan Kissling b8db44d761
update tested Nim versions (#5142)
Add `version-2-0` testing, and make upstream version testing Linux only.

Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-09-06 19:00:15 +00:00
Etan Kissling f420f09ac1
less confusing lint error msg when bumping to very recent commit (#5400)
When bumping to a more recent commit than the configured `branch`,
currently the lint error message is confusing:

```
fatal: error processing shallow info: 4
Submodule 'vendor/nim-chronos': Failed to fetch 'master':
```

This happens when the selected commit is more recent than the latest
one on the `branch`. Comparing the commit dates allows a better message.
2023-09-06 18:52:31 +00:00
Etan Kissling 89d133d0e5
further reduce stack size of LC helpers for Nim 2.0 (#5401)
The `is_next_sync_committee_known` helper allocates a fresh
`SyncCommittee` in the caller and `nimZeroMem`s it on each use.
Use `static` to compare against a compile-time zeroed copy instead.
This also should help reduce stack size far enough to link with Nim 2.0.
2023-09-06 16:05:12 +02:00
Etan Kissling ba8b5d8d67
lower number of `--remote-validators-count` in CI (#5399)
The CI signing node sometimes becomes overloaded, leading to late block.

```
{"lvl":"DBG","ts":"2023-09-05 23:46:34.515+01:00","msg":"REST request body has been sent","remote":"127.0.0.1:5201","request":"/api/v1/eth2/sign/0x82ad1b336ce728978983cbe7c7dd25f757fdc66536f3457cf9f0120c07943f8d464eec55fefcaef9c0d5da6e9f7c1443","size":549,"http_method":"POST"}
```

```
DBG 2023-09-05 23:46:36.098+01:00 Received request                           peer=127.0.0.1:51194 meth=POST uri=/api/v1/eth2/sign/0x82ad1b336ce728978983cbe7c7dd25f757fdc66536f3457cf9f0120c07943f8d464eec55fefcaef9c0d5da6e9f7c1443
```

Especially on the `minimal` preset with short slot timings, this leads
to attestations and sync duties being made for an old head, ultimately:

```
{"lvl":"FAT","ts":"2023-09-05 23:46:41.915+01:00","msg":"Low sync committee participation","topics":"chaindag","slot":15,"num_active_participants":8}
```

Looking at `nimbus_signing_node.0.jsonl` shows that the signing node is
very busy. Lowering the validator count should reduce the latency a bit.
2023-09-06 09:08:33 +00:00
Etan Kissling 99507e19cd
bump `nim-ssz-serialization` to `53428359a6759ac34bb4b16acb234f8477153b8f` (#5398)
- convert func to iterator
- Fix compiler complaining type mismatch for byte array
- workaround for Nim 2.0 issue with the template
2023-09-06 08:51:19 +00:00
Etan Kissling 6af70e2f80
wrap `applyExecutionPayload` in factories for Nim 2.0 (#5395)
In Nim 2.0, the `test_fixture_operations` files fail to compile with:

```
Error: 'result' is of type <Result[system.void, system.cstring]> which cannot be captured as it would violate memory safety, declared here: /nimbus-eth2/tests/consensus_spec/bellatrix/test_fixture_operations.nim(130, 5); using '-d:nimNoLentIterators' helps in some cases. Consider using a <ref Result[system.void, system.cstring]> which can be captured.
```

Wrapping the `applyExecutionPayload` in a factory that takes `path`
avoids this problem.
2023-09-06 10:35:37 +02:00
Etan Kissling 81397342d0
fix relative import paths for Nim 2.0 (#5397)
In Nim 2.0, relative `import` paths are validated more strictly;
this fixes the incorrect paths used in some tests.
2023-09-06 08:17:59 +00:00
Etan Kissling ac3b2b4233
rename `U` to `maxLen` in `statediff` for Nim 2.0 (#5396)
Nim 2.0 gets confused when compiling `all_tests`:

```
Error: undeclared identifier: 'maxLen'
candidates (edit distance, scope distance); see '--spellSuggest':
 (3, 7): 'Table'
 (3, 7): 'len'
 (3, 7): 'max'
```

Renaming the generic parameter `U` to `maxLen` fixes this somehow.
It also increases readability to use the same name consistently.
2023-09-06 02:39:21 +00:00
Eugene Kabanov 4ce8e77b56
Address issue #5362. (#5370)
* Address issue #5362.

* Remove RestSpec object declaration and getSpec() API call declaration.

* Address review comments.
2023-09-05 23:39:01 +03:00
Etan Kissling 8ffb80e954
avoid `Taskpool.spawn` within `{.async.}` for Nim 2.0 (#5393)
In Nim 2.0, attempting to use `Taskpool.spawn` inside `{.async.}` `proc`
leads to `Error: cannot generate destructor for generic type: Isolated`.

Add an intermediate wrapper `proc` that performs the `spawn` operation
to workaround the problem.
2023-09-05 21:36:35 +02:00
Etan Kissling dbb1a63ca9
collapse `partialBeaconBlock` templates into one (#5386)
Have a single `Forky` template for `partialBeaconBlock` production
rather than have several copies that are mostly identical.
2023-09-05 15:59:17 +02:00
tersec 0987628c0b
simplify beacon validator sync committee production for blocks slightly (#5364) 2023-09-05 12:31:33 +02:00
Eugene Kabanov afb5fdae34
Dynamic validators loop (#5385)
* Add periodic dynamic validators polling.
* Add dynamic validators tests.
2023-09-04 22:14:58 +03:00
Etan Kissling be9ecfa1c9
add helpers for processing withdrawals to `libnimbus_lc.a` (#5374)
Similar to the existing helpers for processing transactions / receipts,
extend `libnimbus_lc.a` with support for processing withdrawals as well.
2023-09-04 20:44:03 +02:00
tersec 41ce581149
invalidate only required htr cache portion for immutable validators (#5381) 2023-09-04 19:48:31 +02:00
Jacek Sieka bd513ca2ba
separate CL/EL timing in add block log (#5380)
the metric remains the same in order not to change overall meaning, but
in the logs we can do better - often, the EL is at the core of a
slowdown.
2023-09-04 20:23:29 +03:00
Jacek Sieka 6fab826487
bump unittest2, clean up suite names (#5384)
Suite names were not being used because `test` has to have access to it
during instantiation - this PR cleans things up a little while at the
same time upgrading unittest2.
2023-09-04 13:05:11 +00:00
tersec f99bb881f4
use finalized checkpoint as default trusted node documented target (#5382) 2023-09-02 05:10:54 +00:00
tersec 8725595328
bump nim-ssz-serialization (#5378) 2023-09-01 11:08:52 +00:00
tersec a8c56b1660
update some consensus-spec URLs to v1.4.0-beta.1 (#5379) 2023-09-01 09:31:52 +00:00
Jacek Sieka 6f59c6c46d
nimcrypto: bump (#5376)
aligns `MDigest`
2023-08-31 13:30:59 +00:00