* Reduce the logging-related breaking changes
* Don't disable the stdout log when the `--log-file` option is used
* Rename `--log-stdout` to `--log-format` and hide it
* Deprecate the `--log-file` option
* Address review comments in #3057
* reorder imports in rest_utils
maybe this will help with the mysterious serialization issues
Co-authored-by: Jacek Sieka <jacek@status.im>
This is a temporary revert of the new logic introduced in
d6cd1cd46c, because it leads
to a memory leak in the Eth1Monitor caused by the lack of
pruning of the Eth1Chain when the validator duties are not
executed.
According to the spec, this call should return the positions of
the specified validators within the sync committee. The existing
code was instead returning the indices of the sync sub-committees
where the validator is a member.
* fix stack overflow crash in REST/debug/getStateV2
* introduce `ForkyXxx` for generic type matching of `Xxx` across
branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) -
`Some` is already used for other types of type classes
* consolidate function naming in BeaconChainDB, use some generics
* import `forks.nim` from other spec modules and move `Forked*` helpers
around to resolve circular imports
* remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout
(less data shuffling between the types)
* fix several cases of states being stored on stack in tests, causing
random failures on some platforms
* remove reading json support from ncli - this should be ported to the
rest json reading instead (doesn't currently work because stack sizes)
The sync committee period used to be a plain `uint64`. With the light
client sync relying more heavily on them, it makes sense to introduce
a proper type for them, similar to how they are already used for `Slot`
and `Epoch`. This introduces such a `SyncCommitteePeriod` type.
Furthermore, some usage code dealing with those periods is cleaned up.
So far, the REST config response did not include all spec constants.
The spec for `/eth/v1/config/spec` defines that the response should
include constants for all hard forks known by the beacon node. This
patch extends the corresponding response to include more constants.
* Logging and startup improvements
Color support for released binaries!
* startup scripts no longer log to file by default - this only affects
source builds - released binaries don't support file logging
* add --log-stdout option to control logging to stdout (colors, json)
* detect tty:s vs redirected logs and log accordingly
* add option to disable log colors at runtime
* simplify several "common" logs, showing the most important information
earlier and more clearly
* remove line numbers / file information / tid - these take up space and
are of little use to end users
* still enabled in debug builds and tools
* remove `testnet_servers_image` compile-time option
* server images, released binaries and compile-from-source now offer
the same behaviour and features
* fixes https://github.com/status-im/nimbus-eth2/issues/2326
* fixes https://github.com/status-im/nimbus-eth2/issues/1794
* remove instanteneous block speed from sync message, keeping only
average
before:
```
INF 2021-10-28 16:45:59.000+02:00 Slot start topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:884 lastSlot=2384027 wallSlot=2384028 delay=461us84ns peers=0 head=75a10ee5:3348 headEpoch=104 finalized=cd6804ba:3264 finalizedEpoch=102 sync="wwwwwwwwww:0:0.0000:0.0000:00h00m (3348)"
INF 2021-10-28 16:45:59.046+02:00 Slot end topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:821 slot=2384028 nextSlot=2384029 head=75a10ee5:3348 headEpoch=104 finalizedHead=cd6804ba:3264 finalizedEpoch=102 nextAttestationSlot=-1 nextProposalSlot=-1 nextActionWait=n/a
```
after:
```
INF 2021-10-28 22:43:23.033+02:00 Slot start topics="beacnde" slot=2385815 epoch=74556 sync="DDPDDPUDDD:10:5.2258:01h19m (2361088)" peers=37 head=eacd2dae:2361096 finalized=73782:a4751487 delay=33ms687us715ns
INF 2021-10-28 22:43:23.291+02:00 Slot end topics="beacnde" slot=2385815 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 head=eacd2dae:2361096
```
* fix comment
* documentation updates
* mention `--log-file` may be deprecated in the future
* update various docs
* Add `child_id` field.
* Fix json-rpc api call and bump chronos.
* Bump chronos master and fix compilation warnings.
* One more bump of `chronos`.
* add random import
* export rest_utils a bit more
Co-authored-by: Jacek Sieka <jacek@status.im>
Spec constants currently use `uint16` instead of `GeneralizedIndex` to
define generalized indices in a merkle tree. Adjusted to use the latter
instead to avoid problems when adjusting `GeneralizedIndex`'s bit width.
* auto-bump nim-libp2p
* Remove peer info for other peers
Not definitive, just to test the libp2p's unstable branch
* finish up Remove peer info for other peers
* getKey -> getPublicKey
* bump libp2p
* libp2p bump
Co-authored-by: = <Menduist@users.noreply.github.com>
Co-authored-by: Tanguy <tanguy@status.im>
The validator client was only able to connect to beacon nodes exposing
the exact same set of spec constants that are locally known via their
config/spec REST API. However, that set of spec constants is dynamic.
As the validator client only requires a subset of relevant constants,
this may lead to compatible specs being rejected. This patch widens the
allowed specs by only verifying that the required set of constants are
present in the spec response, ignoring any spec constants that are not
locally known, and ignoring missing spec constants that are locally
known but not included by the remote beacon node when not relevant for
operation of the validator client.
Currently, `ElemType` can only be called on `HashArray`, `HashList` and
`List` types when they are first assigned to a variable called `T`. The
templates have been adjusted so that different variable names may be
used. The templates can now also be applied to in-line computed types,
e.g., `ElemType(typeof(x))`.
* `SyncCommitteeIndex` -> `SyncSubcommitteeIndex`
* `syncCommitteePeriod` -> `sync_committee_period` (spec spelling)
* tighten period comparisons
* fix assert when validating committee message with non-altair state in
REST api
There was a cyrillic T in some big-endian specific code that broke the
compilation on such platforms. This replaces that T with an ASCII T to
fix the build.
* register vc duties with subnet tracker
* fix activation logging during startup
* cache slot signature to avoid duplicate signature work
* schedule aggregation duties one slot at a time to avoid CPU spike at
each epoch
* lower aggregation subnet pre-subscription time to 4 slots (lowers
bandwidth and CPU usage)
* update stability subnets in ENR on startup
* log gossip state
* perform gossip subscriptions just before the next slot starts
* document stuff
* add random include
* don't overwrite subscription state when not subscribed
* log target gossip state
* updating gossip status once is enough
* add test
* remove syncQueueLen - this one is not updated at the end of the sync
and may cause gossip to disconnect itself completely - use a simple head
distance instead
* fix gossip disconnection - if in hysteresis, node.gossipState will be
set to disabled even though we don't disable topic subscriptions
* fix extra duty registration call
* Initial commit.
* Fix path.
* Add validator keys to indices cache mechanism.
Move syncComitteeParticipants to common place.
* Fix sync participants order issue.
* Fix error code when state could not be found.
Refactor `state/validators` to use keysToIndices mechanism.
* Fix RestValidatorIndex to ValidatorIndex conversion TODOs.
* Address review comments.
* Fix REST test rules.
Similar to the existing `RewardInfo`, this PR adds the infrastructure
needed to export epoch processing information from altair+. Because
accounting is done somewhat differently, the PR uses a fork-specific
object to extrct the information in order to make the cost on the spec
side low.
* RewardInfo -> EpochInfo, ForkedEpochInfo
* use array for computing new sync committee
* avoid repeated total active balance computations in block processing
* simplify proposer index check
* simplify epoch transition tests
* pre-compute base increment and reuse in epoch processing, and a few
other small optimizations
This PR introduces the type and does the heavy lifting in terms of
refactoring - the tools that use the accounting will need separate PR:s
(as well as refinements to the exportred information)
* "official" -> "scenarios", like the submodule
* fewer test binaries - various compile hacks have been improved over
time, test suite should follow
* remove obsolete bls tests - there are better test vectors in
nim-blscurve
* remove obsolete mentions of `ssz_testing`
* remove obsolete comments about proc vs globals, unittest2 already uses
proc's
This synchronizes defaultRuntimeConfig with v1.1.1 of the spec.
- Introduces `TERMINAL_TOTAL_DIFFICULTY` and `TERMINAL_BLOCK_HASH`.
- Sets `ALTAIR_FORK_EPOCH` in `mainnet` preset.
This synchronizes defaultRuntimeConfig with v1.1.1 of the spec.
- Introduces `TERMINAL_TOTAL_DIFFICULTY` and `TERMINAL_BLOCK_HASH`.
- Sets `ALTAIR_FORK_EPOCH` in `mainnet` preset.
So far, `withState` and `withBlck` templates could only be used to have
convenience access to fork-agnostic BeaconState and BeaconBlock fields.
This patch:
- injects an additional `stateFork` constant that allows to use
`when` expressions to also access Altair and Merge-specific fields.
- introduces a `withStateAndBlck` template to support operating on both
a `BeaconState` and `BeaconBlock` at a time.
- makes sync committee related functions Merge aware.
- changes a couple if-else trees for forks into case statements so that
forgotten future forks are promoted to compile-time errors.
So far, SSZ tests for `uint128` and `uint256` integers were disabled
due to "compile-time issues". The code has been adjusted to support
those additional test vectors, and the corresponding tests have been
enabled.
There are a few locations in the code that compare the current epoch to
the various FORK_EPOCH constants and branch off into fork-specific code.
When a new fork is introduced, it is sometimes forgotten to update all
of those branch locations. This patch introduces a compile-time check
that ensures that all branches need to be covered exhaustively. This is
done by replacing if-elif structures with case expressions.
There are a number of locations in the code that get attestations on a
forked beacon state. For attestation pools test, a convenience wrapper
was available to reduce clutter. This patch integrates that wrapper into
the core component so that it can also take advantage of the wrapper.
The initialization of a `SyncAggregate` to its default value is not very
intuitive. There is an `init` function in `sync_committee_msg_pool` that
provides a convenience wrapper. This patch exports that initializer so
that the rest of the code base can also take advantage of it.
Adds a function that constructs a Merkle proof for a generalized index.
This will be used during light client sync to update light clients with
a new state (see NEXT_SYNC_COMMITTEE_INDEX / FINALIZED_ROOT_INDEX).
The phase0 and altair overloads of `makeBeaconBlock` slightly differ in
their signatures which makes using them unnecessarily verbose.
- A placeholder `sync_aggregate` argument similar to `executionPayload`
is added to the phase0 overload to match the altair signature.
- A wrapper operating on `ForkedHashedBeaconState` is introduced.
This still doesn't work properly because it leads to a change in
the "eth" ENR field of the client. Since the client uses the value
of this field to look for other nodes on the network, the change
effectively prevents us from finding peers.
There were still a few instances that used the expansion of `errReject`
instead of using the template itself. It seems that those cases were
forgotten as part of other cleanups in #2809. Done now for readability.
* simplify state fork access pattern
* fixes
* unsafeAddr needs to be dereferenced outside of case for best effect
* remove hash_tree_root of ForkedXxx (doesn't make sense)
* simplify state transition
* fix vc
* readd hash_tree_root(forkedbeaconblock)
* readd htr(fhbs) as well
...and add some protections to not hash the wrong items elsewhere
There are still a few cases with manual loops through sync subcommittees
even though an `allSyncCommittees` iterator exists. Adjusted the few
remaining instances to also use the iterator instead.
When sync committee message handling was introduced in #2830, the edge
case of the same validator being selected multiple times as part of a
sync subcommittee was not covered. Not handling that edge case makes
sync contributions have a lower-than-expected participation rate as each
sync validator is only counted up through once per subcommittee.
This patch ensures that this edge case is properly covered.
The P2P spec defines how certain error classes should be handled through
either IGNORE or REJECT verdicts. For sync committee message, the spec
defines that only the first message from each validator per subcommittee
and slot shall be accepted, the rest is ignored. However, current code
rejects those messages instead of ignoring them. Fixed to match spec.
The README file explaining gossip_processing, and the attestation_flow
docs were no longer accurate, as attestations and aggregates no longer
go through a queue (pending batching). This patch updates the docs
accordingly. It also improves some grammar and fixes some typos.
Currently there is a mix of comment formats in the various preset files.
Altair presets have a leading header line, Phase0 presets don't.
Furthermore, the Altair `minimal` header refers to `mainnet` (#2710).
Updated all of the header lines to match the spec.
The current `getStateField` implementation fails at run-time when called
on a post-Altair state. This is improved by replacing the `if` structure
with a `case` expression, which is checked for exhaustive coverage at
compile time. Care is taken to preserve the `unsafeAddr` optimization.
* Placing callbacks into strategic places.
* Initial events call implementation.
* Post rebase fixes.
* Change addSyncContribution() implementation.
* Add `attestation-sent` event.
Remove gcsafe, raises from callbacks implementations.
Move `attestation-received` fire at the end of attestation processing.
* Address review comments.
Some clients disable their metadata V1 after the Altair fork, so we are unable to ping them correctly.
This PR adds a fallback to try metadata V2 if V1 fails.
* Add parallel attestation verification
* Update tests, batchVerify doesn't use the threadpool with only single core (nim-blscurve update)
* bump nim-blscurve
* Debug info for failing eth2 test vectors
* remove submodule eth2-testnets
* verbose debugging of make failure on Windows (libbacktrace?)
* Remove CI debug mode
* initialization convention
* Fix new altair tests
The template `message` extracts fork specific data from a forked block.
However, its current implementation does not compile due to each fork
returning a different type, i.e., the generated `case` code attempts to
return different types from each branch. As the template is unused, it
is safely removed for now.
* add merge SSZ consensus object tests
* add merge attestations, attester slashing, proposer slashing, and voluntary exit test fixtures
* throw catchable exception rather than assert on invalid SingleMemberUnion selector
* hash_tree_root can assert, because it's trusted data by that point
* re-assert on writing, since also trusted data
* beta.4 update
* implement upgrade_to_merge()
Zahary Karadjov
tersec
Jacek Sieka
Etan Kissling
Eugene Kabanov
Kim De Mey
Ștefan Talpalaru
nbc-bump-bot[bot]
sacha
Tanguy
Mamy Ratsimbazafy