Commit Graph

124 Commits

Author SHA1 Message Date
tersec 7faef7827e
fix EIP4844 withBlck (#4411)
* fix EIP4844 withBlck

* don't raiseAssert by default
2022-12-14 18:30:56 +01:00
Jacek Sieka 657d571e25
Use same skipping tense as other logs 2022-12-09 20:15:46 +01:00
Jacek Sieka 6e2a02466e
unify bn/vc doppelganger detection (#4398)
* fix REST liveness endpoint responding even when gossip is not enabled
* fix VC exit code on doppelganger hit
* fix activation epoch not being updated correctly on long deposit
queues
* fix activation epoch being set incorrectly when updating validator
* move most implementation logic to `validator_pool`, add tests
* ensure consistent logging between VC and BN
* add docs
2022-12-09 17:05:55 +01:00
henridf f0329b2212
Types and scaffolding for EIP-4844 (#4365)
* Types and scaffolding for EIP-4844

This commit adds the EIP-4844 spec types, and fills in
scaffolding/boilerplate for the use of these types across the repo.

None of the actual EIP-4844 logic is introduced yet.

This follows the pattern used by @tersec when introducing Capella (#4276).

* use eth2-networks fork

* review feedback: add static check EIP4844_FORK_EPOCH == FAR_FUTURE_EPOCH

* review feedback: remove EIP4844 from /eth/v1/config/spec response

* Cleanup / review feedback

* Fix REST test
2022-12-05 16:29:09 +00:00
tersec 5c16062de9
remove all but truly stub support for `SHARDING_FORK_{EPOCH,VERSION}` (#4385) 2022-12-02 13:33:18 +01:00
tersec 474b0d8502
`withUpdatedState` injects `updatedState` rather than `state` template (#4375) 2022-11-30 16:37:23 +02:00
tersec 61c5ac32d8
automated consensus spec ref URL update to v1.3.0-alpha.1 (#4354) 2022-11-24 19:07:02 +00:00
tersec 1fceb33b2e
more capella (#4350) 2022-11-24 14:38:07 +00:00
Eugene Kabanov 8fa6064b9a
VC: blinded block publishing support (#4332)
* Add blind REST API declarations and implementations.

* shortLog is still not stable.

* Fix shortLog issues.

* Enable disabled logging statements.

* Address review comments.

* Avoid templates suffering from double evaluation of their params

* Address review comments.

* Fix compilation issue.

Co-authored-by: Zahary Karadjov <zahary@status.im>
2022-11-24 09:14:05 +00:00
tersec c8083f2c32
implement more missing capella functionality (#4344) 2022-11-24 09:53:04 +02:00
Eugene Kabanov fb4fea81b5
Fix doppelganger protection in validator duties. (#4345)
Fix missing activationEpoch setup.
2022-11-24 09:48:10 +02:00
Eugene Kabanov eb661565ed
Per-validator doppelganger protection. (#4304)
* Initial commit.

* NextAttestationEntry type.

* Add doppelgangerCheck and actual check.

* Recover deleted check.

* Remove NextAttestainEntry changes.

* More cleanups for NextAttestationEntry.

* Address review comments.

* Remove GENESIS_EPOCH specific check branch.

* Decrease number of full epochs for doppelganger check in VC.

Co-authored-by: zah <zahary@status.im>
2022-11-20 15:55:43 +02:00
tersec 35b1104bea
`block_sim` runs capella by default (#4315) 2022-11-11 10:17:27 +00:00
tersec 909c095e64
initial automated v1.2.0 -> v1.3.0-alpha.0 consensus spec URL update (#4296) 2022-11-08 02:37:28 +00:00
tersec 5b46f0b723
add Capella support to Forked* (#4276)
* add Capella support to Forked*

* remove cruft

* add `OnForkyBlockAdded`
2022-11-02 16:23:30 +00:00
Eugene Kabanov 367e7052f4
VC: some fixes (#4240)
* Skip doppelganger protection for validators which activated just now or in future.

* Fix sync committee duties spam issue.

* Optimize sync committee duties logging statements.

* Fix missing lazyWait.

* Add short path.

* Address #4087.

* Add missing watch for crash.
2022-10-21 16:53:30 +02:00
Eugene Kabanov 9fd4e7405e
Remove validator_duties dependency from validator_client. (#4244) 2022-10-17 13:42:43 +00:00
tersec c367b14ad9
deprecate `--safe-slots-to-import-optimistically` (#4182) 2022-09-29 06:29:49 +00:00
tersec 3c03ba86c1
update consensus spec ref URLs to v1.2.0 (#4164) 2022-09-23 07:56:06 +00:00
tersec 56720dd808
update consensus layer spec ref URLs to v1.2.0-rc.3 (#4143) 2022-09-20 02:08:09 +02:00
Jacek Sieka ef8bab58eb
load suggested fee recipient file also when keymanager is disabled (#4078)
Since these files may have been created in a previous run or manually,
we want to keep loading them even on nodes that don't enable the
keystore API (for example static setups)

Other changes:

* log keystore loading progressively (#3699)
* print initial fee recipient when loading validators
* log dynamic fee recipient updates
2022-09-17 08:30:07 +03:00
tersec 80f44f4491
update consensus layer spec ref URLs to v1.2.0-rc.3 (#4117) 2022-09-13 17:30:11 +00:00
tersec 8be964a152
update consensus layer spec ref URLs to v1.2.0-rc.3 (#4109) 2022-09-10 17:16:38 +00:00
tersec 1d620f0123
consensus spec URL updates to v1.2.0-rc.3 (#4105) 2022-09-09 21:56:06 +00:00
tersec ad0d30093f
state/forkyState cleanup; spec URL updates; rm unused imports (#4052) 2022-08-31 13:29:34 +02:00
tersec 9ae796daed
Cache and resend, rather than recreate, builder API registrations (#4040) 2022-08-31 03:29:03 +03:00
zah b1ac9c9fe4
Fix a potential segfault and various potential stalls (#4003)
* Fixes a segfault during block production when the Keymanager API
  is disabled. The Keymanager is now disabled on half of the local
  testnet nodes to catch such problems in the future.

* Fixes multiple potential stalls from REST requests being done
  without a timeout. From practice, we know that such requests
  can hang forever if not cancelled with a timeout. At best,
  this would be a resource leak, at worst, it may lead to a
  full stall of the client and missed validator duties.

* Changes some Options usages to Opt (for easier use of valueOr)
2022-08-19 21:51:30 +00:00
Eugene Kabanov 250f7b4bdf
Exclusive keystore locking (#3907) 2022-08-08 00:53:20 +03:00
tersec d62d13a23c
MEV block proposal (#3883)
* MEV validator registration

* add nearby canary to detect new beacon chain forks

* remove special MEV graffiti

* web3signer support

* fix trace logging

* Nim 1.2 needs raises Defect

* use template rather than proc in REST JSON parsing

* use --payload-builder-enable and --payload-builder-url

* explicitly default MEV to disabled

* explicitly empty default value for payload builder URL

* revert attestation pool to unstable version
2022-08-01 09:41:47 +03:00
Miran dfd4afc9f2
compatibility with Nim 1.4+ (#3888) 2022-07-29 10:53:42 +00:00
Eugene Kabanov c3d3397843
VC: doppelganger protection (#3877)
* Improve fallback_service.

* Improve logging in fallback_service.

* Apply signal handling for all stages.

* Fix some logging statements.

* Add doppelganger REST api endpoint.
Add some structures to VC.

* Add client API call implementation.

* Initial fix & refactor onceToAll()
Add doppelganger service.
Add doppelganger helpers.

* Add doppelganger checks.

* Move doppelganger log messages to higher levels.

* Fix firstSuccess().

* Bump chronos.

* Post rebase fixes.

* Proper chronos bump.

* Address review comments.

* Attempt to fix finalization test issue.

* Fix nimbus_signing_node.

* Mark validators which are added at GENESIS_SLOT in GENESIS_EPOCH as passed doppelganger validation.

* Do not send empty requests to server.

* Fix log statement.

* Address review comments and re-raise cancellations.

Co-authored-by: zah <zahary@gmail.com>
2022-07-21 19:54:07 +03:00
Jacek Sieka c145916414
cleanups (#3819)
* avoid circular panda imports
* move deposit merkleization helpers to spec/
* normalize validator signature helpers to spec names / params
* remove redundant functions for remote signing
2022-06-29 18:53:59 +02:00
tersec b3d603f364
more CL spec URL updates to v1.2.0-rc.1 (#3657) 2022-05-24 08:26:35 +00:00
tersec c73239f60b
CL spec URL updates to v1.2.0-rc.1 (#3655) 2022-05-23 19:30:24 +00:00
zah 397033a7d6
Improve the compatibility with various Web3Signer versions and configurations (#3640)
* Some Web3Signer versions insist replying with text/plain messages
* When reading blocks, the Web3Signer uses upper-case fork identifiers
  instead of lower-case identifies like the Beacon API.
2022-05-17 19:55:03 +03:00
zah a2ba34f686
Implement all sync committee duties in the validator client (#3583)
Other changes:

* logtrace can now verify sync committee messages and contributions
* Many unnecessary use of pairs() have been removed for consistency
* Map 40x BN response codes to BeaconNodeStatus.Incompatible in the VC
2022-05-10 10:03:40 +00:00
zah 6d11ad6ce1
Support for distributed keystores with multiple remotes based on threshold signatures (#3616)
Other fixes:

* Fix bit rot in the `make prater-dev-deposit` target.
* Correct content-type in the responses of the Nimbus signing node
* Invalid JSON payload was being sent in the web3signer requests
2022-05-10 03:32:12 +03:00
tersec f0ada15dac
automated CL spec ref URL updates from v1.1.9 to v1.1.10 (#3455) 2022-03-02 10:00:21 +00:00
Jacek Sieka 1760f4d7a7
move wallet/deposit commands to separate files (#3372)
These commands have little to do with the "normal" beacon node operation
- ergo, they deserve to live in their own module.

* clean up imports/exports
2022-02-11 21:40:49 +01:00
Eugene Kabanov 40c77e5928
Remote KeyManager API and number of fixes/tests for KeyManager API (#3360)
* Initial commit.

* Fix current test suite.

* Fix keymanager api test.

* Fix wss_sim.

* Add more keystore_management tests.

* Recover deleted isEmptyDir().

* Add `HttpHostUri` distinct type.
Move keymanager calls away from rest_beacon_calls to rest_keymanager_calls.
Add REST serialization of RemoteKeystore and Keystore object.
Add tests for Remote Keystore management API.
Add tests for Keystore management API (Add keystore).
Fix serialzation issues.

* Fix test to use HttpHostUri instead of Uri.

* Add links to specification in comments.

* Remove debugging echoes.
2022-02-07 22:36:09 +02:00
tersec 29e2169585
phase 0 & altair beacon chain and altair validator spec URL updates (#3339) 2022-01-29 13:53:31 +00:00
tersec 89ffa8a1a7
spec URL & copyright year update (#3338) 2022-01-29 01:05:39 +00:00
Jacek Sieka 805e85e1ff
time: spring cleaning (#3262)
Time in the beacon chain is expressed relative to the genesis time -
this PR creates a `beacon_time` module that collects helpers and
utilities for dealing the time units - the new module does not deal with
actual wall time (that's remains in `beacon_clock`).

Collecting the time related stuff in one place makes it easier to find,
avoids some circular imports and allows more easily identifying the code
actually needs wall time to operate.

* move genesis-time-related functionality into `spec/beacon_time`
* avoid using `chronos.Duration` for time differences - it does not
support negative values (such as when something happens earlier than it
should)
* saturate conversions between `FAR_FUTURE_XXX`, so as to avoid
overflows
* fix delay reporting in validator client so it uses the expected
deadline of the slot, not "closest wall slot"
* simplify looping over the slots of an epoch
* `compute_start_slot_at_epoch` -> `start_slot`
* `compute_epoch_at_slot` -> `epoch`

A follow-up PR will (likely) introduce saturating arithmetic for the
time units - this is merely code moves, renames and fixing of small
bugs.
2022-01-11 11:01:54 +01:00
tersec ae61512ee9
rename upgrade_to_{merge,bellatrix}; detect unchanging spec YAMLs (#3265) 2022-01-10 09:39:43 +00:00
tersec 0fd8bf7b56
spec URL updates (#3254) 2022-01-06 18:35:38 +00:00
Zahary Karadjov 54d0d588b1 Implementation of the Keymanager API (BETA)
https://github.com/ethereum/keymanager-APIs
2022-01-04 18:51:45 +02:00
tersec da017d2ca5
update from phase0/altair v1.1.6 URLs to v1.1.8 spec URLs (#3238) 2022-01-04 03:57:15 +00:00
Jacek Sieka 069bccd51b
batch-verify sync messages for a small perf boost (#3151)
* batch-verify sync messages for a small perf boost

Generally reuses the same structure as attestation and aggregate
verification

* normalize `signatures` and `signature_batch` to use the same pattern
of verification
* normalize parameter names, order etc for signature stuff in general
* avoid calling `blsSign` directly - instead, go through `signatures`
consistently
2021-12-09 14:56:54 +02:00
tersec cc51f3fd12
v1.1.{5 -> 6} phase 0 and altair spec URL updates (#3157) 2021-12-03 17:40:23 +00:00
Eugene Kabanov e62c7c7c37
Remote signing client/server. (#3077) 2021-11-30 03:20:21 +02:00
tersec 9e395011d9
update 22 spec URLs to v1.1.5 (#3111) 2021-11-18 08:08:00 +00:00
tersec 2e868dc2ba
mass/mechanical update of 1.1.4 phase0 and altair spec URLs to 1.1.5 (#3067) 2021-11-09 07:40:41 +00:00
tersec eb3ad25859
mass/mechanical update of 1.1.3 phase/altair spec URLs to 1.1.4 (#3058) 2021-11-08 06:18:10 +00:00
tersec 36e37bda40
v1.1.3 spec refs URLs (#3036) 2021-10-27 18:40:17 +00:00
tersec 8307e9c601
mechanical non-merge v1.1.2 to v1.1.3 spec URL updates (#3030) 2021-10-26 16:44:23 +00:00
Jacek Sieka df3fc9525f
import cleanup (#2997)
* import cleanup

...and remove some unused types

* add random imports

* more imports
2021-10-19 16:09:26 +02:00
Jacek Sieka 4f7a8cf79d
register vc duties with subnet tracker (#2949)
* register vc duties with subnet tracker
* fix activation logging during startup
* cache slot signature to avoid duplicate signature work
* schedule aggregation duties one slot at a time to avoid CPU spike at
each epoch
* lower aggregation subnet pre-subscription time to 4 slots (lowers
bandwidth and CPU usage)
* update stability subnets in ENR on startup
* log gossip state
* perform gossip subscriptions just before the next slot starts
* document stuff
* add random include
* don't overwrite subscription state when not subscribed
* log target gossip state
* updating gossip status once is enough
* add test
* remove syncQueueLen - this one is not updated at the end of the sync
and may cause gossip to disconnect itself completely - use a simple head
distance instead
* fix gossip disconnection - if in hysteresis, node.gossipState will be
set to disabled even though we don't disable topic subscriptions
* fix extra duty registration call
2021-10-18 11:11:44 +02:00
tersec 2ad1b7366a
update 62 spec URLs to v1.1.2 (#2979) 2021-10-12 10:17:37 +00:00
tersec 0ae736f397
update 67 spec URLs to v1.1.2 (#2977) 2021-10-12 08:09:59 +00:00
Eugene Kabanov 65257b82f8
Validator key management API (#2755)
Implements https://github.com/ethereum/beacon-APIs/pull/151
2021-10-04 22:08:31 +03:00
tersec 47a6045b05
update a dozen spec URLs (#2929) 2021-09-29 23:43:28 +00:00
Jacek Sieka e47a8cbe42
fixes (#2901)
* export kvstore from beacon_chain_db
* fix rest HashList deserialization
* fix asTrusted
2021-09-27 11:24:58 +02:00
tersec 0ad7216bc4
mass update from beta.3 spec refs to beta.4 spec refs (#2862) 2021-09-10 18:56:03 +00:00
tersec 9e145afda3
update 29 Altair spec ref URLs to beta.3 (#2839) 2021-08-31 12:16:27 +00:00
tersec 092d9350de
eth2.0-specs -> consensus-specs repo rename (#2801) 2021-08-20 23:37:45 +00:00
tersec a0c518cb4f
sync committee/aggregate signature signing and verification (#2784)
* sync committee/aggregate signature signing and verification

* add message signature tests
2021-08-17 08:07:17 +00:00
Jacek Sieka 7a622e8505
rework spec imports (#2779)
The spec imports are a mess to work with, so this branch cleans them up
a bit to ensure that we avoid generic sandwitches and that importing
stuff generally becomes easier.

* reexport crypto/digest/presets because these are part of the public
symbol set of the rest of the spec types
* don't export `merge` types from `base` - this causes circular deps
* fix circular deps in `ssz/spec_types` - this is the first step in
disentangling ssz from spec
* be explicit about phase0 vs altair - longer term, `altair` will become
the "natural" type set, then merge and so on, so no point in giving
`phase0` special preferential treatment
2021-08-12 13:08:20 +00:00
Eugene Kabanov 3b6f4fab4a
New validator client using REST API. (#2651)
* Initial commit.

* Exporting getConfig().

* Add beacon node checking procedures.

* Post rebase fixes.

* Use runSlotLoop() from nimbus_beacon_node.
Fallback implementation.
Fixes for ETH2 REST serialization.

* Add beacon_clock.durationToNextSlot().
Move type declarations from beacon_rest_api to json_rest_serialization.
Fix seq[ValidatorIndex] serialization.
Refactor ValidatorPool and add some utility procedures.
Create separate version of validator_client.

* Post-rebase fixes.
Remove CookedPubKey from validator_pool.nim.

* Now we should be able to produce attestations and aggregate and proofs.
But its not working yet.

* Debugging attestation sending.

* Add durationToNextAttestation.
Optimize some debug logs.
Fix aggregation_bits encoding.
Bump chronos/presto.

* Its alive.

* Fixes for launch_local_testnet script.
Bump chronos.

* Switch client API to not use `/api` prefix.

* Post-rebase adjustments.

* Fix endpoint for publishBlock().

* Add CONFIG_NAME.
Add more checks to ensure that beacon_node is compatible.

* Add beacon committee subscription support to validator_client.

* Fix stacktrace should be an array of strings.
Fix committee subscriptions should not be `data` keyed.

* Log duration to next block proposal.

* Fix beacon_node_status import.

* Use jsonMsgResponse() instead of jsonError().

* Fix graffityBytes usage.
Remove unnecessary `await`.
Adjust creation of SignedBlock instance.
Remove legacy files.

* Rework durationToNextSlot() and durationToNextEpoch() to use `fromNow`.

* Fix race condition for block proposal and attestations for same slot.
Fix local_testnet script to properly kill tasks on Windows.
Bump chronos and nim-http-tools, to allow connections to infura.io (basic auth).

* Catch services errors.
Improve performance of local_testnet.sh script on Windows.
Fix race condition when attestation producing.

* Post-rebase fixes.

* Bump chronos and presto.

* Calculate block publishing delay.
Fix pkill in one more place.

* Add error handling and timeouts to firstSuccess() template.
Add onceToAll() template.
Add checkNodes() procedure.
Refactor firstSuccess() template.
Add error checking to api.nim calls.

* Deprecated usage onceToAll() for better stability.
Address comment and send attestations asap.

* Avoid unnecessary loop when calculating minimal duration.
2021-07-13 13:15:07 +02:00
tersec b1d5609171
remove false OnBlockAdded dependency on phase0 HashedBeaconState (#2661)
* remove false OnBlockAdded dependency on phase.HashedBeaconState

* introduce altair data types into block_clearance; update some alpha.6 spec refs to alpha.7; add get_active_validator_indices_len ForkedHashedBeaconState wrapper

* switch many modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 and remove rm'd G2_POINT_AT_INFINITY

* switch more modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7

* remove unnecessary phase0-only wrapper of get_attesting_indices(); allow signatures_batch to process either fork; remove O(n^2) nested loop in process_inactivity_updates(); add altair support to getAttestationsforTestBlock()

* add Altair versions of asSigVerified(), asTrusted(), and makeBeaconBlock()

* fix spec URL to be Altair for Altair makeBeaconBlock()
2021-06-21 08:35:24 +00:00
Jacek Sieka abe0d7b4ae singe validator key cache
Instead of keeping a validator key list per EpochRef, this PR introduces
a single shared validator key list in ChainDAG, and cleans up some other
ChainDAG and key-related issues.

The PR does not introduce the validator key list in the state transition
- this is because we batch-check all signatures before entering the spec
code, thus the spec code never hits the cache.

A future refactor should _probably_ remove the threadvar altogether.

There's a few other small fixes in here that make the flow easier to
read:

* fix `var ChainDAGRef` -> `ChainDAGRef`
* fix `var QuarantineRef` -> `QuarantineRef`
* consistent `dag` variable name
* avoid using threadvar pubkey cache in most cases
* better error messages in batch signature checking
2021-06-01 20:43:44 +03:00
Jacek Sieka 867d8f3223
Perform attestation check before broadcast (#2550)
Currently, we have a bit of a convoluted flow where when sending
attestations, we start broadcasting them over gossip then pass them to
the attestation validation to include them in the local attestation pool
- it should be the other way around: we should be checking attestations
_before_ gossipping them - this serves as an additional safety net to
ensure that we don't publish junk - this becomes more important when
publishing attestations from the API.

Also, the REST API was performing its own validation meaning
attestations coming from REST would be validated twice - finally, the
JSON RPC wasn't pre-validating and would happily broadcast invalid
attestations.

* Unified attestation production pipeline with the same flow for gossip,
locally and API-produced attestations: all are now validated and entered
into the pool, then broadcast/republished
* Refactor subnet handling with specific SubnetId alias, streamlining
where subnets are computed, avoiding the need to pass around the number
of active validators
* Move some of the subnet handling code to eth2_network
* Use BitArray throughout for subnet handling
2021-05-10 09:13:36 +02:00
Jacek Sieka 7dba1b37dd
remove attestation/aggregate queue (#2519)
With the introduction of batching and lazy attestation aggregation, it
no longer makes sense to enqueue attestations between the signature
check and adding them to the attestation pool - this only takes up
valuable CPU without any real benefit.

* add successfully validated attestations to attestion pool directly
* avoid copying participant list around for single-vote attestations,
pass single validator index instead
* release decompressed gossip memory earlier, specially during async
message validation
* use cooked signatures in a few more places to avoid reloads and errors
* remove some Defect-raising versions of signature-loading
* release decompressed data memory before validating message
2021-04-26 22:39:44 +02:00
Jacek Sieka 2695cfa864
EH cleanup (#2455)
almost 100% raises in nimbus-eth2 now!

* fix some rare exception-related crashes in json-rpc
2021-03-26 07:52:01 +01:00
Mamy Ratsimbazafy 3276dfc683
Consolidate modules by areas [part 1] (#2365)
* Move sync in subfolder

* move validator related thingies in validators

* fix binary builds

* update bounds comment [skip ci]
2021-03-02 11:27:45 +01:00