Commit Graph

287 Commits

Author SHA1 Message Date
zah 5402dfab6b
Correct URLs for the DeleteKeys request in the Keymanager API (#3727)
Other changes:

* Make it easier to run the REST tests locally through a Makefile target
2022-06-19 20:54:12 +03:00
Etan Kissling 61ee0611bb
adopt LC REST API with v0 suffix (without proofs) (#3775)
* adopt LC REST API with v0 suffix (without proofs)

Adopts the light client data REST API used by Lodestar as defined in
https://github.com/ethereum/beacon-APIs/pull/181 with a v0 suffix.

Requests:
- `/eth/v0/beacon/light_client/bootstrap/{block_root}`
- `/eth/v0/beacon/light_client/updates?start_period={start_period}&count={count}`
- `/eth/v0/beacon/light_client/finality_update`
- `/eth/v0/beacon/light_client/optimistic_update`

HTTP Server-Sent Events (SSE):
- `light_client_finality_update_v0`
- `light_client_optimistic_update_v0`

More work is needed to adopt the proofs endpoint, it is not included.

* initialize event queues

* register event topics
2022-06-19 08:57:52 +03:00
Eugene Kabanov 5e7b28cd66
AsyncEventBus missing attestations fix. (#3664) 2022-06-17 18:27:28 +03:00
tersec 27e1625d34
check for and log gossip broadcast failure (#3737)
* check for and log gossip broadcast failure

* switch notices to warns; update LC variables regardless

* don't both return a Result and log sending error

* add metrics counter for failed-due-to-no-peers and removed unnecessary async

* don't report failure of sync committee messages

* remove redundant metric

* document metric being incremented
2022-06-15 08:14:47 +00:00
Eugene Kabanov 1b6651dfc3
Fix /eth/v1/node/syncing (#3720)
* Fix REST `/eth/v1/node/syncing` call to return values even if SyncManager is not running.

* Use syncManager.inProgress as is_syncing indicator.
2022-06-14 22:26:23 +02:00
tersec 51885519a2
lc in sepolia (#3734) 2022-06-11 01:36:45 +00:00
tersec 1dec3ff8b6
remove unused stdlib imports (#3718) 2022-06-09 08:50:36 +00:00
Tanguy 29297e9ce4
Bump libp2p (#3709) 2022-06-08 07:53:50 +02:00
tersec faf4d4a001
initial Capella support in RuntimeConfig (#3698) 2022-06-03 14:42:40 +00:00
Eugene Kabanov 50f9596108
Eliminate rpc_types.nim usage. (#3692) 2022-06-02 09:39:08 +00:00
Jacek Sieka 48f01186d6
fix unnecessary HashList/HashArray cache invalidation (#3660)
* SSZ `[]` -> `mitem`
* `[]` -> `item`

immutable access via mutable instance cannot rely on template
overloading, and `[]` cannot be a `func` because of special seq handling
in compiler.
2022-05-30 13:30:42 +00:00
tersec b3d603f364
more CL spec URL updates to v1.2.0-rc.1 (#3657) 2022-05-24 08:26:35 +00:00
tersec d35d408fa7
remove deprecated JSON-RPC server (#3656)
* remove deprecated JSON-RPC server

* keep the command-line options around as no-ops, temporarily

* service -> server; JSON-RPC is still used elsewhere
2022-05-24 07:23:48 +00:00
tersec c73239f60b
CL spec URL updates to v1.2.0-rc.1 (#3655) 2022-05-23 19:30:24 +00:00
zah a0a6dd2f63
Add a ncli tool for converting a regular keystore into a distributed one (#3634) 2022-05-17 16:50:49 +03:00
zah a2ba34f686
Implement all sync committee duties in the validator client (#3583)
Other changes:

* logtrace can now verify sync committee messages and contributions
* Many unnecessary use of pairs() have been removed for consistency
* Map 40x BN response codes to BeaconNodeStatus.Incompatible in the VC
2022-05-10 10:03:40 +00:00
zah 6d11ad6ce1
Support for distributed keystores with multiple remotes based on threshold signatures (#3616)
Other fixes:

* Fix bit rot in the `make prater-dev-deposit` target.
* Correct content-type in the responses of the Nimbus signing node
* Invalid JSON payload was being sent in the web3signer requests
2022-05-10 03:32:12 +03:00
Etan Kissling fec933432d
update presets / configs / REST config for v1.1.10 (#3614)
Updated outdated presets / configs / REST config to v1.1.10 specs.

- `TERMINAL_BLOCK_HASH_ACTIVATION_EPOCH` and `PROPOSER_SCORE_BOOST` are
  not yet used in `eth2-networks`, added configurability as TODOs.

- `MIN_ANCHOR_POW_BLOCK_DIFFICULTY` is no longer needed, put on ignore
  list as some Altair devnets still reference it.
2022-05-06 09:51:28 +02:00
Jacek Sieka f70ff38b53
enable `styleCheck:usages` (#3573)
Some upstream repos still need fixes, but this gets us close enough that
style hints can be enabled by default.

In general, "canonical" spellings are preferred even if they violate
nep-1 - this applies in particular to spec-related stuff like
`genesis_validators_root` which appears throughout the codebase.
2022-04-08 16:22:49 +00:00
Etan Kissling 2c5f725543
reformat long lines (#3539)
Shortens some long lines by introducing temp variables and line breaks.
2022-03-29 09:15:42 +02:00
Jacek Sieka bc80ac3be1
harden REST API `atSlot` against non-finalized blocks (#3538)
* harden validator API against pre-finalized slot requests
* check `syncHorizon` when responding to validator api requests too far
from `head`
* limit state-id based requests to one epoch ahead of `head`
* put historic data bounds on block/attestation/etc validator production API, preventing them from being used with already-finalized slots
* add validator block smoke tests
* make rest test create a new genesis with the tests running roughly in
the first epoch to allow testing a few more boundary conditions
2022-03-23 12:42:16 +01:00
Jacek Sieka 05ffe7b2bf
Prune `BlockRef` on finalization (#3513)
Up til now, the block dag has been using `BlockRef`, a structure adapted
for a full DAG, to represent all of chain history. This is a correct and
simple design, but does not exploit the linearity of the chain once
parts of it finalize.

By pruning the in-memory `BlockRef` structure at finalization, we save,
at the time of writing, a cool ~250mb (or 25%:ish) chunk of memory
landing us at a steady state of ~750mb normal memory usage for a
validating node.

Above all though, we prevent memory usage from growing proportionally
with the length of the chain, something that would not be sustainable
over time -  instead, the steady state memory usage is roughly
determined by the validator set size which grows much more slowly. With
these changes, the core should remain sustainable memory-wise post-merge
all the way to withdrawals (when the validator set is expected to grow).

In-memory indices are still used for the "hot" unfinalized portion of
the chain - this ensure that consensus performance remains unchanged.

What changes is that for historical access, we use a db-based linear
slot index which is cache-and-disk-friendly, keeping the cost for
accessing historical data at a similar level as before, achieving the
savings at no percievable cost to functionality or performance.

A nice collateral benefit is the almost-instant startup since we no
longer load any large indicies at dag init.

The cost of this functionality instead can be found in the complexity of
having to deal with two ways of traversing the chain - by `BlockRef` and
by slot.

* use `BlockId` instead of `BlockRef` where finalized / historical data
may be required
* simplify clearance pre-advancement
* remove dag.finalizedBlocks (~50:ish mb)
* remove `getBlockAtSlot` - use `getBlockIdAtSlot` instead
* `parent` and `atSlot` for `BlockId` now require a `ChainDAGRef`
instance, unlike `BlockRef` traversal
* prune `BlockRef` parents on finality (~200:ish mb)
* speed up ChainDAG init by not loading finalized history index
* mess up light client server error handling - this need revisiting :)
2022-03-17 17:42:56 +00:00
Jacek Sieka c64bf045f3
remove StateData (#3507)
One more step on the journey to reduce `BlockRef` usage across the
codebase - this one gets rid of `StateData` whose job was to keep track
of which block was last assigned to a state - these duties have now been
taken over by `latest_block_root`, a fairly recent addition that
computes this block root from state data (at a small cost that should be
insignificant)

99% mechanical change.
2022-03-16 08:20:40 +01:00
Jacek Sieka a3bd01b58d
move dependent root computations to `BeaconState` / `EpochRef` (#3478)
* fewer deps on `BlockRef` traversal in anticipation of pruning
* allows identifying EpochRef:s by their shuffling as a first step of
* tighten error handling around missing blocks

using the zero hash for signalling "missing block" is fragile and easy
to miss - with checkpoint sync now, and pruning in the future, missing
blocks become "normal".
2022-03-15 09:24:55 +01:00
Mamy Ratsimbazafy 9fd7305e26
Cleanup RPC pubkey handling (#3489) 2022-03-13 08:12:45 +01:00
Jacek Sieka 4363215a32
relax `BlockRef` database assumptions (#3472)
* remove `getForkedBlock(BlockRef)` which assumes block data exists but
doesn't support archive/backfilled blocks
* fix REST `/eth/v1/beacon/headers` request not returning
archive/backfilled blocks
* avoid re-encoding in REST block SSZ requests (using `getBlockSSZ`)
2022-03-11 13:08:17 +01:00
Zahary Karadjov 5ef2ce4069
Fix #3463 (validator index-out-of-bound errors triggered through the REST API) 2022-03-07 21:49:12 +02:00
tersec c18cd8ee0c
rename random -> prev_randao in Bellatrix for CL specs v1.1.10 (#3460) 2022-03-03 16:08:14 +00:00
Zahary Karadjov 4c01b77773
The remote Keymanager API was not using the URLs indicated in the spec 2022-03-03 11:10:00 +02:00
tersec f0ada15dac
automated CL spec ref URL updates from v1.1.9 to v1.1.10 (#3455) 2022-03-02 10:00:21 +00:00
Etan Kissling 9790c4958b
converter function for reducing blocks to headers (#3410)
This introduces a function to convert `SignedBeaconBlock` to just their
`BeaconBlockHeader` and updates the usages for reduced code duplication.
2022-02-18 21:35:52 +01:00
Zahary Karadjov 922a0d264c Add CORS support for the REST services
The added options work in opt-in fashion. If they are not specified,
the server will respond to all requests as if the CORS specification
doesn't exist. This will result in errors in CORS-enabled clients.

Please note that future versions may support more than one allowed
origin. The option names will stay the same, but the user will be
able to repeat them on the command line (similar to other options
such as --web3-url).

To be documented in the guide in a separate PR.
2022-02-14 18:52:17 +02:00
Eugene Kabanov 1a0bcf0b02
Fix #3267 (#3367)
* Initial commit.

* One more fix.

* Trying to debug the finalization issue.

* Add debug logs to understand signature issue.

* Restore hash_tree_root calculation.

* Remove all the debugging helpers.

* Add `slot` check.

* Address review comment.
2022-02-13 16:21:55 +01:00
Eugene Kabanov 40c77e5928
Remote KeyManager API and number of fixes/tests for KeyManager API (#3360)
* Initial commit.

* Fix current test suite.

* Fix keymanager api test.

* Fix wss_sim.

* Add more keystore_management tests.

* Recover deleted isEmptyDir().

* Add `HttpHostUri` distinct type.
Move keymanager calls away from rest_beacon_calls to rest_keymanager_calls.
Add REST serialization of RemoteKeystore and Keystore object.
Add tests for Remote Keystore management API.
Add tests for Keystore management API (Add keystore).
Fix serialzation issues.

* Fix test to use HttpHostUri instead of Uri.

* Add links to specification in comments.

* Remove debugging echoes.
2022-02-07 22:36:09 +02:00
tersec 8e6a920bf4
rename MERGE_FORK_EPOCH to BELLATRIX_FORK_EPOCH (#3350)
* rename MERGE_FORK_EPOCH to BELLATRIX_FORK_EPOCH

* fix REST test rules
2022-02-02 14:06:55 +01:00
tersec c9aa1bee01
spec URL updates (#3342) 2022-01-31 09:56:59 +00:00
Emil 0051af430b Put `application/json` as a higher preference than `application/octet-stream` 2022-01-30 18:50:14 +02:00
tersec 29e2169585
phase 0 & altair beacon chain and altair validator spec URL updates (#3339) 2022-01-29 13:53:31 +00:00
tersec 89ffa8a1a7
spec URL & copyright year update (#3338) 2022-01-29 01:05:39 +00:00
cheatfate 1287a20b13 Use HTTP status codes instead of status in body. 2022-01-28 15:36:27 +02:00
tersec dcb671617c
add/support TERMINAL_BLOCK_HASH_ACTIVATION_EPOCH (#3303) 2022-01-27 19:52:08 +00:00
tersec 00a347457a
dynamic sync committee subscriptions (#3308)
* dynamic sync committee subscriptions

* fast-path trivial case rather than rely on RNG with probability 1 outcome

Co-authored-by: zah <zahary@gmail.com>

* use func instead of template; avoid calling async function unnecessarily

* avoid unnecessary sync committee topic computation; use correct epoch lookahead; enforce exception/effect tracking

* don't over-optimistically update ENR syncnets; non-looping version of nearSyncCommitteePeriod

* allow separately setting --allow-all-{sub,att,sync}nets

* remove unnecessary async

Co-authored-by: zah <zahary@gmail.com>
2022-01-24 20:40:59 +00:00
Jacek Sieka 61342c2449
limit by-root requests to non-finalized blocks (#3293)
* limit by-root requests to non-finalized blocks

Presently, we keep a mapping from block root to `BlockRef` in memory -
this has simplified reasoning about the dag, but is not sustainable with
the chain growing.

We can distinguish between two cases where by-root access is useful:

* unfinalized blocks - this is where the beacon chain is operating
generally, by validating incoming data as interesting for future fork
choice decisions - bounded by the length of the unfinalized period
* finalized blocks - historical access in the REST API etc - no bounds,
really

In this PR, we limit the by-root block index to the first use case:
finalized chain data can more efficiently be addressed by slot number.

Future work includes:

* limiting the `BlockRef` horizon in general - each instance is 40
bytes+overhead which adds up - this needs further refactoring to deal
with the tail vs state problem
* persisting the finalized slot-to-hash index - this one also keeps
growing unbounded (albeit slowly)

Anyway, this PR easily shaves ~128mb of memory usage at the time of
writing.

* No longer honor `BeaconBlocksByRoot` requests outside of the
non-finalized period - previously, Nimbus would generously return any
block through this libp2p request - per the spec, finalized blocks
should be fetched via `BeaconBlocksByRange` instead.
* return `Opt[BlockRef]` instead of `nil` when blocks can't be found -
this becomes a lot more common now and thus deserves more attention
* `dag.blocks` -> `dag.forkBlocks` - this index only carries unfinalized
blocks from now - `finalizedBlocks` covers the other `BlockRef`
instances
* in backfill, verify that the last backfilled block leads back to
genesis, or panic
* add backfill timings to log
* fix missing check that `BlockRef` block can be fetched with
`getForkedBlock` reliably
* shortcut doppelganger check when feature is not enabled
* in REST/JSON-RPC, fetch blocks without involving `BlockRef`

* fix dag.blocks ref
2022-01-21 13:33:16 +02:00
Dustin Brody 9699858422 rename MERGE_FORK_VERSION to BELLATRIX_FORK_VERSION 2022-01-20 19:33:05 +02:00
Jacek Sieka 4e2d2ff7f4 rest: fix invalid type `RestSyncCommitteeSubscription`
Using the wrong type here causes requests to fail due to the overly
zealous parameter validation - the failure is harmless in the current
duty subscription model, but would have caused more serious failures
down the line.
2022-01-17 22:33:24 +02:00
Jacek Sieka 836f6984bb
move `state_transition` to `Result` (#3284)
* better error messages in api
* avoid `BlockData` copies when replaying blocks
2022-01-17 12:19:58 +01:00
Jacek Sieka e9486f5e5b
state_sim: clean up attestation production (#3274)
* use same naming as everywhere
* avoid iterator bug that leads to state copy
2022-01-12 21:42:03 +01:00
Jacek Sieka 805e85e1ff
time: spring cleaning (#3262)
Time in the beacon chain is expressed relative to the genesis time -
this PR creates a `beacon_time` module that collects helpers and
utilities for dealing the time units - the new module does not deal with
actual wall time (that's remains in `beacon_clock`).

Collecting the time related stuff in one place makes it easier to find,
avoids some circular imports and allows more easily identifying the code
actually needs wall time to operate.

* move genesis-time-related functionality into `spec/beacon_time`
* avoid using `chronos.Duration` for time differences - it does not
support negative values (such as when something happens earlier than it
should)
* saturate conversions between `FAR_FUTURE_XXX`, so as to avoid
overflows
* fix delay reporting in validator client so it uses the expected
deadline of the slot, not "closest wall slot"
* simplify looping over the slots of an epoch
* `compute_start_slot_at_epoch` -> `start_slot`
* `compute_epoch_at_slot` -> `epoch`

A follow-up PR will (likely) introduce saturating arithmetic for the
time units - this is merely code moves, renames and fixing of small
bugs.
2022-01-11 11:01:54 +01:00
Jacek Sieka 20e700fae4
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259)
* Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex

Harden the use of `CommitteeIndex` et al to prevent future issues by
using a distinct type, then validating before use in several cases -
datatypes in spec are kept simple though so that invalid data still can
be read.

* fix invalid epoch used in REST
`/eth/v1/beacon/states/{state_id}/committees` committee length (could
return invalid data)
* normalize some variable names
* normalize committee index loops
* fix `RestAttesterDuty` to use `uint64` for `validator_committee_index`
* validate `CommitteeIndex` on ingress in REST API
* update rest rules with stricter parsing
* better REST serializers
* save lots of memory by not using `zip` ...at least a few bytes!
2022-01-09 01:28:49 +02:00
Jacek Sieka 6f7e0e3393
REST cleanups (#3255)
* REST cleanups

* reject out-of-range committee requests
* print all hex values as lower-case
* allow requesting state information by head state root
* turn `DomainType` into array (follow spec)
* `uint_to_bytesXX` -> `uint_to_bytes` (follow spec)
* fix wrong dependent root in `/eth/v1/validator/duties/proposer/`
* update documentation - `--subscribe-all-subnets` is no longer needed
when using the REST interface with validator clients
* more fixes
* common helpers for dependent block
* remove test rules obsoleted by more strict epoch tests
* fix trailing commas

* Update docs/the_nimbus_book/src/rest-api.md
* Update docs/the_nimbus_book/src/rest-api.md

Co-authored-by: sacha <sacha@status.im>
2022-01-08 22:06:34 +02:00
Jacek Sieka ba99c8fe4f
update era file documentation / impl (#3226)
Overhaul of era files, including documentation and reference
implementations

* store blocks, then state, then slot indices for easy lookup at low
cost
* document era file rationale
* altair+ support in era writer
2022-01-07 11:13:19 +01:00
Emil 2a12d1c49f Support more content-types when specifying the Graffiti string 2022-01-06 10:56:59 +02:00
Jacek Sieka 0e2b4e39fa
REST JSON support improvements (#3232)
* support downloading blocks / states via JSON in addition to SSZ -
slow, but needed for infura support - SSZ is still used when server
supports it
* use common forked block/state reader in REST API
* fix stack overflows in REST JSON decoder
* fix invalid serialization of `justification_bits` in
`/eth/v1/debug/beacon/states` and `/eth/v2/debug/beacon/states`
* fix REST client to use `/eth/...` instead of `/api/eth/...`, update
"default" urls to expose REST api via `/eth` as well as this is what the
standard says - `/api` was added early on based on an example "base url"
in the spec that has been removed since
* expose Nimbus REST extensions via `/nimbus` in addition to
`/api/nimbus` to stay consistent with `/eth`
* fix invalid state root when reading states via REST
* fix recursive imports in `spec/ssz_codec`
* remove usages of `serialization.useCustomSerialization` - fickle
2022-01-06 08:38:40 +01:00
Jacek Sieka 0a4728a241
Handle access to historical data for which there is no state (#3217)
With checkpoint sync in particular, and state pruning in the future,
loading states or state-dependent data may fail. This PR adjusts the
code to allow this to be handled gracefully.

In particular, the new availability assumption is that states are always
available for the finalized checkpoint and newer, but may fail for
anything older.

The `tail` remains the point where state loading de-facto fails, meaning
that between the tail and the finalized checkpoint, we can still get
historical data (but code should be prepared to handle this as an
error).

However, to harden the code against long replays, several operations
which are assumed to work only with non-final data (such as gossip
verification and validator duties) now limit their search horizon to
post-finalized data.

* harden several state-dependent operations by logging an error instead
of introducing a panic when state loading fails
* `withState` -> `withUpdatedState` to differentiate from the other
`withState`
* `updateStateData` can now fail if no state is found in database - it
is also hardened against excessively long replays
* `getEpochRef` can now fail when replay fails
* reject blocks with invalid target root - they would be ignored
previously
* fix recursion bug in `isProposed`
2022-01-05 19:38:04 +01:00
zah fba1f08a5e
Implement #3129 (Optimized history traversals in the REST API) (#3219)
* Fix REST some rest call signatures and implement a simple API benchmark tool

* Implement #3129 (Optimized history traversals in the REST API)

Other notable changes:

The `updateStateData` procedure in the `blockchain_dag.nim` module is
optimized to not rewind down to the last snapshot state saved in the
database if the supplied input state can be used as a starting point
instead.

* Disallow await in withStateForBlockSlot
2022-01-05 15:49:10 +01:00
tersec 66c9b7fbce
shift block_sim fork epochs; allow VC to work with non-multiple-of-3 SECONDS_PER_SLOT (#3244) 2022-01-05 13:41:39 +00:00
Zahary Karadjov 54d0d588b1 Implementation of the Keymanager API (BETA)
https://github.com/ethereum/keymanager-APIs
2022-01-04 18:51:45 +02:00
tersec b81c06edab
rename Beacon{Block,State}Fork.Merge to Bellatrix; update copyright years (#3240) 2022-01-04 09:45:38 +00:00
tersec da017d2ca5
update from phase0/altair v1.1.6 URLs to v1.1.8 spec URLs (#3238) 2022-01-04 03:57:15 +00:00
tersec d4680df8d2
convert between engine and consensus ExecutionPayloads (#3228)
* convert between engine and consensus ExecutionPayloads
2022-01-03 13:22:56 +01:00
Jacek Sieka c270ec21e4
Validator monitoring (#2925)
Validator monitoring based on and mostly compatible with the
implementation in Lighthouse - tracks additional logs and metrics for
specified validators so as to stay on top on performance.

The implementation works more or less the following way:
* Validator pubkeys are singled out for monitoring - these can be
running on the node or not
* For every action that the validator takes, we record steps in the
process such as messages being seen on the network or published in the
API
* When the dust settles at the end of an epoch, we report the
information from one epoch before that, which coincides with the
balances being updated - this is a tradeoff between being correct
(waiting for finalization) and providing relevant information in a
timely manner)
2021-12-20 20:20:31 +01:00
tersec 6ef3834f4a
fix type-conversions-to-self, unexport from nimbus_beacon_node, and rm unused vars/procs (#3211) 2021-12-20 12:21:17 +01:00
Jacek Sieka 9f27f0d97c
BlockId reform (#3176)
* BlockId reform

Introduce `BlockId` that helps track a root/slot pair - this prepares
the codebase for backfilling and handling out-of-dag blocks

* move block dag code to separate module
* fix finalised state root in REST event stream
* fix finalised head computation on head update, when starting from
checkpoint
* clean up chaindag init
* revert `epochAncestor` change in introduced in #3144 that would return
an epoch ancestor from the canoncial history instead of the given
history, causing `EpochRef` keys to point to the wrong block
2021-12-09 19:06:21 +02:00
Jacek Sieka 89d6a1b403
Introduce slot->BlockRef mapping for finalized chain (#3144)
* Introduce slot->BlockRef mapping for finalized chain

The finalized chain is linear, thus we can use a seq to lookup blocks by
slot number.

Here, we introduce such a seq, even though in the future, it should
likely be backed by a database structure instead, or, more likely, a
flat era file with a flat lookup index.

This dramatically speeds up requests by slot, such as those coming from
the REST interface or GetBlocksByRange, as these are currently served by
a linear iteration from head.

* fix REST block requests to not return blocks from an earlier slot when
the given slot is empty
* fix StateId interpretation such that it doesn't treat state roots as
block roots
* don't load full block from database just to return its root
2021-12-06 20:52:35 +02:00
Jacek Sieka 850eece949 Broadcast blocks before integrating in ChainDAG
This PR fixes two issues with block publishing:

* Gossip-valid blocks are published before integrating them into the
chain, giving broadcasting a head start, both for rest block and
* Outright invalid blocks from the API that could lead to the descoring
of the node are no longer broadcast

Bonus:

* remove undocumented and duplicated `post_v1_validator_block` JSON-RPC
call
2021-12-06 17:15:47 +02:00
tersec cc51f3fd12
v1.1.{5 -> 6} phase 0 and altair spec URL updates (#3157) 2021-12-03 17:40:23 +00:00
zah 74c63ed740
More efficient implementation of the 'POST beacon_committee_subscriptions' API (#3153) 2021-12-03 17:04:58 +02:00
cheatfate 40a9da385d Fix for /eth/v1/node/identity call missing `syncnets` parameter. 2021-12-01 18:39:43 +02:00
Eugene Kabanov e62c7c7c37
Remote signing client/server. (#3077) 2021-11-30 03:20:21 +02:00
zah 3aa804035f
Allow /api/eth/v1/validator/duties/sync/{epoch} to be called for epochs in the next sync committee period (#3133) 2021-11-30 03:14:31 +02:00
Jacek Sieka b22d86e161
REST/JSON-RPC: speed up several requests (#3092)
REST/JSON-RPC and a few more also invalidate caches unnecessarily,
similar to https://github.com/status-im/nimbus-eth2/pull/3089

* avoid copying validator on balance request
2021-11-12 23:29:28 +01:00
Jacek Sieka ec650c7fd7
Support starting from altair (#3054)
* Support starting from altair

* hide `finalized-checkpoint-` - they are incomplete and usage may cause
crashes
* remove genesis detection code (broken, obsolete)
* enable starting ChainDAG from altair checkpoints - this is a
prerequisite for checkpoint sync (TODO: backfill)
* tighten checkpoint state conditions
* show error when starting from checkpoint with existing database (not
supported)
* print rest-compatible JSON in ncli/state_sim
* altair/merge support in ncli
* more altair/merge support in ncli_db
* pre-load header to speed up loading
* fix forked block decoding
2021-11-10 13:39:08 +02:00
zah 3545d4d1e1
Address review comments in #3057 (#3069)
* Address review comments in #3057

* reorder imports in rest_utils

maybe this will help with the mysterious serialization issues

Co-authored-by: Jacek Sieka <jacek@status.im>
2021-11-09 20:21:36 +01:00
Zahary Karadjov ba42b2b316 Correct implementation of the /validator/duties/sync/{epoch} API
According to the spec, this call should return the positions of
the specified validators within the sync committee. The existing
code was instead returning the indices of the sync sub-committees
where the validator is a member.
2021-11-09 11:45:00 +02:00
tersec 2e868dc2ba
mass/mechanical update of 1.1.4 phase0 and altair spec URLs to 1.1.5 (#3067) 2021-11-09 07:40:41 +00:00
tersec 5c296c08b7
17 spec URL updates to v1.1.4 (#3060) 2021-11-08 10:30:52 +00:00
tersec eb3ad25859
mass/mechanical update of 1.1.3 phase/altair spec URLs to 1.1.4 (#3058) 2021-11-08 06:18:10 +00:00
Jacek Sieka ea0a191723
Better REST/RPC error messages (#3046)
* Better REST/RPC error messages
* homogenise block logging (root first)
* homegenise message verification pipeline (verify in
`gossip_verification`, act in `eth2_processor`)
* use `subcommitteeIdx` consistently
* log each sent contribution
* fix block_sim
* fix block topic
* don't recalc root on gossip block validation
* move position loop into sync pool
2021-11-05 17:39:47 +02:00
Jacek Sieka a086cf01ac
altair fork handling cleanups (#3050)
* fix stack overflow crash in REST/debug/getStateV2
* introduce `ForkyXxx` for generic type matching of `Xxx` across
branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) -
`Some` is already used for other types of type classes
* consolidate function naming in BeaconChainDB, use some generics
* import `forks.nim` from other spec modules and move `Forked*` helpers
around to resolve circular imports
* remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout
(less data shuffling between the types)
* fix several cases of states being stored on stack in tests, causing
random failures on some platforms
* remove reading json support from ncli - this should be ported to the
rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 08:34:34 +01:00
Etan Kissling 1943258c66
extend REST config with more spec constants (#2962)
So far, the REST config response did not include all spec constants.
The spec for `/eth/v1/config/spec` defines that the response should
include constants for all hard forks known by the beacon node. This
patch extends the corresponding response to include more constants.
2021-11-02 19:23:31 +01:00
Eugene Kabanov 7c9a6b7170
Improve chronos.Future tracking. (#2988)
* Add `child_id` field.

* Fix json-rpc api call and bump chronos.

* Bump chronos master and fix compilation warnings.

* One more bump of `chronos`.

* add random import

* export rest_utils a bit more

Co-authored-by: Jacek Sieka <jacek@status.im>
2021-10-27 14:01:11 +02:00
Eugene Kabanov 7c49d657a6
Fix regression in REST validators and validator_balances requests. (#3022)
* Fix state validators and validator_balances responses when validator's identifier could not be found in state.

* Add tests.
2021-10-25 15:52:00 +02:00
Jacek Sieka 421bf936ff
odds and ends (#3015)
* `allSyncCommittees` => `allSyncSubcommittees`
* simplify `_snappy` topic generation (avoid pointless string copies)
* simplify gossip id generator (avoid pointless string copies)
* avoid redundant syncnet ENR updates
* simplify topic validation (allow only validated topics)
2021-10-21 15:09:19 +02:00
nbc-bump-bot[bot] 29b26f3f6b
nim-libp2p auto bump (#2840)
* auto-bump nim-libp2p

* Remove peer info for other peers

Not definitive, just to test the libp2p's unstable branch

* finish up Remove peer info for other peers

* getKey -> getPublicKey

* bump libp2p

* libp2p bump

Co-authored-by: = <Menduist@users.noreply.github.com>
Co-authored-by: Tanguy <tanguy@status.im>
2021-10-21 13:01:29 +02:00
Jacek Sieka 9cf32c3748 clean up sync subcommittee handling
* `SyncCommitteeIndex` -> `SyncSubcommitteeIndex`
* `syncCommitteePeriod` -> `sync_committee_period` (spec spelling)
* tighten period comparisons
* fix assert when validating committee message with non-altair state in
REST api
2021-10-20 22:59:13 +03:00
Jacek Sieka bf6ad41d7d add drop and sync committee metrics
* use storeBlock for processing API blocks
* avoid double block dump
* count all gossip metrics at the same spot
* simplify block broadcast
2021-10-20 18:20:12 +03:00
Jacek Sieka c247702ebc normalize subnet logging
* call it subnet id everywhere
* log aggregate sent from VC
* log subnet with aggregate
2021-10-20 15:06:44 +03:00
Jacek Sieka df3fc9525f
import cleanup (#2997)
* import cleanup

...and remove some unused types

* add random imports

* more imports
2021-10-19 16:09:26 +02:00
Jacek Sieka c40cc6cec1 clean up fork enum and field names
* single naming strategy
* simplify some fork code
* simplify forked block production
2021-10-19 11:06:38 +03:00
Jacek Sieka 4f7a8cf79d
register vc duties with subnet tracker (#2949)
* register vc duties with subnet tracker
* fix activation logging during startup
* cache slot signature to avoid duplicate signature work
* schedule aggregation duties one slot at a time to avoid CPU spike at
each epoch
* lower aggregation subnet pre-subscription time to 4 slots (lowers
bandwidth and CPU usage)
* update stability subnets in ENR on startup
* log gossip state
* perform gossip subscriptions just before the next slot starts
* document stuff
* add random include
* don't overwrite subscription state when not subscribed
* log target gossip state
* updating gossip status once is enough
* add test
* remove syncQueueLen - this one is not updated at the end of the sync
and may cause gossip to disconnect itself completely - use a simple head
distance instead
* fix gossip disconnection - if in hysteresis, node.gossipState will be
set to disabled even though we don't disable topic subscriptions
* fix extra duty registration call
2021-10-18 11:11:44 +02:00
Eugene Kabanov 073389377e
Allow duplicate validator identifiers in REST requests. (#2992)
Missing validators are not error anymore.
Fix tests.
2021-10-18 10:54:20 +02:00
Eugene Kabanov 7319f150e8
Number of REST fixes. (#2987)
* Initial commit.

* Fix path.

* Add validator keys to indices cache mechanism.
Move syncComitteeParticipants to common place.

* Fix sync participants order issue.

* Fix error code when state could not be found.
Refactor `state/validators` to use keysToIndices mechanism.

* Fix RestValidatorIndex to ValidatorIndex conversion TODOs.

* Address review comments.

* Fix REST test rules.
2021-10-14 12:38:38 +02:00
Jacek Sieka 1da4192f3f
json-rpc: fix crashes on altair+ blocks in certain operations (#2983)
For the v2 interfaces, REST is recommeded instead
2021-10-13 13:20:18 +03:00
Etan Kissling 9ee134324b
allow `withXxx` to access fork-specific fields (#2943)
So far, `withState` and `withBlck` templates could only be used to have
convenience access to fork-agnostic BeaconState and BeaconBlock fields.
This patch:
- injects an additional `stateFork` constant that allows to use
  `when` expressions to also access Altair and Merge-specific fields.
- introduces a `withStateAndBlck` template to support operating on both
  a `BeaconState` and `BeaconBlock` at a time.
- makes sync committee related functions Merge aware.
- changes a couple if-else trees for forks into case statements so that
  forgotten future forks are promoted to compile-time errors.
2021-10-06 20:05:06 +03:00
Eugene Kabanov 0fffdaf96d
Number of REST fixes. (#2961)
* Fix /eth/v1/config/spec response.
Cache /eth/v1/node/version response.

* Fix REST test rule.

* Workaround for `state_id` issue.

* Fix /eth/v1/events `head`, `chain_reorg` and `finalized_checkpoint` responses.

* Fix client configuration object.
2021-10-06 15:43:03 +02:00
Eugene Kabanov 65257b82f8
Validator key management API (#2755)
Implements https://github.com/ethereum/beacon-APIs/pull/151
2021-10-04 22:08:31 +03:00
Zahary Karadjov 66b9f33edd Definitions related to the new execution engine JSON-RPC API 2021-09-30 00:20:21 +03:00
Eugene Kabanov 1db6a92c4f
Fix double "data" enclosure in /eth/v1/beacon/states/{state_id}/sync_committees response. (#2908) 2021-09-28 20:08:23 +02:00
Etan Kissling c510ffb16a
use `allSyncCommittees` everywhere (#2917)
There are still a few cases with manual loops through sync subcommittees
even though an `allSyncCommittees` iterator exists. Adjusted the few
remaining instances to also use the iterator instead.
2021-09-28 18:02:01 +00:00
tersec 2b2846b468
implement forked merge state/block support (#2890)
* implement forked state/block support

* merge support for containsOrphan; import cleanup; 80-column lines

* add merge block header operations and slot sanity fixture

* add epoch state transition tests; implement is_valid_gas_limit(), is_merge_block(), is_execution_enabled(), and compute_timestamp_at_slot()

* implement process_execution_payload() and add merge deposit operations tests

* add merge block sanity tests

* add merge case to syncCommitteeParticipants

* v1.1.0-beta.5 updates

* reduce getTestStates-based memory usage; don't try to REST-serialize ExecutionPayload transactions without underlying support

* add execution payload tests; switch var to let in tests/official/
2021-09-27 14:22:58 +00:00