Commit Graph

80 Commits

Author SHA1 Message Date
Etan Kissling 3a4b87358a
update to v1 of LC REST API (#4232)
The LC REST API has been merged into the ethereum/beacon-APIs specs:
- https://github.com/ethereum/beacon-APIs/pull/247

Update URLs to v1 and update REST tests. Note that REST tests do not
start with Altair, so the tested BN will return empty / error responses.
2022-10-13 02:16:49 +02:00
Etan Kissling 7202acddff
update for latest LC REST proposal (#4213)
Implements the latest proposal for providing LC data via REST, as of
https://github.com/ethereum/beacon-APIs/pull/247 with a v0 suffix.

Requests:
- `/eth/v0/beacon/light_client/bootstrap/{block_root}`
- `/eth/v0/beacon/light_client/updates?start_period={start_period}&count={count}`
- `/eth/v0/beacon/light_client/finality_update`
- `/eth/v0/beacon/light_client/optimistic_update`

HTTP Server-Sent Events (SSE):
- `light_client_finality_update_v0`
- `light_client_optimistic_update_v0`
2022-10-04 13:38:09 +02:00
Etan Kissling 83cd104d07
add section headers for REST serialization (#4210)
Adds missing REST serialization sections and makes them consistent.
2022-10-03 22:29:07 +00:00
Etan Kissling 231eb8a590
use same style for sending block and state rsps (#4209)
For JSON responses, "eth-consensus-version" header is handled in
`eth2_rest_serialization` for states and `rest_beacon_api` for blocks.
Align them to also be handled in `eth2_rest_serialization` for blocks.
2022-10-04 01:05:52 +03:00
Eugene Kabanov a845450283
REST API: add ssz encoding for publishBlock (#4154) 2022-09-30 00:00:53 +03:00
Jacek Sieka af9ec577d0
nicer error message for failed backfill (#4188)
* nicer error message for failed backfill

Many checkpoint sources don't support block download

* RestGenericError -> RestErrorMessage

...and other assorted fixes to bring rest types closer to spec

* fix tests
2022-09-29 23:55:18 +03:00
Eugene Kabanov 8778e1cf8d
Fix REST generic error parsing. (#4189)
* Fix REST generic error parser.
* Unescape test vectors.
* Fix RestGenericError writer and tests, to encode `code` as `Number`.
2022-09-28 18:47:15 +00:00
Etan Kissling d76df065a1
rm unused `dump` function for `KeystoresAndSlashingProtection` (#4170)
In `eth2_rest_serialization` there was a `dump` function for
`KeystoresAndSlashingProtection` that does not seem to be used.
Removes that unused function.
2022-09-26 21:09:51 +02:00
Etan Kissling 7ac95c6ee7
move `EpochParticipationFlags` serialization to others (#4167)
The REST `writeValue/readValue` serialization functions are grouped up
with the exception of `EpochParticipationFlags`. Moving that to others.
2022-09-23 18:29:31 +00:00
Etan Kissling 77ea188c92
add `sszResponsePlain` for pre-encoded SSZ data (#4171)
For pre-encoded JSON REST responses we have `jsonResponsePlain`.
Adds a `sszResponsePlain` function to serve similar purpose for SSZ.
This avoids caller having to explicitly specify Http200 and media type.
2022-09-23 15:51:04 +00:00
Etan Kissling 8b11778e14
use `withState` in `jsonResponseState` (#4169)
The generic sandwich preventing `forks.withState` from working in REST
serialization is no longer applicable. Update to use `withState`.
2022-09-23 13:49:22 +00:00
Etan Kissling cd626cf645
delete unused REST version constants (#4166)
Removes unused `Phase0Version` and `AltairVersion` constants.
2022-09-23 12:48:53 +00:00
zah ad63bba446
Support Prysm and Ethdo Keystores (Fixes #4107) (#4149) 2022-09-20 01:09:56 +03:00
Eugene Kabanov ca871a5435
Fix HTTP/REST clients HTTP Content-Type header parsers. (#4139)
* Fix client HTTP content-type parsers.

* Fix tests.

* Address review comment and apply wildcard checks for generic decodeBytes.
2022-09-19 12:17:29 +03:00
zah fca20e08d6
Keymanager API for the validator client (#3976)
* Keymanager API for the validator client
* Properly treat the 'description' field as optional when loading Keystores
* Spec-compliant serialization of the slashing data in Keymanager's DeleteKeys response ()

Fixes #3940
Fixes #3964
Closes #3884 by adding test
2022-08-19 13:30:07 +03:00
zah 06a5c67e62
The body of prepare_beacon_proposer is an array (Fixes #3920) (#3938) 2022-08-09 12:53:02 +03:00
Eugene Kabanov fe5435ed96
Address #3624 (#3934) 2022-08-06 11:55:40 +00:00
tersec d62d13a23c
MEV block proposal (#3883)
* MEV validator registration

* add nearby canary to detect new beacon chain forks

* remove special MEV graffiti

* web3signer support

* fix trace logging

* Nim 1.2 needs raises Defect

* use template rather than proc in REST JSON parsing

* use --payload-builder-enable and --payload-builder-url

* explicitly default MEV to disabled

* explicitly empty default value for payload builder URL

* revert attestation pool to unstable version
2022-08-01 09:41:47 +03:00
Miran dfd4afc9f2
compatibility with Nim 1.4+ (#3888) 2022-07-29 10:53:42 +00:00
zah cd04f27c37
Implement the /eth/v1/validator/prepare_beacon_proposer end-point (#3901) 2022-07-25 23:12:53 +03:00
Eugene Kabanov c3d3397843
VC: doppelganger protection (#3877)
* Improve fallback_service.

* Improve logging in fallback_service.

* Apply signal handling for all stages.

* Fix some logging statements.

* Add doppelganger REST api endpoint.
Add some structures to VC.

* Add client API call implementation.

* Initial fix & refactor onceToAll()
Add doppelganger service.
Add doppelganger helpers.

* Add doppelganger checks.

* Move doppelganger log messages to higher levels.

* Fix firstSuccess().

* Bump chronos.

* Post rebase fixes.

* Proper chronos bump.

* Address review comments.

* Attempt to fix finalization test issue.

* Fix nimbus_signing_node.

* Mark validators which are added at GENESIS_SLOT in GENESIS_EPOCH as passed doppelganger validation.

* Do not send empty requests to server.

* Fix log statement.

* Address review comments and re-raise cancellations.

Co-authored-by: zah <zahary@gmail.com>
2022-07-21 19:54:07 +03:00
zah 20d45e69b5
Re-enabled requireAllFields after a fix in nim-json-serialization (#3871)
* Re-enabled requireAllFields after a fix in nim-json-serialization

The problem was that `Option[T]` fields were not treated as optional
when requireAllFields is set to true. This is now fixed in NJS.

* Add makefile targets for recreating the Jenkins simulation runs

* Fix a discrepancy with the REST spec
2022-07-15 03:19:19 +03:00
Etan Kissling 47a1b11d5d
revert `requireAllFields` regression (#3870)
#3864 introduced a regression by turning on `requireAllFields` globally
for JSON parsing. Certain endpoints such as `RestSyncInfo` have optional
fields that do not parse correctly without additional changes. This is
reverted for now to restore previous behaviour and unblock CI testing.
2022-07-14 03:33:15 +00:00
zah 806536a040
[Keymanager API] Support for the feerecipient end-points (#3864)
Other changes:

* The Keymanager error responses differ from the Beacon API responses.
  'keymanagerApiError' replaces the former usages of 'jsonError'.

* Return status code 401 and 403 for authorization errors in accordance
  to the spec.

* Eliminate inconsistencies in the REST JSON parsing. Some of the code
  paths allowed missing fields.

* Added logging of serialization failure details at DEBUG level.
2022-07-13 17:45:04 +03:00
tersec e0e7af7aff
MEV boost script for ropsten and sepolia builder network (#3851)
* MEV boost script for ropsten and sepolia builder network

* networkInfo not a global variable
2022-07-12 20:50:12 +03:00
tersec 56cd6e2b45
rm unused imports and fix stylecheck issues (#3784) 2022-06-21 21:01:45 +02:00
Eugene Kabanov eb6b7affee
Add the `execution_optimistic` flag to REST API responses. (#3780)
* Initial commit

* Make `events` API spec compliant.

* Add `Eth-Consensus-Version` in responses.

* Bump chronos to get redirect with headers working.

* Add `is_optimistic` field and handling to syncing RestSyncInfo.
2022-06-20 08:53:39 +03:00
zah c24c737866
Fix #3650 (participation format in BeaconState result is out of spec) (#3776)
* Fix #3650 (participation format in BeaconState result is out of spec)
* Make EpochParticipationFlags a distinct type
2022-06-20 08:38:56 +03:00
Eugene Kabanov c4383e21f4
Fix #3760. (#3768)
* Address #3760.

Add test vectors for deserialization.
2022-06-19 09:08:21 +03:00
Etan Kissling 61ee0611bb
adopt LC REST API with v0 suffix (without proofs) (#3775)
* adopt LC REST API with v0 suffix (without proofs)

Adopts the light client data REST API used by Lodestar as defined in
https://github.com/ethereum/beacon-APIs/pull/181 with a v0 suffix.

Requests:
- `/eth/v0/beacon/light_client/bootstrap/{block_root}`
- `/eth/v0/beacon/light_client/updates?start_period={start_period}&count={count}`
- `/eth/v0/beacon/light_client/finality_update`
- `/eth/v0/beacon/light_client/optimistic_update`

HTTP Server-Sent Events (SSE):
- `light_client_finality_update_v0`
- `light_client_optimistic_update_v0`

More work is needed to adopt the proofs endpoint, it is not included.

* initialize event queues

* register event topics
2022-06-19 08:57:52 +03:00
Zahary Karadjov 9750a2a760
Change the log level for unrecognized json fields from debug to trace 2022-06-09 16:08:50 +03:00
tersec 1262e50fa5
document MEV/mergemock test script (#3694) 2022-06-06 16:55:02 +03:00
zah 68fb3962c8
More spec-compliant handling of unknown fields in REST json (#3647)
* More spec-compliant handling of unknown fields in REST json

* Address review comments
2022-05-20 18:25:26 +03:00
zah 397033a7d6
Improve the compatibility with various Web3Signer versions and configurations (#3640)
* Some Web3Signer versions insist replying with text/plain messages
* When reading blocks, the Web3Signer uses upper-case fork identifiers
  instead of lower-case identifies like the Beacon API.
2022-05-17 19:55:03 +03:00
tersec e7ed7332b3
MEV REST serialization and call signatures (#3625) 2022-05-17 13:56:19 +00:00
zah a2ba34f686
Implement all sync committee duties in the validator client (#3583)
Other changes:

* logtrace can now verify sync committee messages and contributions
* Many unnecessary use of pairs() have been removed for consistency
* Map 40x BN response codes to BeaconNodeStatus.Incompatible in the VC
2022-05-10 10:03:40 +00:00
zah 6d11ad6ce1
Support for distributed keystores with multiple remotes based on threshold signatures (#3616)
Other fixes:

* Fix bit rot in the `make prater-dev-deposit` target.
* Correct content-type in the responses of the Nimbus signing node
* Invalid JSON payload was being sent in the web3signer requests
2022-05-10 03:32:12 +03:00
Jacek Sieka f70ff38b53
enable `styleCheck:usages` (#3573)
Some upstream repos still need fixes, but this gets us close enough that
style hints can be enabled by default.

In general, "canonical" spellings are preferred even if they violate
nep-1 - this applies in particular to spec-related stuff like
`genesis_validators_root` which appears throughout the codebase.
2022-04-08 16:22:49 +00:00
Eugene Kabanov 1a0bcf0b02
Fix #3267 (#3367)
* Initial commit.

* One more fix.

* Trying to debug the finalization issue.

* Add debug logs to understand signature issue.

* Restore hash_tree_root calculation.

* Remove all the debugging helpers.

* Add `slot` check.

* Address review comment.
2022-02-13 16:21:55 +01:00
Eugene Kabanov 40c77e5928
Remote KeyManager API and number of fixes/tests for KeyManager API (#3360)
* Initial commit.

* Fix current test suite.

* Fix keymanager api test.

* Fix wss_sim.

* Add more keystore_management tests.

* Recover deleted isEmptyDir().

* Add `HttpHostUri` distinct type.
Move keymanager calls away from rest_beacon_calls to rest_keymanager_calls.
Add REST serialization of RemoteKeystore and Keystore object.
Add tests for Remote Keystore management API.
Add tests for Keystore management API (Add keystore).
Fix serialzation issues.

* Fix test to use HttpHostUri instead of Uri.

* Add links to specification in comments.

* Remove debugging echoes.
2022-02-07 22:36:09 +02:00
Emil efbd939108 Make `handleValidatorExitCommand` work with `REST API` 2022-01-25 14:00:29 +02:00
tersec 351c2fd48a
rename mergeData to bellatrixData and mergeFork to bellatrixFork (#3315) 2022-01-24 16:23:13 +00:00
Jacek Sieka 1df549143e
rest: fix GraffitiBytes serialization (#3299) 2022-01-20 13:31:55 +01:00
Jacek Sieka 4e2d2ff7f4 rest: fix invalid type `RestSyncCommitteeSubscription`
Using the wrong type here causes requests to fail due to the overly
zealous parameter validation - the failure is harmless in the current
duty subscription model, but would have caused more serious failures
down the line.
2022-01-17 22:33:24 +02:00
Jacek Sieka 68247f81b3
Trusted node sync (#3209)
* Trusted node sync

Trusted node sync, aka checkpoint sync, allows syncing tyhe chain from a
trusted node instead of relying on a full sync from genesis.

Features include:

* sync from any slot, including the latest finalized slot
* backfill blocks either from the REST api (default) or p2p (#3263)

Future improvements:

* top up blocks between head in database and some other node - this
makes for an efficient backup tool
* recreate historical state to enable historical queries

* fixes

* load genesis from network metadata
* check checkpoint block root against state
* fix invalid block root in rest json decoding
* odds and ends

* retry looking for epoch-boundary checkpoint blocks
2022-01-17 10:27:08 +01:00
Jacek Sieka 20e700fae4
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259)
* Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex

Harden the use of `CommitteeIndex` et al to prevent future issues by
using a distinct type, then validating before use in several cases -
datatypes in spec are kept simple though so that invalid data still can
be read.

* fix invalid epoch used in REST
`/eth/v1/beacon/states/{state_id}/committees` committee length (could
return invalid data)
* normalize some variable names
* normalize committee index loops
* fix `RestAttesterDuty` to use `uint64` for `validator_committee_index`
* validate `CommitteeIndex` on ingress in REST API
* update rest rules with stricter parsing
* better REST serializers
* save lots of memory by not using `zip` ...at least a few bytes!
2022-01-09 01:28:49 +02:00
Jacek Sieka 6f7e0e3393
REST cleanups (#3255)
* REST cleanups

* reject out-of-range committee requests
* print all hex values as lower-case
* allow requesting state information by head state root
* turn `DomainType` into array (follow spec)
* `uint_to_bytesXX` -> `uint_to_bytes` (follow spec)
* fix wrong dependent root in `/eth/v1/validator/duties/proposer/`
* update documentation - `--subscribe-all-subnets` is no longer needed
when using the REST interface with validator clients
* more fixes
* common helpers for dependent block
* remove test rules obsoleted by more strict epoch tests
* fix trailing commas

* Update docs/the_nimbus_book/src/rest-api.md
* Update docs/the_nimbus_book/src/rest-api.md

Co-authored-by: sacha <sacha@status.im>
2022-01-08 22:06:34 +02:00
tersec 8242e57f41
initial migration from spec/datatypes/{merge => bellatrix} (#3249) 2022-01-06 12:25:35 +01:00
Jacek Sieka 0e2b4e39fa
REST JSON support improvements (#3232)
* support downloading blocks / states via JSON in addition to SSZ -
slow, but needed for infura support - SSZ is still used when server
supports it
* use common forked block/state reader in REST API
* fix stack overflows in REST JSON decoder
* fix invalid serialization of `justification_bits` in
`/eth/v1/debug/beacon/states` and `/eth/v2/debug/beacon/states`
* fix REST client to use `/eth/...` instead of `/api/eth/...`, update
"default" urls to expose REST api via `/eth` as well as this is what the
standard says - `/api` was added early on based on an example "base url"
in the spec that has been removed since
* expose Nimbus REST extensions via `/nimbus` in addition to
`/api/nimbus` to stay consistent with `/eth`
* fix invalid state root when reading states via REST
* fix recursive imports in `spec/ssz_codec`
* remove usages of `serialization.useCustomSerialization` - fickle
2022-01-06 08:38:40 +01:00
Zahary Karadjov 54d0d588b1 Implementation of the Keymanager API (BETA)
https://github.com/ethereum/keymanager-APIs
2022-01-04 18:51:45 +02:00