Commit Graph

2237 Commits

Author SHA1 Message Date
zah ff3766e79b
Fix the Prater ENR fork id (#2798) 2021-08-20 15:04:29 +02:00
tersec 4678a2bee7
move BeaconClock from ChainDAG to BeaconNode (#2796) 2021-08-20 08:58:15 +00:00
tersec 317b6de4e6
send attestations and exit messages on fork-appropriate topic (#2773)
* send attestations and exit messages on fork-appropriate topic

* document why use wall clock over attestation slot

* centralize some fork-topic-picking-logic in eth2_network

* pick up new test in summary

* allow specified GetTimeFn for testing purposes

* add GenesisTime and use it in eth2_network

* replace GetTimeFn and GenesisTime with GetBeaconTimeFn
2021-08-19 10:45:31 +00:00
Jacek Sieka a7a65bce42
disentangle eth2 types from the ssz library (#2785)
* reorganize ssz dependencies

This PR continues the work in
https://github.com/status-im/nimbus-eth2/pull/2646,
https://github.com/status-im/nimbus-eth2/pull/2779 as well as past
issues with serialization and type, to disentangle SSZ from eth2 and at
the same time simplify imports and exports with a structured approach.

The principal idea here is that when a library wants to introduce SSZ
support, they do so via 3 files:

* `ssz_codecs` which imports and reexports `codecs` - this covers the
basic byte conversions and ensures no overloads get lost
* `xxx_merkleization` imports and exports `merkleization` to specialize
and get access to `hash_tree_root` and friends
* `xxx_ssz_serialization` imports and exports `ssz_serialization` to
specialize ssz for a specific library

Those that need to interact with SSZ always import the `xxx_` versions
of the modules and never `ssz` itself so as to keep imports simple and
safe.

This is similar to how the REST / JSON-RPC serializers are structured in
that someone wanting to serialize spec types to REST-JSON will import
`eth2_rest_serialization` and nothing else.

* split up ssz into a core library that is independendent of eth2 types
* rename `bytes_reader` to `codec` to highlight that it contains coding
and decoding of bytes and native ssz types
* remove tricky List init overload that causes compile issues
* get rid of top-level ssz import
* reenable merkleization tests
* move some "standard" json serializers to spec
* remove `ValidatorIndex` serialization for now
* remove test_ssz_merkleization
* add tests for over/underlong byte sequences
* fix broken seq[byte] test - seq[byte] is not an SSZ type

There are a few things this PR doesn't solve:

* like #2646 this PR is weak on how to handle root and other
dontSerialize fields that "sometimes" should be computed - the same
problem appears in REST / JSON-RPC etc

* Fix a build problem on macOS

* Another way to fix the macOS builds

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2021-08-18 20:57:58 +02:00
tersec a060985abc
unexport various parts of tests/ and remove unused code (#2794) 2021-08-18 13:58:43 +00:00
Jacek Sieka 70259e4e64
treat gossip decoding errors more strictly (#2793)
* penalize peers for sending gossip messages that fail decoding
* add metrics for decoding/decompression errors
* clean up obsolete exception handlers
2021-08-18 14:30:05 +02:00
Jacek Sieka bfe5e74607
fix produceBlock response to return correct type (#2792) 2021-08-18 11:00:16 +02:00
tersec 6e46445da2
switch result = foo to expression return; unexport rest of logtrace symbols (#2788) 2021-08-17 09:51:39 +00:00
tersec a0c518cb4f
sync committee/aggregate signature signing and verification (#2784)
* sync committee/aggregate signature signing and verification

* add message signature tests
2021-08-17 08:07:17 +00:00
Jacek Sieka 63717531dc remove remaining traces of nim-prompt
upstream dead, as is feature in eth2
2021-08-16 21:56:50 +03:00
Jacek Sieka 07728e4f7c
altair devnet 3 (#2782) 2021-08-15 19:50:52 +02:00
tersec 89d9aa9240
sync committee topic names; borrow -> template/distinctbase (#2781) 2021-08-15 16:50:31 +02:00
Jacek Sieka 7a622e8505
rework spec imports (#2779)
The spec imports are a mess to work with, so this branch cleans them up
a bit to ensure that we avoid generic sandwitches and that importing
stuff generally becomes easier.

* reexport crypto/digest/presets because these are part of the public
symbol set of the rest of the spec types
* don't export `merge` types from `base` - this causes circular deps
* fix circular deps in `ssz/spec_types` - this is the first step in
disentangling ssz from spec
* be explicit about phase0 vs altair - longer term, `altair` will become
the "natural" type set, then merge and so on, so no point in giving
`phase0` special preferential treatment
2021-08-12 13:08:20 +00:00
Jacek Sieka 9697b73e71
forkedbeaconstate_helpers -> forks (#2772)
Simpler module name for stuff that covers forks

* check that runtime config matches database state
* also include some assorted altair cleanups
* use "standard" genesis fork in local testnet to work around missing
runtime config support
2021-08-10 22:46:35 +02:00
Jacek Sieka 92aba71604
Pyrmont altair fork (#2771)
Prater upgrading to altair at epoch 61650 -
https://github.com/eth2-clients/eth2-networks/pull/56

also includes altair-devnet-2 support in binary
2021-08-10 10:16:43 +02:00
tersec 2eb6246b90
altair enr switching (#2770)
* altair enr switching

* altair transitions only happen on epoch boundaries

* determine ENR period by wall time, not chain head
2021-08-10 08:19:13 +02:00
Jacek Sieka a663ed65f4
Merge branch 'merge-stable' into unstable 2021-08-09 15:00:58 +02:00
tersec 2afe2802b6
altair topic switching (#2767)
* altair topic switching

* remove validate{Committee,Validator}IndexOr unused within branch
2021-08-09 12:54:45 +00:00
Jacek Sieka 7bb76a6cd1
Merge remote-tracking branch 'origin/stable' into merge-stable 2021-08-09 13:14:28 +02:00
Eugene Kabanov b69f566ed1
Implementation for v2 REST API calls. (#2731)
* Implementation for v2 rest api calls.

* fix phase0 block deserialization

Co-authored-by: Jacek Sieka <jacek@status.im>
2021-08-09 08:08:18 +02:00
Zahary Karadjov 1e958cfb0a version 1.4.2 2021-08-05 14:28:17 +03:00
Jacek Sieka ee79c10a7d
update validator key cache on startup (#2760)
* update validator key cache on startup

Versions prior to 1.1.0 do not write a validator key cache at all.

Versions from 1.4.0 and upwards require an immutable validator key cache
to verify blocks - normally, block verification fills the cache but that
assumes that at least one block was verified by a version that has the
key cache.

Taken together, this breaks direct upgrades from anything <1.1.0 to
1.4.0.

The fix is simply to refresh fill the cache from an existing state on
startup.

* also log serious block validation failures at info level
2021-08-05 11:26:10 +03:00
Jacek Sieka 3d7bee8502
REST API client, JSON-RPC cleanups (#2756)
This refactoring puts the JSON-RPC and REST APIs on more equal footing
by renaming and moving things around, creating a separation between
client and server, and documenting what they are - the aim is to have a
simple-to-use base to start from when developing API clients, as well as
make it easier to navigate the code when looking for the legacy JSON-RPC
interface vs the new REST API.

* move REST client, serialization and supporting types to spec/eth2_apis
* REST stuff now starts with `rest_`, JSON-RPC stuff starts with `rpc_`,
more or less
* simplify imports such that there's a simple module to import for both
server and client
* map REST type and proc names to yaml spec more closely - in
particular, reuse operation and type names in `rest_types` to make
comparisons against spec more easy
* cleaner separation between client and server modules - modules common
between server and client such as `rest_types` and serialization move to
the spec folder - this allows the client to be built with less knowledge
about server internals
2021-08-03 17:17:11 +02:00
tersec d638ca0c7f
log overall sent aggregated attestation message signatures (#2754)
* log overall sent aggregated attestation message signatures

* log aggregated attestations via REST API
2021-08-03 10:32:55 +00:00
tersec a2c1b96ac9
prevent resyncing from genesis with altair head block (#2750) 2021-08-01 10:20:43 +02:00
Kim De Mey 06b2a1642d
Remove incorrect enode import (#2748) 2021-07-29 20:53:58 +02:00
tersec 5f3225c6e5
fix invalid_signature_no_participants test case (#2741) 2021-07-27 14:04:41 +00:00
nbc-bump-bot[bot] f68061b2d8
nim-libp2p auto bump (#2647)
* auto-bump nim-libp2p

* update peerhook for libp2p

Co-authored-by: = <Menduist@users.noreply.github.com>
Co-authored-by: Tanguy Cizain <tanguycizain@gmail.com>
2021-07-26 12:36:17 +00:00
tersec 38ce948647
partial altair merge (#2735)
* partial altair merge

* exclude still-in-flux sync committee data structures from partial merge

* undo the remaining sync_committee mention
2021-07-26 09:51:14 +00:00
Eugene Kabanov f9cd98702e
Fix getBlockHeaders() should return array. (#2732)
Fix tests.
2021-07-24 19:22:05 +02:00
Eugene Kabanov f0c30e31b4
VC: various fixes (#2730)
* Fix firstSuccess() template missing timeouts.

* Fix validator race condition.
Fix logs to be compatible with beacon_node logs.
Add CatchableError handlers to avoid crashes.
Move some logs from Notice to Debug level.
Fix some [unused] warnings.

* Fix block proposal issue for slots in the past and from the future.

* Change sent to published.

* Address review comments #1.
2021-07-19 14:31:02 +00:00
Kim De Mey 3e3e17fec3
Remove raises Exception and bump nim-eth (#2729) 2021-07-19 14:25:11 +02:00
tersec 9d9c37c561
some whole-file copies from altair branch (#2728)
* some whole-file copies from altair branch

* rpc/node_api and rpc/node_rest_api also need to be copied

* remove new sync committee-related functionality
2021-07-19 11:58:30 +00:00
tersec aebc606cb7
tighten local network simulation correctness checking (#2706)
* tighten local network simulation correctness checking

* rename rejectFirmly to errReject
2021-07-19 11:58:22 +00:00
zah d9f2a91374
Remove the obsolete testnet0/1 scripts (#2727)
Also fixes `make eth2_network_simulation`
2021-07-16 13:02:27 +03:00
Jacek Sieka 2d6a661ac6
Syncv2 (#2723)
* bump libp2p

* altair sync v2

Use V2 sync requests after the altair fork has happened, according to
the wall clock

* Fix the behavior of the v1 req/resp calls after Altair

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2021-07-15 21:01:07 +02:00
Eugene Kabanov 754aeec2b0
Fix and optimize aggregate and proofs generation algorithm. (#2722) 2021-07-15 08:17:32 +00:00
tersec e4afc36d71
use ForkedTrustedSignedBeaconBlock (#2720)
* use ForkedTrustedSignedBeaconBlock

* remove --subscribe-all-subnets

* https://ethereum.github.io/eth2.0-APIs/#/Beacon/getBlock implementation was passing through forked beaconblocks
2021-07-14 12:18:52 +00:00
Ștefan Talpalaru 840935ddc2
limit validator balance metric label values (#2719) 2021-07-14 08:22:03 +02:00
Jacek Sieka 3f9c1fdf4e
More RuntimeConfig cleanup (#2716)
* remove from BeaconChainDB (doesn't depend on runtime config)
* eth2-testnets -> eth2-networks
* use `cfg` name throughout
2021-07-13 16:27:10 +02:00
Eugene Kabanov 3b6f4fab4a
New validator client using REST API. (#2651)
* Initial commit.

* Exporting getConfig().

* Add beacon node checking procedures.

* Post rebase fixes.

* Use runSlotLoop() from nimbus_beacon_node.
Fallback implementation.
Fixes for ETH2 REST serialization.

* Add beacon_clock.durationToNextSlot().
Move type declarations from beacon_rest_api to json_rest_serialization.
Fix seq[ValidatorIndex] serialization.
Refactor ValidatorPool and add some utility procedures.
Create separate version of validator_client.

* Post-rebase fixes.
Remove CookedPubKey from validator_pool.nim.

* Now we should be able to produce attestations and aggregate and proofs.
But its not working yet.

* Debugging attestation sending.

* Add durationToNextAttestation.
Optimize some debug logs.
Fix aggregation_bits encoding.
Bump chronos/presto.

* Its alive.

* Fixes for launch_local_testnet script.
Bump chronos.

* Switch client API to not use `/api` prefix.

* Post-rebase adjustments.

* Fix endpoint for publishBlock().

* Add CONFIG_NAME.
Add more checks to ensure that beacon_node is compatible.

* Add beacon committee subscription support to validator_client.

* Fix stacktrace should be an array of strings.
Fix committee subscriptions should not be `data` keyed.

* Log duration to next block proposal.

* Fix beacon_node_status import.

* Use jsonMsgResponse() instead of jsonError().

* Fix graffityBytes usage.
Remove unnecessary `await`.
Adjust creation of SignedBlock instance.
Remove legacy files.

* Rework durationToNextSlot() and durationToNextEpoch() to use `fromNow`.

* Fix race condition for block proposal and attestations for same slot.
Fix local_testnet script to properly kill tasks on Windows.
Bump chronos and nim-http-tools, to allow connections to infura.io (basic auth).

* Catch services errors.
Improve performance of local_testnet.sh script on Windows.
Fix race condition when attestation producing.

* Post-rebase fixes.

* Bump chronos and presto.

* Calculate block publishing delay.
Fix pkill in one more place.

* Add error handling and timeouts to firstSuccess() template.
Add onceToAll() template.
Add checkNodes() procedure.
Refactor firstSuccess() template.
Add error checking to api.nim calls.

* Deprecated usage onceToAll() for better stability.
Address comment and send attestations asap.

* Avoid unnecessary loop when calculating minimal duration.
2021-07-13 13:15:07 +02:00
Kim De Mey 81d54a3c89
Avoid trying to close discv5 transport when discv5 is not enabled (#2713) 2021-07-13 09:58:45 +02:00
Jacek Sieka 23eea197f6
Implement split preset/config support (#2710)
* Implement split preset/config support

This is the initial bulk refactor to introduce runtime config values in
a number of places, somewhat replacing the existing mechanism of loading
network metadata.

It still needs more work, this is the initial refactor that introduces
runtime configuration in some of the places that need it.

The PR changes the way presets and constants work, to match the spec. In
particular, a "preset" now refers to the compile-time configuration
while a "cfg" or "RuntimeConfig" is the dynamic part.

A single binary can support either mainnet or minimal, but not both.
Support for other presets has been removed completely (can be readded,
in case there's need).

There's a number of outstanding tasks:

* `SECONDS_PER_SLOT` still needs fixing
* loading custom runtime configs needs redoing
* checking constants against YAML file

* yeerongpilly support

`build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG`

* load fork epoch from config

* fix fork digest sent in status
* nicer error string for request failures
* fix tools

* one more

* fixup

* fixup

* fixup

* use "standard" network definition folder in local testnet

Files are loaded from their standard locations, including genesis etc,
to conform to the format used in the `eth2-networks` repo.

* fix launch scripts, allow unknown config values

* fix base config of rest test

* cleanups

* bundle mainnet config using common loader
* fix spec links and names
* only include supported preset in binary

* drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 15:01:38 +02:00
Mamy André-Ratsimbazafy 6f441191e4
v1.4.1 changelog and version 2021-07-10 15:18:32 +02:00
Ștefan Talpalaru 0aef63948f
add version metric (using labels) (#2711) 2021-07-09 07:41:44 +02:00
Jacek Sieka bd684d0eea
increase block attestation wait time (#2705)
We generally send out attestations 250 ms after the block arrives.
Recent efficiency improvements have led to a slightly increased
incidence of "slot 0" issues  where attestations are dropped by other
nodes because they have not yet had time to process the block due to
epoch processing taking time.

This PR mitigates the problem by increasing the window between receiving
the block and sending out attestations.
2021-07-07 12:11:51 +02:00
Jacek Sieka be756454a4
flush stdout logs (#2669)
else pipes end up not getting log output until much later
2021-07-07 12:11:12 +02:00
zah eb2dc5cbbb
Implement the new Altair req/resp protocols (#2676)
* Implement the new Altair req/resp protocols

Also fixes the altair message-id computation by providing the correct
forkdigest prefix in `isAltairTopic`.

Co-authored-by: Tanguy Cizain <tanguycizain@gmail.com>
2021-07-07 12:09:47 +03:00
Jacek Sieka 7825d12448
increase block attestation wait time (#2705)
We generally send out attestations 250 ms after the block arrives.
Recent efficiency improvements have led to a slightly increased
incidence of "slot 0" issues  where attestations are dropped by other
nodes because they have not yet had time to process the block due to
epoch processing taking time.

This PR mitigates the problem by increasing the window between receiving
the block and sending out attestations.
2021-07-06 15:11:18 +02:00
tersec ac7f719382
use isomorphicCast between beacon block types (#2698) 2021-07-06 14:32:49 +02:00
tersec 5e9b932c3c
avoid recomputing base_reward_per_increment() across attestations (#2704)
* avoid recomputing base_reward_per_increment() across attestations

* add adapter template to fix nbench process_attestations build

* nfuzz also uses process_attestations()
2021-07-06 06:19:06 +00:00
tersec 62212c8493
make isomorphicCast available outside immutable beacon chain database (#2696)
* make isomorphicCast available outside immutable beacon chain database
2021-07-02 17:38:14 +00:00
tersec 68ca600d89
update 20 more spec refs to v1.1.0-alpha.8 (#2690) 2021-07-01 12:13:59 +00:00
tersec c9cec88a23
update some spec refs to v1.1.0-alpha.8 (#2689) 2021-07-01 08:55:16 +00:00
tersec 7577f8c2ef
add blockchain_dag altair database reading; add rollback tests (#2683)
* add blockchain_dag altair database reading; add rollback tests; fix some unnecessary type conversions

* remove debugging scaffolding

* proposeSignedBlock() will need to be async for merge; introduce altair types to VC
2021-06-29 15:09:29 +00:00
Jacek Sieka cc8f7c26a0
flush stdout logs (#2669)
else pipes end up not getting log output until much later
2021-06-29 16:53:36 +03:00
tersec 60f601e4e6
cache base_reward_per_increment for 5x+ Altair block_sim perf increase (#2681) 2021-06-28 06:54:06 +00:00
tersec ec54173667
optimize phase 0 -> Altair transition state processing (#2679) 2021-06-25 12:51:25 +00:00
Tanguy Cizain aeef550d0a
Altair message-id topic (#2671) 2021-06-25 03:07:46 +03:00
tersec 445def6c8b
block_clearance, ncli, and ncli_db Altair state saving (#2672)
* block_clearance, ncli, and ncli_db Altair state saving

* avoid invalidating SSZ hash caches with every assignment
2021-06-24 18:34:08 +00:00
tersec 41e0a7abc0
introduce database support for Altair (#2667)
* introduce immutable Altair BeaconState

* add database support for Altair blocks and states

* add tests for Altair get/put/contains/delete state

* enable blockchain_dag Altair state database storing

* properly return error on getting missing altair block
2021-06-24 07:11:47 +00:00
tersec ae1abf24af
add Altair support to block quarantine/clearance and block_sim (#2662)
* add Altair support to the block quarantine

* switch some spec/datatypes imports to spec/datatypes/base

* add Altair support to block_clearance

* allow runtime configuration of Altair transition slot

* enable Altair in block_sim, including in CI
2021-06-23 14:43:18 +00:00
Mamy André-Ratsimbazafy 8dc4db51c0
Merge branch 'stable' into unstable 2021-06-23 10:43:23 +02:00
Mamy André-Ratsimbazafy 4ef784fa68
v1.4.0 2021-06-21 14:02:12 +02:00
Eugene Kabanov 533cd25e1c
Add CONFIG_NAME parameter to REST API /eth/v1/config/spec response. (#2652)
* Add CONFIG_NAME parameter to /eth/v1/config/spec response.

* Add missed comma.

* Use `const_preset` instead of CONFIG_NAME.
2021-06-21 11:31:10 +02:00
tersec b1d5609171
remove false OnBlockAdded dependency on phase0 HashedBeaconState (#2661)
* remove false OnBlockAdded dependency on phase.HashedBeaconState

* introduce altair data types into block_clearance; update some alpha.6 spec refs to alpha.7; add get_active_validator_indices_len ForkedHashedBeaconState wrapper

* switch many modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 and remove rm'd G2_POINT_AT_INFINITY

* switch more modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7

* remove unnecessary phase0-only wrapper of get_attesting_indices(); allow signatures_batch to process either fork; remove O(n^2) nested loop in process_inactivity_updates(); add altair support to getAttestationsforTestBlock()

* add Altair versions of asSigVerified(), asTrusted(), and makeBeaconBlock()

* fix spec URL to be Altair for Altair makeBeaconBlock()
2021-06-21 08:35:24 +00:00
Eugene Kabanov f8eb906b51
Add CONFIG_NAME parameter to REST API /eth/v1/config/spec response. (#2652)
* Add CONFIG_NAME parameter to /eth/v1/config/spec response.

* Add missed comma.

* Use `const_preset` instead of CONFIG_NAME.
2021-06-18 14:43:54 +02:00
tersec 9616220280
implement Altair attestation pool cache init (#2659)
* implement Altair attestation pool cache init

* remove code duplication around previous/current epoch updates
2021-06-17 17:13:14 +00:00
Jacek Sieka a3322acee9
Revert "fix assertion in beacon block creation rollback/restore (#2655)"
This reverts commit 9d48097de0.
2021-06-17 16:52:11 +02:00
Mamy Ratsimbazafy e14873757e
Fix #2595 - recompute wallSlot across async calls (#2650) 2021-06-17 16:43:58 +02:00
tersec 9ad8310157
doppelganger detection walltime refactor (#2656)
* strawman doppelganger detection walltime refactor

* move DoppelgangerProtection to Eth2Processor

* increase comment precision

* document difference between broadcastStartEpoch and nodeLaunchSlot, and allow for one-slot overlap to avoid false positives on intra-slot restarts
2021-06-17 16:43:58 +02:00
tersec 9d48097de0
fix assertion in beacon block creation rollback/restore (#2655) 2021-06-17 16:43:57 +02:00
Mamy Ratsimbazafy 2adf54e44a
Fix #2595 - recompute wallSlot across async calls (#2650) 2021-06-17 16:38:25 +02:00
tersec 87ed9e62a1
doppelganger detection walltime refactor (#2656)
* strawman doppelganger detection walltime refactor

* move DoppelgangerProtection to Eth2Processor

* increase comment precision

* document difference between broadcastStartEpoch and nodeLaunchSlot, and allow for one-slot overlap to avoid false positives on intra-slot restarts
2021-06-17 11:51:04 +00:00
tersec 53d05060c9
fix assertion in beacon block creation rollback/restore (#2655) 2021-06-17 09:22:39 +02:00
Jacek Sieka a5711ecf18
remove a few obsolete raises pops (#2654) 2021-06-16 14:45:05 +02:00
tersec 1c3314f08b
update to Altair as of v1.1.0-alpha.7 (#2649)
* update to Altair as of v1.1.0-alpha.7

* introduce Altair types into attestation pool

* avoid allocating/copying pubkeys excessively in get_next_sync_committee()
2021-06-14 17:42:46 +00:00
tersec 146fa48454
use ForkedHashedBeaconState in StateData (#2634)
* use ForkedHashedBeaconState in StateData

* fix FAR_FUTURE_EPOCH -> slot overflow; almost always use assign()

* avoid stack allocation in maybeUpgradeStateToAltair()

* create and use dispatch functions for check_attester_slashing(), check_proposer_slashing(), and check_voluntary_exit()

* use getStateRoot() instead of various state.data.hbsPhase0.root

* remove withStateVars.hashedState(), which doesn't work as a design anymore

* introduce spec/datatypes/altair into beacon_chain_db

* fix inefficient codegen for getStateField(largeStateField)

* state_transition_slots() doesn't either need/use blocks or runtime presets

* combine process_slots(HBS)/state_transition_slots(HBS) which differ only in last-slot htr optimization

* getStateField(StateData, ...) was replaced by getStateField(ForkedHashedBeaconState, ...)

* fix rollback

* switch some state_transition(), process_slots, makeTestBlocks(), etc to use ForkedHashedBeaconState

* remove state_transition(phase0.HashedBeaconState)

* remove process_slots(phase0.HashedBeaconState)

* remove state_transition_block(phase0.HashedBeaconState)

* remove unused callWithBS(); separate case expression from if statement

* switch back from nested-ref-object construction to (ref Foo)(Bar())
2021-06-11 20:51:46 +03:00
cheatfate b09a3a6b4a Fix `aggregation_bits` json serialization. 2021-06-11 01:08:17 +03:00
Jacek Sieka 9193be9b7b
fix epoch logging (fixes #2283) (#2642)
Also put epoch first to disambiguate vs slot
2021-06-11 01:07:16 +03:00
Zahary Karadjov 651a806915
Fix another logging property type inconsistency 2021-06-10 12:17:17 +03:00
Zahary Karadjov 45430f2904
Fix an accidentally commented out line 2021-06-10 11:42:40 +03:00
Zahary Karadjov ff8212e58e
Fix #2287 2021-06-10 11:39:53 +03:00
Jacek Sieka dc9ced03f1 space is printable
Printing space as a string in graffiti makes a few more graffitis
visible
2021-06-10 10:40:00 +03:00
Jacek Sieka d859bc12f0
write uncompressed validator keys to database (#2639)
* write uncompressed validator keys to database

Loading 150k+ validator keys on startup in compressed format takes a lot
of time - better store them in uncompressed format which makes behaviour
just after startup faster / more predictable.

* refactor cached validator key access
* fix isomorphic cast to work with non-var instances
* remove cooked pubkey cache - directly use database cache in chaindag
as well (one less cache to keep in sync)
* bump blscurve, introduce loadValid for known-to-be-valid keys
2021-06-10 10:37:02 +03:00
Jacek Sieka 5974fb0e7d speed up initial migration
an isolated transaction when loading the database helps keep migration
time down on first start after upgrade
2021-06-09 20:04:20 +03:00
Kim De Mey 961b29ad5f
Remove no longer needed raw Exception raises for queryRandom (#2633) 2021-06-08 22:23:19 +02:00
tersec 8ebd496fbe
Altair transition tests (#2624)
* Working Altair transition tests

* with fixed upstream test vectors, remove state root workaround

* switch upgrade_to_altair() to returning a reference

* remove test_state_transition

* fix invalid fork state/block combinations error messages

* avoid memory copies by reintroducing state_transition_slots(var SomeHashedBeaconState)
2021-06-04 10:38:00 +00:00
Jacek Sieka b11da2cb34 fix state cache loading
* load the cache of the current state epoch instead of the target state
epoch, when applying states and slots
* load state cache for each slot/block (for longer slot jumps)
* load state cache after full updateStateData
* look up two state cache epochs, instead of the same epoch twice :)
2021-06-03 21:37:52 +03:00
Eugene Kabanov 84f3b2dd09
Increase REST service timeout setting to 2xSECONDS_PER_SLOT. (#2627)
Fix `ansi_c` warning on Windows.
2021-06-03 11:43:04 +02:00
tersec 28a5bca71a
split state_transition() into slots/block parts and use only block where appropriate (#2630) 2021-06-03 11:42:25 +02:00
Johann Bauer 141c39cfaa
Improve explanation of the --doppelganger-detection option in --help. (#2626)
* Improve explanation of the --doppelganger-detection option in --help.

* Update beacon_chain/conf.nim

Co-authored-by: 0xmiel <sacha.saint-leger@ethereum.org>

Co-authored-by: 0xmiel <sacha.saint-leger@ethereum.org>
2021-06-03 11:20:42 +02:00
Jacek Sieka 0fb02b5206 log state update duration, lower info threshold for detail logging 2021-06-01 20:43:44 +03:00
Jacek Sieka abe0d7b4ae singe validator key cache
Instead of keeping a validator key list per EpochRef, this PR introduces
a single shared validator key list in ChainDAG, and cleans up some other
ChainDAG and key-related issues.

The PR does not introduce the validator key list in the state transition
- this is because we batch-check all signatures before entering the spec
code, thus the spec code never hits the cache.

A future refactor should _probably_ remove the threadvar altogether.

There's a few other small fixes in here that make the flow easier to
read:

* fix `var ChainDAGRef` -> `ChainDAGRef`
* fix `var QuarantineRef` -> `QuarantineRef`
* consistent `dag` variable name
* avoid using threadvar pubkey cache in most cases
* better error messages in batch signature checking
2021-06-01 20:43:44 +03:00
Zahary Karadjov ed2f6f753d Allow the custom testnet metadata to specify a path to the genesis file 2021-06-01 15:50:50 +03:00
tersec ea9ceb693a
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622)
* update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot()

* update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
Jacek Sieka 9b89f58089
revert advance back to trace 2021-06-01 14:09:11 +02:00
tersec a808a6755a
add Altair fork tests for upgrade_to_altair() (#2620)
* add Altair fork tests for upgrade_to_altair()

* Deposit processing ensures all keys are valid
2021-05-31 12:54:53 +00:00
tersec 8912196335
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619)
* fix Altair epoch state transitions

* re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
Jacek Sieka 60df17786e avoid reading legacy db on write
* don't consider legacy database when writing state - this read is slow
on kvstore
* avoid epoch transition when there's an exact match in cache already
* simplify init to only consider checkpoint states
2021-05-30 12:32:51 +03:00
Jacek Sieka df7bc87af5 Pre-compute slot transition for clearance state
This way we perform the expensive epoch processing before the block
arrives.

Of course, this may lead to speculative misses which in turn lead to
replays - it's likely that in the case of a miss, we'll see a replay
regardless.
2021-05-30 12:04:09 +03:00
tersec aa6177814e
re-enable state_transition() rollback in phase0 (#2613) 2021-05-29 03:54:56 +00:00
Jacek Sieka 2df8a3b28d
add more block processing durations (#2611) 2021-05-28 21:03:20 +02:00
Zahary Karadjov 0afb5be6b4 A simple script for fetching detailed deposit data 2021-05-28 19:34:20 +03:00
Jacek Sieka 7f52ffb8d9
clean up block processing (#2610)
* gossip_to_consensus -> block_processor (it's processing only blocks,
but not only from gossip)
* measure queue and validation time for blocks
* measure assignment and state loading times for updateStateData
* avoid some unnecessary block copies in block sync
* warn that database is corrupt if we hit tail without a state
2021-05-28 19:34:00 +03:00
tersec c06ffc7804
proposed structure for altair (#2323)
* proposed structure for hf1

* refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim

* hf1 is Altair

* some syncing with alpha 2

* adjust epoch processing to disambiguate access to RewardFlags

* relocate StateData to stay consistent with meaning phase 0 StateData

* passes v1.1.0 alpha 5 SSZ consensus object tests

* Altair block header test fixtures work

* fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work

* deposit operation Altair test fixtures work

* slot sanity and all but a couple epoch transition tests switched to Altair

* attestation Altair test fixtures work

* Altair block sanity test fixtures work

* add working altair sync committee tests

* improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
tersec 46c5a0110a
log doppelganger attestation signature; rm withState.HashedBeaconState uses (#2608) 2021-05-28 15:51:15 +03:00
Jacek Sieka 5be1f8bf93
Revert "increase sqlite cache size (#2607)"
This reverts commit f55c4bc402.
2021-05-28 11:35:23 +02:00
Jacek Sieka f55c4bc402
increase sqlite cache size (#2607)
This is a test to see how the prater nodes react
2021-05-27 21:16:04 +02:00
Jacek Sieka d16da06c92 ncli_db: validator performance database tool
Record attestation performance per epoch in sqlite database
2021-05-27 19:14:26 +03:00
cheatfate 90e3fb246f Fix assertion crash when posting block via REST API. 2021-05-27 18:59:30 +03:00
Jacek Sieka ab70f371e1
Revert "create new database in separate file (#2596)" (#2604)
This reverts commit eebc828778.

Adding a separate file turns out not to be enough. This PR reverts the
separate file change.

Another theory is that the large kvstore table causes cache thrashing -
all database connections share a common page cache which would explain
the poor performance of the separate file solution.
2021-05-27 12:59:42 +02:00
Mamy André-Ratsimbazafy d05c9dbcf4 improve batch crypto sanity checks error reporting 2021-05-26 18:17:12 +03:00
Jacek Sieka 18d26071d8
hotfix migration
should have just kept the full copy-pasted dbseq init
2021-05-26 09:56:21 +02:00
Jacek Sieka eebc828778
create new database in separate file (#2596)
The V1 table structure shows great improvements in performance, but if
there's an old `kvstore` without rowid:s, these benefits are nullified:
reorgs during writes and deletes remain expensive (even if the
degradation is reduced somewhat).

This PR creates the tables in a new file instead, and uses the old file
as a read-only store - this has several interesting properties:

* the old database is left completely untouched - this guarantees that
downgrades work smooth (they'll only need to resync their missing
portions)
* starting sync after this PR means only a v1 database is created
* v0 databases stick around - no migration is performed (for now)

Future PR:s can introduce migration of the data from one database to
another - a simply copy will take hours which is downtime we want to
avoid - at that point, it might make sense to migrate straight to era
files instead.
2021-05-26 09:07:18 +02:00
tersec 3a59dc16c2
use v1.1.0 phase 0 test vectors (#2592)
* use v1.1.0 phase 0 test vectors

* re-export process_final_updates()
2021-05-24 10:42:40 +02:00
Jacek Sieka 584fcd50c1
ncli: fix inclusion distance statistic (#2587) 2021-05-24 10:40:45 +02:00
tersec 0b0bfd1de0
use StateData in place of BeaconState outside state transition code (#2551)
* use StateData in place of BeaconState outside state transition code

* propagate more StateData usage

* remove withStateVars().state

* wrap get_beacon_committee(BeaconState, ...) as gbc(StateData, ...)

* switch makeAttestation() to use StateData

* use StateData wrapper/dispatcher for get_committee_count_per_slot()

* convert AttestationCache.init(), weak subjectivity functions, and updateValidatorMetrics()

* add get_shuffled_active_validator_indices(StateData) and get_block_root_at_slot(StateData)

* switch makeAttestationData() to StateData

* sync AllTests-mainnet.md after rebase
2021-05-21 09:23:28 +00:00
cheatfate be5661eebc Fix /api/eth/v1/validator/aggregate_and_proofs call 2021-05-21 09:47:13 +03:00
cheatfate 6869a1ad33 Convert all the tuples to objects because of de-serialization of tuples issue. 2021-05-21 09:47:13 +03:00
cheatfate d61b5f46e1 Added missing redirection for POST /eth/v1/beacon/pool/attestations. 2021-05-20 19:42:08 +03:00
cheatfate d4970e07d2 Use atSlot() instead of getBlockByXXX(). 2021-05-20 19:42:08 +03:00
Jacek Sieka 8dbd796401 prune `validatorIndexFromPubKey` table 2021-05-20 14:10:23 +03:00
Zahary Karadjov acee2d3709
Remove an inappropriate merge artifact 2021-05-20 13:55:33 +03:00
Zahary Karadjov dc49a51654
Merge stable into unstable (take 2) 2021-05-20 13:52:09 +03:00
Zahary Karadjov b7aa30adfd
Merge stable into unstable 2021-05-20 13:50:40 +03:00
tersec d8bb91d9a9
partially integrate eth1 merge changes (#2548)
* partially integrate eth1 merge changes

* use hexToSeqByte() and validate execution engine opaque transaction length

* remove incorrect REST serialization code
2021-05-20 10:44:13 +00:00
Eugene Kabanov 5b5ea2e813
Fix integer overflow issue in sync_manager. (#2564)
* Make Refactor rewind point assignment more concrete.

* Fix overflow issue in getRewindPoint().
Add tests.
2021-05-18 12:25:14 +02:00
Eugene Kabanov cf06c4e87e
Make REST server more compatible with Lighthouse and Teku validator clients. (#2575)
* Allow REST server to parse arrays with comma delimiter.

* Fix compilation issues because of new presto framework.
2021-05-18 12:24:57 +02:00
Zahary Karadjov 2cb1396969
Log the slashing DB pruning time 2021-05-17 21:42:28 +03:00
Zahary Karadjov f6a1f602b4
Handle changes in the latest version of Confutils 2021-05-17 21:42:28 +03:00
Zahary Karadjov b9924214ab
Better error-handling for the slashingdb import/export feature
* Error when specifying an invalid --data-dir (or --validator-dir)
* Error when entering an invalid validator public key (e.g. invalid hex value)
* Warning when attempting to export a validator not present in the local database

Some unnecessary remains of the v1 mode has been removed as well
2021-05-17 21:42:23 +03:00
Jacek Sieka 97f4e1fffe
Db1 cont (#2573)
* Revert "Revert "Upgrade database schema" (#2570)"

This reverts commit 6057c2ffb4.

* ssz: fix loading empty lists into existing instances

Not a problem earlier because we didn't reuse instances

* bump nim-eth

* bump nim-web3
2021-05-17 18:37:26 +02:00
Zahary Karadjov 5c313b958e
Simplify the slashing db import/export CLI 2021-05-17 17:12:03 +03:00
tersec 6057c2ffb4
Revert "Upgrade database schema" (#2570)
This reverts commit 22ddf74752.
2021-05-17 06:34:44 +00:00
Mamy André-Ratsimbazafy dacc508992
slashing import integrated in NBC 2021-05-16 21:48:38 +03:00
Mamy André-Ratsimbazafy 0574531c43
add restore from slashing DB 2021-05-16 21:45:24 +03:00
Jacek Sieka 895ccd1c95 clean up imports (#2557) 2021-05-14 20:08:07 +03:00
Jacek Sieka 22ddf74752 Upgrade database schema
The `kvstore` design we're using now turns out to not be the best way to
use `sqlite` - in particular, there are some significant benefits to
using rowid in certain situations and to keep data in separate tables.

With this branch, there are massive improvements in startup time
(seconds instead of minutes) and state/block storage and pruning times
(milliseconds instead of seconds) - these improvements can in particular
be seen on slow drives and translate directly into better attestation
performance.

* update kvstore to new keyspace design
* remove `DirStoreRef` and the hidden `--state-db-kind` option - this
was an experiment to store large blobs in files, but with the new
kvstore, there's no compelling reason to do so
* remove `DbMap` - unused and would need updating for new keyspace
design
* introduce separate tables for each data type (blocks, states etc)
* remove "WITHOUT ROWID" pessimization for tables with large blobs
* close DbSeq statements explicitly (and earlier)
* store beacon block summaries in separate table, without SSZ
compression and load them all with single query on startup
* stop storing backwards compat full states
* mark genesis beacon block as trusted
* avoid faststreams when loading SSZ data
* remove `DisagreementBehavior` (unused)
2021-05-14 20:05:23 +03:00
Zahary Karadjov 0a91c3c592
Prepare for the v1.3.0 release (more to come) 2021-05-12 16:28:10 +03:00
Jacek Sieka 0022015a91
clean up imports (#2557) 2021-05-12 14:31:02 +02:00
Jacek Sieka 0a477eb2d3
fix subnet logic (#2555)
This PR decreases the lead subscription time which should help
decrease bandwidth usage and CPU making the subscription for future
aggregation happen a bit later. There's room for more tuning here,
probably.

* fix missing negation from in #2550
* fix silly bitarray issues
* decrease subnet lead subscription time
* log all subnet switching source data
* rename subnet trackers to refer to stability and aggregate subnets
* more tests
2021-05-11 22:03:40 +02:00
Mamy Ratsimbazafy 149ff49c8e
Remove correlated queries in finalization pruning, all use indexes (#2554) 2021-05-11 10:41:37 +02:00
Mamy Ratsimbazafy e6b559a35a
Slashing db pruning [Merge only after v2 has been default for 1 noticeable release] (#2452)
* Enable slashing DB pruning

* integrate slashing DB pruning with onSlotEnd

* rebase tests
2021-05-10 16:32:28 +02:00
Jacek Sieka 867d8f3223
Perform attestation check before broadcast (#2550)
Currently, we have a bit of a convoluted flow where when sending
attestations, we start broadcasting them over gossip then pass them to
the attestation validation to include them in the local attestation pool
- it should be the other way around: we should be checking attestations
_before_ gossipping them - this serves as an additional safety net to
ensure that we don't publish junk - this becomes more important when
publishing attestations from the API.

Also, the REST API was performing its own validation meaning
attestations coming from REST would be validated twice - finally, the
JSON RPC wasn't pre-validating and would happily broadcast invalid
attestations.

* Unified attestation production pipeline with the same flow for gossip,
locally and API-produced attestations: all are now validated and entered
into the pool, then broadcast/republished
* Refactor subnet handling with specific SubnetId alias, streamlining
where subnets are computed, avoiding the need to pass around the number
of active validators
* Move some of the subnet handling code to eth2_network
* Use BitArray throughout for subnet handling
2021-05-10 09:13:36 +02:00
Jacek Sieka 646923c3dd
add attestation stats tool to ncli_db (#2539)
This also makes future efforts to provide metrics and logs for
attestation efficiency easier

* Export rewards from epoch transition
* Use less memory for reward calculation (bool -> set[enum], field
alignment)
* Reuse reward memory when replaying, avoiding spike
* Allow replaying any range in ncli_db benchmark
2021-05-07 13:36:21 +02:00
Jacek Sieka 5cd5da74c4
Gossipsub unsubscribe fixes (fixes #2540) (#2545) 2021-05-07 06:31:28 +02:00
tersec 1d6c8ee9ab
store full state 4x less often (#2542) 2021-05-06 07:36:18 +02:00
tersec dd43a2c3b0
bump nim-eth2-scenarios to get merge SSZ test vectors (#2541) 2021-05-05 15:35:36 +00:00
Jacek Sieka 427c0f307c
avoid extraneous hash root calculation (#2537)
When applying a block, we'll currently compute a state root for the
state after slot processing but before block processing - this is
unnecessary when a block is being applied because the intermediate state
root is never observed.
2021-05-05 08:54:21 +02:00
Jacek Sieka 4d74c742da
move ENRForkID into `spec` (#2538)
* move ENRForkID into `spec`

also get rid of strformat in topic formation and fix some case
discrepancies

* also move `Eth2Metadata`
2021-05-04 17:28:48 +02:00
Jacek Sieka efdf759cc0
avoid some slashing protection queries (#2528)
This PR reduces the number of database queries for slashing protection
from 5 reads and 1 write to 2 reads and 1 write in the optimistic case.

In the process, it removes user-level support for writing the database
in the version 1 format in order to simplify the code flow, and prevent
code rot. In particular, the v1 format was not covered by any unit tests
and has no advantages over v2. The concrete code to read and write it
remains for now, in particular to support upgrades from v1 to v2.

The branch also removes the use of concepts which doesn't work with
checked exceptions - in particular, this highlights code that both
raises exceptions and returns error codes, which could be cleaned up in
the future.

* Cache internal validator ID
* Rely on unique index to check for trivial duplicate votes
* Combine two surround vote queries into one
* Combine API for checking and registering slashing into single function

The slashing DB is normally not a bottleneck, but may become one with
high attached validator counts.
2021-05-04 15:17:28 +02:00
tersec 290b889ce6
non-intrusive, novel portions of merge (#2535) 2021-05-04 11:54:19 +00:00
tersec e0f4d28116
rename initialize_beacon_state to initialize_beacon_state_from_eth1 (#2536) 2021-05-04 12:19:11 +02:00
Ștefan Talpalaru 51fde853d5
Merge branch 'stable' into unstable 2021-05-03 15:26:56 +02:00
Zahary Karadjov f369682b3f
v1.2.2 2021-05-03 11:13:12 +03:00
Chris Hobcroft a3b9990beb
Fixed confusing typo (#2533)
"One or more" appears to be a better description than "One of more"
2021-05-02 17:01:49 +02:00
Zahary Karadjov e60dd8e4f6 Stronger validation for third-party Eth1Data votes 2021-05-01 10:24:31 +03:00
Jacek Sieka ce49da6c0a
Introduce unittest2 and junit reports (#2522)
* Introduce unittest2 and junit reports

* fix XML path

* don't combine multiple CI runs

* fixup

* public combined report also

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-28 18:41:02 +02:00
Dustin Brody 7f42d38219 rename initialize_beacon_state{_from_eth1,}; suppress warnings when doppelganger detection disabled 2021-04-28 00:12:41 +03:00
Eugene Kabanov 0c3302b826
REST API test framework and tests. (#2492)
* REST API test framework and tests.

* Fix ValidatorIndex tests to properly handle int32, but not uint32 values.

* Fix tests to follow latest REST fixes.

* refactor restapi.sh

and add it to the test suite

* Fix issues.
Add delay timeout which is required.

* Fix restapi.sh script for Windows.

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-27 23:46:24 +03:00
Jacek Sieka 7dba1b37dd
remove attestation/aggregate queue (#2519)
With the introduction of batching and lazy attestation aggregation, it
no longer makes sense to enqueue attestations between the signature
check and adding them to the attestation pool - this only takes up
valuable CPU without any real benefit.

* add successfully validated attestations to attestion pool directly
* avoid copying participant list around for single-vote attestations,
pass single validator index instead
* release decompressed gossip memory earlier, specially during async
message validation
* use cooked signatures in a few more places to avoid reloads and errors
* remove some Defect-raising versions of signature-loading
* release decompressed data memory before validating message
2021-04-26 22:39:44 +02:00
Zahary Karadjov 8cd570fc52
Merge branch 'stable' into unstable 2021-04-20 18:17:54 +03:00
Zahary Karadjov 2faaf391a0
v1.2.1 2021-04-20 16:55:37 +03:00
Jacek Sieka 54d6884c89
fix sync issue when upgrading from 1.1.0-inited db
This patch writes a full genesis state to `kvstore` if one was missing,
which fixes 1.2.0 restarting sync when upgrading from 1.1.0, or when
downgrading to a pre-1.1.0 release.
2021-04-20 16:55:18 +03:00
Zahary Karadjov 9b57535f33
Merge branch 'stable' into unstable 2021-04-19 19:55:20 +03:00
Zahary Karadjov e1a8049ed5
v1.2.0 2021-04-19 19:54:24 +03:00
Ștefan Talpalaru 5b32f7ca17 Nim-1.2.12 2021-04-19 10:53:58 +03:00
Jacek Sieka daf98e4330 fix committee vs subnet confusion 2021-04-18 14:17:45 +03:00
tersec 99fccaee6e
more abstraction over BeaconState (#2509)
* more abstraction over BeaconState

* use HashedBeaconState copy of htr
2021-04-16 08:49:37 +00:00
Ștefan Talpalaru 85acf55ad8
remove duplicate metric (#2507) 2021-04-15 13:42:40 +02:00
Jacek Sieka f1f424cc2d attestation processing speedups
* avoid creating indexed attestation just to check signatures - above
all, don't create it when not checking signatures ;)
* avoid pointer op when adding attestation to pool
* better iterator for yielding attestations
* add metric / log for attestation packing time
2021-04-14 21:51:17 +03:00
Zahary Karadjov b0912b8eaf
Remove some unused parameters 2021-04-14 16:15:22 +03:00
tersec 050e3ac48b
abstract over more BeaconState usage (#2496) 2021-04-14 11:34:35 +02:00
Dmitriy Ryajov 3e3137ec14 pass max peers to builder 2021-04-14 10:15:29 +03:00
Jacek Sieka 4ed2e34a9e Revamp attestation pool
This is a revamp of the attestation pool that cleans up several aspects
of attestation processing as the network grows larger and block space
becomes more precious.

The aim is to better exploit the divide between attestation subnets and
aggregations by keeping the two kinds separate until it's time to either
produce a block or aggregate. This means we're no longer eagerly
combining single-vote attestations, but rather wait until the last
moment, and then try to add singles to all aggregates, including those
coming from the network.

Importantly, the branch improves on poor aggregate quality and poor
attestation packing in cases where block space is running out.

A basic greed scoring mechanism is used to select attestations for
blocks - attestations are added based on how much many new votes they
bring to the table.

* Collect single-vote attestations separately and store these until it's
time to make aggregates
* Create aggregates based on single-vote attestations
* Select _best_ aggregate rather than _first_ aggregate when on
aggregation duty
* Top up all aggregates with singles when it's time make the attestation
cut, thus improving the chances of grabbing the best aggregates out
there
* Improve aggregation test coverage
* Improve bitseq operations
* Simplify aggregate signature creation
* Make attestation cache temporary instead of storing it in attestation
pool - most of the time, blocks are not being produced, no need to keep
the data around
* Remove redundant aggregate storage that was used only for RPC
* Use tables to avoid some linear seeks when looking up attestation data
* Fix long cleanup on large slot jumps
* Avoid some pointers
* Speed up iterating all attestations for a slot (fixes #2490)
2021-04-13 20:24:02 +03:00
Dustin Brody 398c151b7d
revert change 2021-04-13 18:50:06 +02:00
Dustin Brody d6fa4d06bc
abstract over more BeaconState usage 2021-04-13 18:47:44 +02:00
cheatfate c2c3fdb471 Simplification and compiler error fix. 2021-04-13 17:07:41 +03:00
cheatfate fc5ca8ec35 Address #2487 and #2488. 2021-04-13 17:07:41 +03:00
cheatfate 79d4735355 Address #2491. 2021-04-13 17:07:41 +03:00
cheatfate b0a3008a1b Address #2486. 2021-04-13 17:07:41 +03:00
cheatfate 3417871370 Address #2489. 2021-04-13 17:07:41 +03:00
cheatfate 477decbcf5 Address #2490. 2021-04-13 17:07:41 +03:00
tersec 498c998552
abstract over most withStateVars/withState state var usage (#2484)
* abstract over most withStateVars/withState state var usage

* cleanups
2021-04-13 15:05:44 +02:00
Dmitriy Ryajov 1d6518eaaa use `new` instad of `init` 2021-04-12 20:28:39 +03:00
Dmitriy Ryajov 920de91c37 use libp2p builder 2021-04-12 20:28:39 +03:00
cheatfate 9cd946a192 Proper fix for ValidatorIndex supported values. 2021-04-09 21:42:13 +03:00
cheatfate af1a4d0c05 Simplify validator_id filter's logic. 2021-04-09 21:42:13 +03:00
cheatfate 5268d03b68 Finish error message to constants process. 2021-04-09 21:42:13 +03:00
cheatfate b3ad573b1f Make error messages to be constants. 2021-04-09 21:42:13 +03:00
cheatfate 934cc453d9 Add "BETA version" note to the `--rest` server command line parameters descriptions. 2021-04-09 21:42:13 +03:00
cheatfate d443801818 Fix /api/eth/v1/validator/duties/attester/{epoch} request body ValidatorIndex issues. 2021-04-09 21:42:13 +03:00
cheatfate c56cb0a485 Fix /api/eth/v1/node/peers arguments handling.
Fix maximum Epoch handling.
2021-04-09 21:42:13 +03:00
cheatfate 23b2e663f1 Fix /api/eth/v1/node/peers response. 2021-04-09 21:42:13 +03:00
cheatfate 55cbb82ab5 Fix /api/eth/v1/node/peer_count response. 2021-04-09 21:42:13 +03:00
cheatfate 5885068c63 Rebase and fixes. 2021-04-09 21:42:13 +03:00
cheatfate dba69f0ddf Add partial implementation for /api/eth/v1/beacon/headers 2021-04-09 21:42:13 +03:00
cheatfate 7a8e265251 Fix ValidatorIndex issues with workaround. 2021-04-09 21:42:13 +03:00
cheatfate 878d0e7337 Fix validator filtering. 2021-04-09 21:42:13 +03:00
cheatfate a6147f7b9d Fix `/eth/v1/config/fork_schedule` path. 2021-04-09 21:42:13 +03:00
cheatfate 64ddf4490a Add prefixes "Rest" to public tuples. 2021-04-09 21:42:13 +03:00
cheatfate 80e79aef97 Add TODO comments for missing implementations.
Change default REST port to use 5052 (Lighthouse).
Add missing checks for maximum amount of validator ids.
2021-04-09 21:42:13 +03:00
cheatfate 2cf0d3d831 Simplification of POST/api/nimbus/v1/graffiti and GET/api/nimbus/v1/graffiti call implementation. 2021-04-09 21:42:13 +03:00
Zahary Karadjov 5266db3485 Add a POST /api/nimbus/v1/graffiti API 2021-04-09 21:42:13 +03:00
cheatfate ec48e30ea8 Fix fork/schedule procedure comment. 2021-04-09 21:42:13 +03:00
cheatfate a1303f3e50 Change json serialization framework.
Make all calls which are possible.
2021-04-09 21:42:13 +03:00
cheatfate b7f36be73c Attempt to find proper readValue. 2021-04-09 21:42:13 +03:00
cheatfate 9de65fa293 Fixing issues after bump. 2021-04-09 21:42:13 +03:00
cheatfate 72695dd62a Rest API initial implementation. 2021-04-09 21:42:13 +03:00
cheatfate c4d891f583 Fix sync_manager.nim to return proper status.
Bump REST API dependencies.
2021-04-09 21:42:13 +03:00
tersec 79bb0d5379
only deserialize attestation and aggregation gossiped signatures once (#2472)
* only deserialize attestation and aggregation gossiped signatures once

* re-indent some aggregate checks into block scope

* spelling

* remove debugging assertion

* put part of gossip validation back into block context

* attestation pool test signature loading isn't so unsafe, and exportRaw isn't free

* remove more development doAsserts; don't exportRaw in loops
2021-04-09 14:59:24 +02:00
Zahary Karadjov 9776fbfe17
Merge branch 'version-1.1.0' into unstable 2021-04-08 20:50:06 +03:00
Zahary Karadjov 61669f269d Clarify that the deposit cretion tools are intended for testnets 2021-04-08 19:58:22 +03:00
Zahary Karadjov 40859e4cdc Restore the deposits create command 2021-04-08 19:58:22 +03:00
Jacek Sieka 7165e0ac31
Reset cached indices when resetting cache on SSZ read (#2480)
* Reset cached indices when resetting cache on SSZ read

When deserializing into an existing structure, the cache should be
cleared - goes for json also. Also improve error messages.
2021-04-08 13:11:04 +03:00
Zahary Karadjov ba59dd85cd Support for fallback web3 providers; Fix resource leaks on Eth1 monitor restarts 2021-04-08 12:46:42 +03:00
tersec d3cad92693
remove some BeaconState use and abstract over other uses (#2482)
* remove some BeaconState use and abstract over other uses

* remove out-of-context comment
2021-04-08 08:24:25 +00:00
Zahary Karadjov 7451c4171f
v1.1.0-rc; Enable v2 doppelganger detection by default 2021-04-07 00:38:47 +03:00
Jacek Sieka beceb060c4
Write state diffs to separate table (and experimentally, files instead of db) (#2460) 2021-04-06 21:56:45 +03:00
Jacek Sieka 10d99c166c
print attestation/aggregate drop notice once per slot (#2475)
* add metrics for queue-related drops
* avoid importing beacon node conf in processor
2021-04-06 13:59:11 +02:00
tersec 8d7792e6e9
add Altair domains and participation flags; clean up imports (#2462) 2021-04-04 16:24:45 +00:00
Mamy Ratsimbazafy 6b13cdce36
Batch attestations (#2439)
* batch attestations

* Fixes (but now need to investigate the chronos 0 .. 4095 crash similar to https://github.com/status-im/nimbus-eth2/issues/1518

* Try to remove the processing loop to no avail :/

* batch aggregates

* use resultsBuffer size for triggering deadline schedule

* pass attestation pool tests

* Introduce async gossip validators. May fix the 4096 bug (reentrancy issue?) (similar to sync unknown blocks #1518)

* Put logging at debug level, add speed info

* remove unnecessary batch info when it is known to be one

* downgrade some logs to trace level

* better comments [skip ci]

* Address most review comments

* only use ref for async proc

* fix exceptions in eth2_network

* update async exceptions in gossip_validation

* eth2_network 2nd pass

* change to sleepAsync

* Update beacon_chain/gossip_processing/batch_validation.nim

Co-authored-by: Jacek Sieka <jacek@status.im>

Co-authored-by: Jacek Sieka <jacek@status.im>
2021-04-02 16:36:43 +02:00
Jacek Sieka f821bc878e
Remove `-d:insecure` compile option (#2468)
With metrics running on top of chronos, the metrics server no longer
needs to be compiled in conditionally - it remains disabled by default.
2021-04-01 14:44:11 +02:00
Jacek Sieka e746cb09c2
Fix block number request (#2470)
leading 0's not tolerated
2021-04-01 14:43:03 +02:00
Jacek Sieka 8f441588cd
Add a few help defaults, format all of them the same (#2284) (#2467)
No support in confutils, so so this code simply hardcodes it
2021-04-01 12:28:57 +00:00
Jacek Sieka 3cd7cebc7c
Fix block dag pruning frequency (#2469)
Should always prune after finality change but not more than once
2021-04-01 13:26:17 +02:00
tersec bd8b60f8c8
use epochref for get_committee_assignments() to avoid repeated shuffling (#2463)
* use epochref for get_committee_assignments() to avoid repeated shuffling

* remove unnecessary imports and StateCache() construction
2021-03-30 15:01:47 +00:00
Mamy Ratsimbazafy a9938a2067
Fix pruning time display (#2461)
* Fix pruning time display

* remove import
2021-03-30 09:40:28 +02:00
tersec 49a5667288
update some v1.1.0 alpha1 to alpha2 (#2457)
* update some v1.1.0 alpha1 to alpha2

* remove unused getDepositMessage overload and move other out of datatypes/base

* bump nim-eth2-scenarios to download v1.1.0-alpha.2 test vectors

* construct object rather than result
2021-03-29 19:17:48 +00:00
Zahary Karadjov 1af07ad124
Simple RPC for modifying the Graffiti bytes 2021-03-26 19:42:20 +02:00
Jacek Sieka 74732a23fe
json cleanups (#2456)
* move json-rpc specific marshalling to rpc
* serialize Epoch/Slot with cast to avoid Defect
* avoid a few eth1 deps
* simplify imports
2021-03-26 15:11:06 +01:00
Jacek Sieka 2695cfa864
EH cleanup (#2455)
almost 100% raises in nimbus-eth2 now!

* fix some rare exception-related crashes in json-rpc
2021-03-26 07:52:01 +01:00
Mamy Ratsimbazafy 596cf3d674
Fix comment about backpressure handling in verif queue [skip ci] (#2453) 2021-03-25 15:09:53 +01:00
Zahary Karadjov 2eacfc4685 Bump modules to take advantage of the new Json format flavors support
Since quite a lot of additional procs were now compiled as generics, this lead to compiler bugs that had
to be worked-around:

* The `Domain` type was renamed to `Eth2Domain` to avoid compilation errors
  due to conflicts with `nativesockets.Domain`.
  Similarly, `eth2_network.KeyPair` was renamed to `NetKeyPair`.

* A new more robust version of `hexToByteArray` was added to stew
2021-03-25 09:37:35 +02:00
Jacek Sieka 8b76ceed52
Fix minor exception effect issues (#2448)
Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.
2021-03-24 17:20:55 +01:00
Kim De Mey a4a2c1c0e1
Add discovery query with forkId and attnets filter (#2420) 2021-03-24 11:48:53 +01:00
tersec 36311bfc05
incorporate proposals into nextActionWait; switch some proc to func (#2438) 2021-03-24 10:05:04 +00:00
Zahary Karadjov a727fc8e52
Merge stable into unstable 2021-03-24 03:24:20 +02:00
Zahary Karadjov 8391d3f83e
v1.0.12 2021-03-24 03:23:14 +02:00
kdeme adcae8d416 Avoid dialing peers that will be connection limited by libp2p 2021-03-23 14:46:52 +02:00
tersec 3076f5c3b6
rm std/random from beacon_chain and rm attestation timing randomness (#2442)
* remove added attestation timing randomness

* remove os/random from rest of beacon_chain, primarily deposit_contract

* remove scaffolding

* randomize std/random seed in beacon node and validator client

* use CSPRNG to more securely seed std/random
2021-03-23 06:57:10 +00:00
Giovanni Petrantoni 9558946df4
Expose gossip direct peers (#2246)
* Expose gossip direct peers

* direct-peer fixes

* add explicit info about direct peer

* build fixes wip
2021-03-23 15:10:17 +09:00
Zahary Karadjov 5fd82ddaf5
Merge stable into unstable 2021-03-22 23:13:05 +02:00
Zahary Karadjov 25695d1e4f
v1.0.11 2021-03-22 19:25:34 +02:00
tersec 85289269d1
increase block proposal speed with many validators (#2423)
* increase block proposal speed with many validators

* document CookedSig rationale
2021-03-22 19:02:27 +02:00
tersec dfd99ec943
Altair (HF1/v1.1.0) minimal and mainnet presets/constants (#2444)
* Altair mainnet & minimal presets

* std/math not used
2021-03-22 14:44:45 +00:00
Mentor Palokaj 895fe4baf1
Make it clear that the default port for ETH2 is 9000 udp/tcp (#2421)
* Add default port info

* Add default port info

* Edit CLI and docs to reflect 9000 port

* Apply suggestions from code review

Co-authored-by: Jacek Sieka <arnetheduck@gmail.com>
2021-03-22 08:22:56 +01:00
tersec baa43ae7f7
add stricter ABI-compatibility checks for punned types (#2434)
* add stricter ABI-compatibility checks for punned types

* enforce type isomorphism for each pair of cast types to eliminate potential safety hole
2021-03-19 12:30:20 +00:00
Jacek Sieka 01fe564e46
chronos-based metrics (#2432)
This opens up the road for removing `-d:insecure` for metrics.
2021-03-19 10:36:38 +01:00