status-im/nimbus-eth2
2
0
Fork 0
mirror of https://github.com/status-im/nimbus-eth2.git synced 2025-01-30 16:25:31 +00:00
Code Issues Projects Releases Wiki Activity
6,969 Commits 115 Branches 117 Tags 132 MiB
Commit Graph

1863 Commits

Author SHA1 Message Date
Jacek Sieka
b22d86e161
REST/JSON-RPC: speed up several requests (#3092) 
REST/JSON-RPC and a few more also invalidate caches unnecessarily,
similar to https://github.com/status-im/nimbus-eth2/pull/3089

* avoid copying validator on balance request
 2021-11-12 23:29:28 +01:00
Jacek Sieka
00bbc8e0fe
speed up epoch processing 6x+ (#3089) 
* speed up epoch processing 6x+

This change above all helps contain long replay times on epoch change, reorg
and deep history inspection via REST/RPC

* most effective balances don't actually change due to MAX_EFFECTIVE_BALANCE
* ditto for inactivity scores
* avoid signature check for trusted sync aggregates

pre:
```
./ncli_db --db:mainnet_0/db bench --start-slot=-3200
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed

    3468.621,        0.000,     3468.621,     3468.621,            1, Initialize DB
       0.357,        0.938,        0.171,       52.752,         3155, Load block from database
   15691.471,        0.000,    15691.471,    15691.471,            1, Load state from database
       6.100,        9.469,        0.033,      526.816,         3101, Advance slot, non-epoch
     579.131,        9.523,      566.936,      610.328,          100, Advance slot, epoch
      18.551,       16.317,       12.664,      136.668,         3155, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database load
       0.000,        0.000,        0.000,        0.000,            0, Database store
```

post:
```
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3488.541,        0.000,     3488.541,     3488.541,            1, Initialize DB
       0.369,        1.123,        0.183,       63.208,         3155, Load block from database
   13430.642,        0.000,    13430.642,    13430.642,            1, Load state from database
       6.522,        1.721,        0.034,       36.708,         3101, Advance slot, non-epoch
      89.074,        3.162,       83.573,      101.436,          100, Advance slot, epoch
      18.325,       18.346,       13.005,      145.040,         3155, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database load
       0.000,        0.000,        0.000,        0.000,            0, Database store
```

* Update beacon_chain/spec/state_transition_block.nim

Co-authored-by: zah <zahary@gmail.com>

* avoid copying validator data in accessor

```
    5291.227,        0.000,     5291.227,     5291.227,            1, Initialize DB
       0.436,        0.928,        0.138,       51.438,         3155, Load block from database
   11962.826,        0.000,    11962.826,    11962.826,            1, Load state from database
       6.477,        1.675,        0.037,       34.174,         3101, Advance slot, non-epoch
      76.633,        3.705,       71.106,       98.085,          100, Advance slot, epoch
      18.301,       18.593,       13.208,      149.153,         3155, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database load
       0.000,        0.000,        0.000,        0.000,            0, Database store
```

* work around compiler bug

Co-authored-by: zah <zahary@gmail.com>
 2021-11-11 20:24:29 +01:00
tersec
97ad5d49b0
kintsugi merge vector tests (#3072) 2021-11-10 13:41:02 +02:00
Jacek Sieka
ec650c7fd7
Support starting from altair (#3054) 
* Support starting from altair

* hide `finalized-checkpoint-` - they are incomplete and usage may cause
crashes
* remove genesis detection code (broken, obsolete)
* enable starting ChainDAG from altair checkpoints - this is a
prerequisite for checkpoint sync (TODO: backfill)
* tighten checkpoint state conditions
* show error when starting from checkpoint with existing database (not
supported)
* print rest-compatible JSON in ncli/state_sim
* altair/merge support in ncli
* more altair/merge support in ncli_db
* pre-load header to speed up loading
* fix forked block decoding
 2021-11-10 13:39:08 +02:00
kdeme
03a70fbf36 Use nim-ssz-serialization module and rm local ssz code 2021-11-10 13:37:24 +02:00
tersec
5c48982280
a dozen spec URL updates to v1.1.5 (#3078) 2021-11-10 08:12:41 +00:00
tersec
59db4b4f9c
remove intsets usage (#3075) 2021-11-10 02:18:52 +00:00
zah
3545d4d1e1
Address review comments in #3057 (#3069) 
* Address review comments in #3057

* reorder imports in rest_utils

maybe this will help with the mysterious serialization issues

Co-authored-by: Jacek Sieka <jacek@status.im>
 2021-11-09 20:21:36 +01:00
tersec
941eb609ba
update statediffs to work with Altair (#3061) 
* update statediffs to work with Altair
 2021-11-09 16:17:57 +00:00
Zahary Karadjov
ba42b2b316 Correct implementation of the /validator/duties/sync/{epoch} API 
According to the spec, this call should return the positions of
the specified validators within the sync committee. The existing
code was instead returning the indices of the sync sub-committees
where the validator is a member.
 2021-11-09 11:45:00 +02:00
tersec
48eba59971
manually-verified v1.1.5 spec URL updates (#3068) 2021-11-09 08:54:59 +00:00
tersec
2e868dc2ba
mass/mechanical update of 1.1.4 phase0 and altair spec URLs to 1.1.5 (#3067) 2021-11-09 07:40:41 +00:00
tersec
b76326f4b8
use v1.1.5 consensus spec test vectors (#3066) 2021-11-09 06:46:24 +00:00
tersec
95b0ecc5a2
only invalidate {current,previous}_epoch_participation flag cache once (#3063) 2021-11-09 02:44:02 +00:00
tersec
5c296c08b7
17 spec URL updates to v1.1.4 (#3060) 2021-11-08 10:30:52 +00:00
tersec
eb3ad25859
mass/mechanical update of 1.1.3 phase/altair spec URLs to 1.1.4 (#3058) 2021-11-08 06:18:10 +00:00
Dustin Brody
023a6a9387 use v1.1.4 test vectors 2021-11-07 21:42:11 +02:00
Jacek Sieka
ea0a191723
Better REST/RPC error messages (#3046) 
* Better REST/RPC error messages
* homogenise block logging (root first)
* homegenise message verification pipeline (verify in
`gossip_verification`, act in `eth2_processor`)
* use `subcommitteeIdx` consistently
* log each sent contribution
* fix block_sim
* fix block topic
* don't recalc root on gossip block validation
* move position loop into sync pool
 2021-11-05 17:39:47 +02:00
Jacek Sieka
a086cf01ac
altair fork handling cleanups (#3050) 
* fix stack overflow crash in REST/debug/getStateV2
* introduce `ForkyXxx` for generic type matching of `Xxx` across
branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) -
`Some` is already used for other types of type classes
* consolidate function naming in BeaconChainDB, use some generics
* import `forks.nim` from other spec modules and move `Forked*` helpers
around to resolve circular imports
* remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout
(less data shuffling between the types)
* fix several cases of states being stored on stack in tests, causing
random failures on some platforms
* remove reading json support from ncli - this should be ported to the
rest json reading instead (doesn't currently work because stack sizes)
 2021-11-05 08:34:34 +01:00
Jacek Sieka
98668dbf16
reuse pre-computed root for sig verification (#3048) 
no point re-hashing arguments
 2021-11-05 05:50:01 +00:00
Etan Kissling
51aef34be7
introduce strong typing for sync committee periods (#3040) 
The sync committee period used to be a plain `uint64`. With the light
client sync relying more heavily on them, it makes sense to introduce
a proper type for them, similar to how they are already used for `Slot`
and `Epoch`. This introduces such a `SyncCommitteePeriod` type.
Furthermore, some usage code dealing with those periods is cleaned up.
 2021-11-02 21:32:34 +01:00
Etan Kissling
1943258c66
extend REST config with more spec constants (#2962) 
So far, the REST config response did not include all spec constants.
The spec for `/eth/v1/config/spec` defines that the response should
include constants for all hard forks known by the beacon node. This
patch extends the corresponding response to include more constants.
 2021-11-02 19:23:31 +01:00
tersec
36e37bda40
v1.1.3 spec refs URLs (#3036) 2021-10-27 18:40:17 +00:00
Eugene Kabanov
7c9a6b7170
Improve chronos.Future tracking. (#2988) 
* Add `child_id` field.

* Fix json-rpc api call and bump chronos.

* Bump chronos master and fix compilation warnings.

* One more bump of `chronos`.

* add random import

* export rest_utils a bit more

Co-authored-by: Jacek Sieka <jacek@status.im>
 2021-10-27 14:01:11 +02:00
tersec
968a5b3bec
some spec refs to v1.1.3 (#3032) 2021-10-26 18:42:48 +00:00
tersec
8307e9c601
mechanical non-merge v1.1.2 to v1.1.3 spec URL updates (#3030) 2021-10-26 16:44:23 +00:00
Etan Kissling
d32c25ea32
use GeneralizedIndex type for spec constants (#3026) 
Spec constants currently use `uint16` instead of `GeneralizedIndex` to
define generalized indices in a merkle tree. Adjusted to use the latter
instead to avoid problems when adjusting `GeneralizedIndex`'s bit width.
 2021-10-26 12:30:36 +00:00
Jacek Sieka
421bf936ff
odds and ends (#3015) 
* `allSyncCommittees` => `allSyncSubcommittees`
* simplify `_snappy` topic generation (avoid pointless string copies)
* simplify gossip id generator (avoid pointless string copies)
* avoid redundant syncnet ENR updates
* simplify topic validation (allow only validated topics)
 2021-10-21 15:09:19 +02:00
nbc-bump-bot[bot]
29b26f3f6b
nim-libp2p auto bump (#2840) 
* auto-bump nim-libp2p

* Remove peer info for other peers

Not definitive, just to test the libp2p's unstable branch

* finish up Remove peer info for other peers

* getKey -> getPublicKey

* bump libp2p

* libp2p bump

Co-authored-by: = <Menduist@users.noreply.github.com>
Co-authored-by: Tanguy <tanguy@status.im>
 2021-10-21 13:01:29 +02:00
Etan Kissling
f5791122f6 widen allowed specs for validator client 
The validator client was only able to connect to beacon nodes exposing
the exact same set of spec constants that are locally known via their
config/spec REST API. However, that set of spec constants is dynamic.
As the validator client only requires a subset of relevant constants,
this may lead to compatible specs being rejected. This patch widens the
allowed specs by only verifying that the required set of constants are
present in the spec response, ignoring any spec constants that are not
locally known, and ignoring missing spec constants that are locally
known but not included by the remote beacon node when not relevant for
operation of the validator client.
 2021-10-21 00:59:05 +03:00
Jacek Sieka
9cf32c3748 clean up sync subcommittee handling 
* `SyncCommitteeIndex` -> `SyncSubcommitteeIndex`
* `syncCommitteePeriod` -> `sync_committee_period` (spec spelling)
* tighten period comparisons
* fix assert when validating committee message with non-altair state in
REST api
 2021-10-20 22:59:13 +03:00
Jacek Sieka
bf6ad41d7d add drop and sync committee metrics 
* use storeBlock for processing API blocks
* avoid double block dump
* count all gossip metrics at the same spot
* simplify block broadcast
 2021-10-20 18:20:12 +03:00
tersec
c0a2f1c98e
refactor executionPayload tests; reduce HashSet creation (#3003) 2021-10-20 13:36:38 +02:00
zah
44d762119c
Client definitions for the sync committee REST APIs (#3002) 2021-10-19 19:44:05 +02:00
tersec
a6ddb80779
refactor EF operations fixtures (#2998) 
* refactor EF operations fixtures

* for phase0/altair, check completeness of operations directories
 2021-10-19 16:38:41 +02:00
Jacek Sieka
c40cc6cec1 clean up fork enum and field names 
* single naming strategy
* simplify some fork code
* simplify forked block production
 2021-10-19 11:06:38 +03:00
tersec
ecf2dbbcf3
add Altair and Merge rewards tests (#2995) 2021-10-18 16:54:17 +02:00
Jacek Sieka
5b33783898
load altair state from database on startup (#2994) 
* fixes replay from last phase0 block on startup
* better forked block loading
 2021-10-18 14:32:54 +02:00
Eugene Kabanov
75a8fe3b2c
Increase REST server default limits. (#2991) 
* Increase REST server default limits.

* Fix compilation error.

* Fix tests.
 2021-10-18 09:14:44 +02:00
tersec
b66e7c8573
v1.1.3 tests; re-enable minimal-preset altair transition tests (#2993) 2021-10-16 07:17:07 +00:00
Eugene Kabanov
7319f150e8
Number of REST fixes. (#2987) 
* Initial commit.

* Fix path.

* Add validator keys to indices cache mechanism.
Move syncComitteeParticipants to common place.

* Fix sync participants order issue.

* Fix error code when state could not be found.
Refactor `state/validators` to use keysToIndices mechanism.

* Fix RestValidatorIndex to ValidatorIndex conversion TODOs.

* Address review comments.

* Fix REST test rules.
 2021-10-14 12:38:38 +02:00
Etan Kissling
05908859b7
support merkle multiproof verification (#2980) 
This introduces spec-compatible support to verify merkle multiproofs for
use, e.g., in light client syncing.
 2021-10-14 10:28:22 +02:00
tersec
6cc8757930
update 18 spec URLs to v1.1.2; switch from eth2.0-specs to consensus-specs (#2990) 2021-10-14 06:30:21 +00:00
tersec
10981639f1
update 27 spec URLs to v1.1.2 (#2989) 2021-10-13 16:49:06 +00:00
Jacek Sieka
f90b2b8b1f
reward accounting for altair+ (#2981) 
Similar to the existing `RewardInfo`, this PR adds the infrastructure
needed to export epoch processing information from altair+. Because
accounting is done somewhat differently, the PR uses a fork-specific
object to extrct the information in order to make the cost on the spec
side low.

* RewardInfo -> EpochInfo, ForkedEpochInfo
* use array for computing new sync committee
* avoid repeated total active balance computations in block processing
* simplify proposer index check
* simplify epoch transition tests
* pre-compute base increment and reuse in epoch processing, and a few
other small optimizations

This PR introduces the type and does the heavy lifting in terms of
refactoring - the tools that use the accounting will need separate PR:s
(as well as refinements to the exportred information)
 2021-10-13 16:24:36 +02:00
tersec
2eb9a608a4
add payloadId; add merge vector test script; remove consensusValidated (#2982) 2021-10-13 16:08:50 +02:00
Jacek Sieka
1da4192f3f
json-rpc: fix crashes on altair+ blocks in certain operations (#2983) 
For the v2 interfaces, REST is recommeded instead
 2021-10-13 13:20:18 +03:00
tersec
1f51331dd3
use specified execution_payload when constructing blocks; update Geth launch script (#2975) 
* use specified execution_payload when constructing blocks; update Geth launch script

* revert DepositContractState change
 2021-10-12 13:37:40 +02:00
Jacek Sieka
0c4bfb1911
test fixture cleanups (#2976) 
* "official" -> "scenarios", like the submodule
* fewer test binaries - various compile hacks have been improved over
time, test suite should follow
* remove obsolete bls tests - there are better test vectors in
nim-blscurve
* remove obsolete mentions of `ssz_testing`
* remove obsolete comments about proc vs globals, unittest2 already uses
proc's
 2021-10-12 13:36:52 +02:00
tersec
2ad1b7366a
update 62 spec URLs to v1.1.2 (#2979) 2021-10-12 10:17:37 +00:00