status-im/nimbus-eth2
2
0
Fork 0
mirror of https://github.com/status-im/nimbus-eth2.git synced 2025-01-09 13:56:23 +00:00
Code Issues Projects Releases Wiki Activity
6,341 Commits 108 Branches 116 Tags 242 MiB
Commit Graph

342 Commits

Author SHA1 Message Date
tersec
8307e9c601
mechanical non-merge v1.1.2 to v1.1.3 spec URL updates (#3030) 2021-10-26 16:44:23 +00:00
Jacek Sieka
9cf32c3748 clean up sync subcommittee handling 
* `SyncCommitteeIndex` -> `SyncSubcommitteeIndex`
* `syncCommitteePeriod` -> `sync_committee_period` (spec spelling)
* tighten period comparisons
* fix assert when validating committee message with non-altair state in
REST api
 2021-10-20 22:59:13 +03:00
Etan Kissling
05908859b7
support merkle multiproof verification (#2980) 
This introduces spec-compatible support to verify merkle multiproofs for
use, e.g., in light client syncing.
 2021-10-14 10:28:22 +02:00
tersec
6cc8757930
update 18 spec URLs to v1.1.2; switch from eth2.0-specs to consensus-specs (#2990) 2021-10-14 06:30:21 +00:00
tersec
10981639f1
update 27 spec URLs to v1.1.2 (#2989) 2021-10-13 16:49:06 +00:00
tersec
1f51331dd3
use specified execution_payload when constructing blocks; update Geth launch script (#2975) 
* use specified execution_payload when constructing blocks; update Geth launch script

* revert DepositContractState change
 2021-10-12 13:37:40 +02:00
Etan Kissling
711c08f804
add function to build merkle proofs (#2874) 
Adds a function that constructs a Merkle proof for a generalized index.
This will be used during light client sync to update light clients with
a new state (see NEXT_SYNC_COMMITTEE_INDEX / FINALIZED_ROOT_INDEX).
 2021-09-29 15:02:34 +02:00
tersec
1dc94aa36f
update 40 spec URLs to v1.1.0 (#2918) 2021-09-28 18:23:15 +00:00
tersec
aec5cf2b1b
update 31 spec reference URLs to v1.1.0 (#2910) 2021-09-28 07:46:06 +00:00
tersec
2b2846b468
implement forked merge state/block support (#2890) 
* implement forked state/block support

* merge support for containsOrphan; import cleanup; 80-column lines

* add merge block header operations and slot sanity fixture

* add epoch state transition tests; implement is_valid_gas_limit(), is_merge_block(), is_execution_enabled(), and compute_timestamp_at_slot()

* implement process_execution_payload() and add merge deposit operations tests

* add merge block sanity tests

* add merge case to syncCommitteeParticipants

* v1.1.0-beta.5 updates

* reduce getTestStates-based memory usage; don't try to REST-serialize ExecutionPayload transactions without underlying support

* add execution payload tests; switch var to let in tests/official/
 2021-09-27 14:22:58 +00:00
Zahary Karadjov
a71de3feda Address review comments 2021-09-14 12:12:49 +03:00
Zahary Karadjov
3e4faaa213 Add .editorconfig file and remove trailing newlines 2021-09-14 12:12:49 +03:00
Zahary Karadjov
add7daa30c Light client sync spec functions 2021-09-14 12:12:49 +03:00
tersec
97b0070774
add merge SSZ consensus object tests (#2858) 
* add merge SSZ consensus object tests

* add merge attestations, attester slashing, proposer slashing, and voluntary exit test fixtures

* throw catchable exception rather than assert on invalid SingleMemberUnion selector

* hash_tree_root can assert, because it's trusted data by that point

* re-assert on writing, since also trusted data

* beta.4 update

* implement upgrade_to_merge()
 2021-09-11 10:01:05 +02:00
tersec
0ad7216bc4
mass update from beta.3 spec refs to beta.4 spec refs (#2862) 2021-09-10 18:56:03 +00:00
tersec
ea722c6f6e
update some spec refs to beta.3 (#2853) 2021-09-08 14:49:04 +00:00
tersec
9e145afda3
update 29 Altair spec ref URLs to beta.3 (#2839) 2021-08-31 12:16:27 +00:00
tersec
eeba2869fc
add sync committee types and some helpers (#2829) 2021-08-28 09:00:00 +00:00
tersec
98547e0c6b
double altair epoch slot processing speed (#2811) 
* approximately double altair epoch processing speed

* don't redundantly clear state balances cache

* only invalidate altair state balances cache once in process_rewards_and_penalties()
 2021-08-24 22:09:03 +02:00
Jacek Sieka
a7a65bce42
disentangle eth2 types from the ssz library (#2785) 
* reorganize ssz dependencies

This PR continues the work in
https://github.com/status-im/nimbus-eth2/pull/2646,
https://github.com/status-im/nimbus-eth2/pull/2779 as well as past
issues with serialization and type, to disentangle SSZ from eth2 and at
the same time simplify imports and exports with a structured approach.

The principal idea here is that when a library wants to introduce SSZ
support, they do so via 3 files:

* `ssz_codecs` which imports and reexports `codecs` - this covers the
basic byte conversions and ensures no overloads get lost
* `xxx_merkleization` imports and exports `merkleization` to specialize
and get access to `hash_tree_root` and friends
* `xxx_ssz_serialization` imports and exports `ssz_serialization` to
specialize ssz for a specific library

Those that need to interact with SSZ always import the `xxx_` versions
of the modules and never `ssz` itself so as to keep imports simple and
safe.

This is similar to how the REST / JSON-RPC serializers are structured in
that someone wanting to serialize spec types to REST-JSON will import
`eth2_rest_serialization` and nothing else.

* split up ssz into a core library that is independendent of eth2 types
* rename `bytes_reader` to `codec` to highlight that it contains coding
and decoding of bytes and native ssz types
* remove tricky List init overload that causes compile issues
* get rid of top-level ssz import
* reenable merkleization tests
* move some "standard" json serializers to spec
* remove `ValidatorIndex` serialization for now
* remove test_ssz_merkleization
* add tests for over/underlong byte sequences
* fix broken seq[byte] test - seq[byte] is not an SSZ type

There are a few things this PR doesn't solve:

* like #2646 this PR is weak on how to handle root and other
dontSerialize fields that "sometimes" should be computed - the same
problem appears in REST / JSON-RPC etc

* Fix a build problem on macOS

* Another way to fix the macOS builds

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
 2021-08-18 20:57:58 +02:00
Jacek Sieka
7a622e8505
rework spec imports (#2779) 
The spec imports are a mess to work with, so this branch cleans them up
a bit to ensure that we avoid generic sandwitches and that importing
stuff generally becomes easier.

* reexport crypto/digest/presets because these are part of the public
symbol set of the rest of the spec types
* don't export `merge` types from `base` - this causes circular deps
* fix circular deps in `ssz/spec_types` - this is the first step in
disentangling ssz from spec
* be explicit about phase0 vs altair - longer term, `altair` will become
the "natural" type set, then merge and so on, so no point in giving
`phase0` special preferential treatment
 2021-08-12 13:08:20 +00:00
tersec
38ce948647
partial altair merge (#2735) 
* partial altair merge

* exclude still-in-flux sync committee data structures from partial merge

* undo the remaining sync_committee mention
 2021-07-26 09:51:14 +00:00
tersec
68ca600d89
update 20 more spec refs to v1.1.0-alpha.8 (#2690) 2021-07-01 12:13:59 +00:00
tersec
c9cec88a23
update some spec refs to v1.1.0-alpha.8 (#2689) 2021-07-01 08:55:16 +00:00
tersec
1c3314f08b
update to Altair as of v1.1.0-alpha.7 (#2649) 
* update to Altair as of v1.1.0-alpha.7

* introduce Altair types into attestation pool

* avoid allocating/copying pubkeys excessively in get_next_sync_committee()
 2021-06-14 17:42:46 +00:00
tersec
c06ffc7804
proposed structure for altair (#2323) 
* proposed structure for hf1

* refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim

* hf1 is Altair

* some syncing with alpha 2

* adjust epoch processing to disambiguate access to RewardFlags

* relocate StateData to stay consistent with meaning phase 0 StateData

* passes v1.1.0 alpha 5 SSZ consensus object tests

* Altair block header test fixtures work

* fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work

* deposit operation Altair test fixtures work

* slot sanity and all but a couple epoch transition tests switched to Altair

* attestation Altair test fixtures work

* Altair block sanity test fixtures work

* add working altair sync committee tests

* improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
 2021-05-28 15:25:58 +00:00
tersec
8d7792e6e9
add Altair domains and participation flags; clean up imports (#2462) 2021-04-04 16:24:45 +00:00
Zahary Karadjov
2eacfc4685 Bump modules to take advantage of the new Json format flavors support 
Since quite a lot of additional procs were now compiled as generics, this lead to compiler bugs that had
to be worked-around:

* The `Domain` type was renamed to `Eth2Domain` to avoid compilation errors
  due to conflicts with `nativesockets.Domain`.
  Similarly, `eth2_network.KeyPair` was renamed to `NetKeyPair`.

* A new more robust version of `hexToByteArray` was added to stew
 2021-03-25 09:37:35 +02:00
tersec
97f7284e51
bump spec refs from v1.0.0 to v1.0.1 and update copyright years (#2357) 2021-02-25 13:37:22 +00:00
Jacek Sieka
8a09286423
tune attestation params (#2301) 
* a little bit more priority to attestation processing
* better implementation for bytes_to_uint64
 2021-02-08 16:13:02 +01:00
Jacek Sieka
5713a3ce4c
performance fixes (#2259) 
* performance fixes

* don't mark tree cache as dirty on read-only List accesses
* store only blob in memory for keys and signatures, parse blob lazily
* compare public keys by blob instead of parsing / converting to raw
* compare Eth2Digest using non-constant-time comparison
* avoid some unnecessary validator copying

This branch will in particular speed up deposit processing which has
been slowing down block replay.

Pre (mainnet, 1600 blocks):

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3450.269,        0.000,     3450.269,     3450.269,            1, Initialize DB
       0.417,        0.822,        0.036,       21.098,         1400, Load block from database
      16.521,        0.000,       16.521,       16.521,            1, Load state from database
      27.906,       50.846,        8.104,     1507.633,         1350, Apply block
      52.617,       37.029,       20.640,      135.938,           50, Apply epoch block
```

Post:

```
    3502.715,        0.000,     3502.715,     3502.715,            1, Initialize DB
       0.080,        0.560,        0.035,       21.015,         1400, Load block from database
      17.595,        0.000,       17.595,       17.595,            1, Load state from database
      15.706,       11.028,        8.300,      107.537,         1350, Apply block
      33.217,       12.622,       17.331,       60.580,           50, Apply epoch block
```

* more perf fixes

* load EpochRef cache into StateCache more aggressively
* point out security concern with public key cache
* reuse proposer index from state when processing block
* avoid genericAssign in a few more places
* don't parse key when signature is unparseable
* fix `==` overload for Eth2Digest
* preallocate validator list when getting active validators
* speed up proposer index calculation a little bit
* reuse cache when replaying blocks in ncli_db
* avoid a few more copying loops

```
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3279.158,        0.000,     3279.158,     3279.158,            1, Initialize DB
       0.072,        0.357,        0.035,       13.400,         1400, Load block from database
      17.295,        0.000,       17.295,       17.295,            1, Load state from database
       5.918,        9.896,        0.198,       98.028,         1350, Apply block
      15.888,       10.951,        7.902,       39.535,           50, Apply epoch block
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```

* clear full balance cache before processing rewards and penalties

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3947.901,        0.000,     3947.901,     3947.901,            1, Initialize DB
       0.124,        0.506,        0.026,      202.370,       363345, Load block from database
      97.614,        0.000,       97.614,       97.614,            1, Load state from database
       0.186,        0.188,        0.012,       99.561,       357262, Advance slot, non-epoch
      14.161,        5.966,        1.099,      395.511,        11524, Advance slot, epoch
       1.372,        4.170,        0.017,      276.401,       363345, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```
 2021-01-25 13:04:18 +01:00
tersec
271df8b604
bump 1.0.0rc-0 spec refs to 1.0.0 (#1974) 2020-11-09 14:18:55 +00:00
Zahary Karadjov
14b2d4324d openarray -> openArray 2020-11-03 23:23:10 +02:00
tersec
1994ffe5a0
update 130+ spec references from v0.12.3 to v1.0.0-rc1 (#1854) 2020-10-12 08:59:24 +00:00
Zahary Karadjov
aed291128a Add support for starting from weak subjectivity checkpoints 
Also removes the `genesis.ssz` file stored in the data folder.
The `medalla-fast-sync` target has been adapted to use the new features.
 2020-10-07 09:32:03 +03:00
tersec
6398a43cc1
update 120+ beacon_chain and validator spec refs from v0.12.2 to v0.12.3 (#1740) 2020-09-24 19:04:10 +02:00
tersec
b3b578501a
mark comments documenting functions as a whole as such (#1613) 2020-09-08 08:54:55 +00:00
Jacek Sieka
2081c4f505
avoid unnecessary seq allocations (#1573) 
"sigh"
 2020-08-27 08:32:51 +02:00
tersec
bc6eefe31e
add --enable-logtrace argument to launch_local_testnet (#1502) 
* add --enable-logtrace argument to launch_local_testnet

* scan for all available logfiles

* remove specific filename references

* update v0.11.3 spec ref to v0.12.2
 2020-08-16 11:12:19 +02:00
tersec
dad3dd5809
update 111 v0.12.1 beacon chain spec refs to v0.12.2 (#1380) 2020-07-27 18:49:46 +02:00
Jacek Sieka
48cebc7157
spec cleanups (#1379) 
* add helper for beacon committee length (used for quickly validating
attestations)
* refactor some attestation checks to do cheap checks early
* validate attestation epoch before computing active validator set
* clean up documentation / comments
* fill state cache on demand
 2020-07-27 16:04:44 +00:00
tersec
20a2525390
v0.12.2 beacon chain protocol update (#1378) 2020-07-27 12:59:57 +02:00
tersec
90708a8287
Prefer converting int` to uint64 and switch foo.len.uint64 to .len64 (#1375) 
* avoid converting from uint64 to int, and where most feasible, int type conversion at all

* .len.uint64 -> .len64

* fix 32-bit compilation

* try keeping state_sim loop variable/bounds as int for 32-bit Azure

* len64 -> lenu64
 2020-07-26 20:55:48 +02:00
Jacek Sieka
e0a18a3105
cache beacon committee size calculation (#1363) 
* cache beacon committee size calculation

this fixes a bug in get_validator_churn_limit as well

* fix

* make committee counts consistently uint64

mixing feels like the worst of the two worlds
 2020-07-23 19:01:07 +02:00
Jacek Sieka
fb2f742972
Fork choice fixes 2 (#1356) 
* fork choice cleanup

* enable v2 pruning
* prefer `get_current_epoch`
* fix finalization check to use correct epoch

* small cleanups

* add `count_active_validators`
* remove misleading logs
* fix justified checkpoint slot calculation in rpc
 2020-07-22 23:01:44 +02:00
tersec
26e893ffc2
restore EpochRef and flush statecaches on epoch transitions (#1312) 
* restore EpochRef and flush statecaches on epoch transitions

* more targeted cache invalidation

* remove get_empty_per_epoch_cache(); implement simpler but still faster get_beacon_proposer_index()/compute_proposer_index() approach; add some abstraction layer for accessing the shuffled validator indices cache

* reduce integer type conversions

* remove most of rest of integer type conversion in compute_proposer_index()
 2020-07-15 12:44:18 +02:00
zah
0be77f9cbc
Restore compilation with custom presets (#1309) 
This also assigns precise types to the constants in the minimal
and mainnet presets in order to reduce the chance of compilation
errors when custom presets are used (previously, only the custom
presets have precisely assigned types for the constants).
 2020-07-13 16:44:58 +02:00
Zahary Karadjov
3ec6a02b12
Merge devel and resolve conflicts 2020-07-10 02:02:40 +03:00
Dustin Brody
4140b3b9d9 update 29 spec refs to v0.12.1 2020-07-08 20:49:25 +00:00
Zahary Karadjov
c4af4e2f35
Working test suite with run-time presets 2020-07-08 02:02:14 +03:00