status-im/nimbus-eth2
2
0
Fork 0
mirror of https://github.com/status-im/nimbus-eth2.git synced 2025-01-10 06:16:25 +00:00
Code Issues Projects Releases Wiki Activity
3,814 Commits 109 Branches 116 Tags 244 MiB
Commit Graph

3814 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Jacek Sieka
a223d62b07
Cleanups (#3123) 
Renames and cleanups split out from the validator monitoring branch, so
as to reduce conflict area vs other PR:s

* add constants for expected message timing
* name validators after the messages they validate, mostly, to make
grepping easier
* unify field naming of EpochInfo across forks to make cross-fork code
easier
 2021-11-25 13:20:36 +01:00
tersec
cc0dbd5bc0
implement terminal-total-difficulty-override; keep kintsugi TTDs consistent (#3118) 
* implement terminal-total-difficulty-override; keep TTD consistent for m2 scripts/docs

* use Option[uint64] instead of uint64
 2021-11-25 10:53:31 +00:00
Jacek Sieka
9c2f43ed0e
Speed up altair block processing 2x (#3115) 
* Speed up altair block processing >2x

Like #3089, this PR drastially speeds up historical REST queries and
other long state replays.

* cache sync committee validator indices
* use ~80mb less memory for validator pubkey mappings
* batch-verify sync aggregate signature (fixes #2985)
* document sync committee hack with head block vs sync message block
* add batch signature verification failure tests

Before:

```
../env.sh nim c -d:release -r ncli_db --db:mainnet_0/db bench --start-slot:-1000
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    5830.675,        0.000,     5830.675,     5830.675,            1, Initialize DB
       0.481,        1.878,        0.215,       59.167,          981, Load block from database
    8422.566,        0.000,     8422.566,     8422.566,            1, Load state from database
       6.996,        1.678,        0.042,       14.385,          969, Advance slot, non-epoch
      93.217,        8.318,       84.192,      122.209,           32, Advance slot, epoch
      20.513,       23.665,       11.510,      201.561,          981, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database load
       0.000,        0.000,        0.000,        0.000,            0, Database store
```

After:

```
    7081.422,        0.000,     7081.422,     7081.422,            1, Initialize DB
       0.553,        2.122,        0.175,       66.692,          981, Load block from database
    5439.446,        0.000,     5439.446,     5439.446,            1, Load state from database
       6.829,        1.575,        0.043,       12.156,          969, Advance slot, non-epoch
      94.716,        2.749,       88.395,      100.026,           32, Advance slot, epoch
      11.636,       23.766,        4.889,      205.250,          981, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database load
       0.000,        0.000,        0.000,        0.000,            0, Database store
```

* add comment
 2021-11-24 13:43:50 +01:00
Zahary Karadjov
88c623e250 Add support for HTTPS Web3 providers 2021-11-23 15:56:18 +02:00
Ștefan Talpalaru
2d986c5346
bump nimbus-build-system (#3117) 2021-11-23 02:43:04 +01:00
Ștefan Talpalaru
4639b04b2a
bump nimbus-build-system (#3116) 2021-11-22 21:21:13 +01:00
Jacek Sieka
95dd846a9b
Make sync horizon configurable (#3113) 
Currently, we don't have a good answer to the question "are we synced
yet" - the sync manager syncs based on the peers it's connected to, but
just because some peer looks like it should be synced from doesn't mean
we're out of sync.

Instead, we use a very silly time-based heuristic - the problem with
that is that the network can go into a rut where nobody produces blocks
- better heuristics would be needed here, but in the meantime, a command
line option can get us out of a tight spot - this PR places such an
option in the client, in the unlikely event it should be needed (most
likely in a testnet).
 2021-11-18 20:35:26 +01:00
Jacek Sieka
f19a497eec
ncli_db: add putState, putBlock (#3096) 
* ncli_db: add putState, putBlock

These tools allow modifying an existing nimbus database for the purpose
of recovery or reorg, moving the head, tail and genesis to arbitrary
points.

* remove potentially expensive `putState` in `BeaconStateDB`
* introduce `latest_block_root` which computes the root of the latest
applied block from the `latest_block_header` field (instead of passing
it in separately)
* avoid some unnecessary BeaconState copies during init
* discover https://github.com/nim-lang/Nim/issues/19094
* prefer `HashedBeaconState` in a few places to avoid recomputing state
root
* fetch latest block root from state when creating blocks
* harden `get_beacon_proposer_index` against invalid slots and document
* move random spec function tests to `test_spec.nim`
* avoid unnecessary state root computation before block proposal
 2021-11-18 13:02:43 +01:00
Tanguy
4bcdccab9a
Bump libp2p (#3112) 2021-11-18 10:57:27 +01:00
tersec
9e395011d9
update 22 spec URLs to v1.1.5 (#3111) 2021-11-18 08:08:00 +00:00
tersec
39f6a6534e
document how to run merge local testnet with Nethermind (#3110) 2021-11-17 20:45:39 +01:00
Ștefan Talpalaru
c673525591
book: add FRITZ!Box UPnP setup (#3106) 
* book: add FRITZ!Box UPnP setup

* Update docs/the_nimbus_book/src/networking.md

* Update docs/the_nimbus_book/src/networking.md

* Update docs/the_nimbus_book/src/networking.md

Co-authored-by: sacha <sacha@status.im>
 2021-11-16 10:46:27 +00:00
sacha
14cb68c72f
Book: upgrade / downgrade page (#3105) 
* cp

* cp
 2021-11-15 20:07:30 +01:00
Eugene Kabanov
eda42f4a89
Bump nim-chronos. (#3101) 2021-11-15 18:34:07 +00:00
tersec
c5fa7387f9
update kintsugi docs (#3104) 2021-11-15 17:01:27 +00:00
tersec
fde73b5a70
bump nim-web3 to switch payload ID from uint64 to 8-byte buffer (#3103) 2021-11-15 14:55:03 +00:00
Zahary Karadjov
5e62df5ad5 /bin/bash -> /usr/bin/env bash (needed on NixOS) 2021-11-15 15:45:01 +02:00
tersec
b4d27b36bc
rename eth1Chain to depositsChain (#3094) 2021-11-15 12:01:47 +00:00
tersec
ab742468e8
rename EthAddress to ExecutionAddresss per spec (#3100) 2021-11-15 10:36:07 +00:00
tersec
6f8eff8f13
merge topic sub/unsub infrastructure (#3099) 2021-11-14 09:00:25 +01:00
tersec
8cab7a74e7
remove shared/altair-devnet-3 support (#3098) 2021-11-14 08:57:27 +01:00
tersec
00d066f7dd
add merge gossip validation, except for beaconblocks (#3095) 2021-11-13 22:26:02 +01:00
Jacek Sieka
222674b203
better logging on invalid database (#3097) 
* remove redundant test report
 2021-11-13 17:27:28 +01:00
Jacek Sieka
f3881cbf73
Merge remote-tracking branch 'origin/stable' into unstable 2021-11-13 11:04:32 +01:00
Jacek Sieka
b22d86e161
REST/JSON-RPC: speed up several requests (#3092) 
REST/JSON-RPC and a few more also invalidate caches unnecessarily,
similar to https://github.com/status-im/nimbus-eth2/pull/3089

* avoid copying validator on balance request
 2021-11-12 23:29:28 +01:00
Kim De Mey
2da9309d2a
Remove SSZ files forgotten in move to ssz repo (#3091) 2021-11-12 13:36:35 +01:00
Jacek Sieka
00bbc8e0fe
speed up epoch processing 6x+ (#3089) 
* speed up epoch processing 6x+

This change above all helps contain long replay times on epoch change, reorg
and deep history inspection via REST/RPC

* most effective balances don't actually change due to MAX_EFFECTIVE_BALANCE
* ditto for inactivity scores
* avoid signature check for trusted sync aggregates

pre:
```
./ncli_db --db:mainnet_0/db bench --start-slot=-3200
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed

    3468.621,        0.000,     3468.621,     3468.621,            1, Initialize DB
       0.357,        0.938,        0.171,       52.752,         3155, Load block from database
   15691.471,        0.000,    15691.471,    15691.471,            1, Load state from database
       6.100,        9.469,        0.033,      526.816,         3101, Advance slot, non-epoch
     579.131,        9.523,      566.936,      610.328,          100, Advance slot, epoch
      18.551,       16.317,       12.664,      136.668,         3155, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database load
       0.000,        0.000,        0.000,        0.000,            0, Database store
```

post:
```
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3488.541,        0.000,     3488.541,     3488.541,            1, Initialize DB
       0.369,        1.123,        0.183,       63.208,         3155, Load block from database
   13430.642,        0.000,    13430.642,    13430.642,            1, Load state from database
       6.522,        1.721,        0.034,       36.708,         3101, Advance slot, non-epoch
      89.074,        3.162,       83.573,      101.436,          100, Advance slot, epoch
      18.325,       18.346,       13.005,      145.040,         3155, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database load
       0.000,        0.000,        0.000,        0.000,            0, Database store
```

* Update beacon_chain/spec/state_transition_block.nim

Co-authored-by: zah <zahary@gmail.com>

* avoid copying validator data in accessor

```
    5291.227,        0.000,     5291.227,     5291.227,            1, Initialize DB
       0.436,        0.928,        0.138,       51.438,         3155, Load block from database
   11962.826,        0.000,    11962.826,    11962.826,            1, Load state from database
       6.477,        1.675,        0.037,       34.174,         3101, Advance slot, non-epoch
      76.633,        3.705,       71.106,       98.085,          100, Advance slot, epoch
      18.301,       18.593,       13.208,      149.153,         3155, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database load
       0.000,        0.000,        0.000,        0.000,            0, Database store
```

* work around compiler bug

Co-authored-by: zah <zahary@gmail.com>
 2021-11-11 20:24:29 +01:00
Mamy Ratsimbazafy
9f4a55b82b
SSZ markdown test update missing [skip ci] (#3086) 2021-11-10 14:47:02 +01:00
tersec
97ad5d49b0
kintsugi merge vector tests (#3072) 2021-11-10 13:41:02 +02:00
Jacek Sieka
ec650c7fd7
Support starting from altair (#3054) 
* Support starting from altair

* hide `finalized-checkpoint-` - they are incomplete and usage may cause
crashes
* remove genesis detection code (broken, obsolete)
* enable starting ChainDAG from altair checkpoints - this is a
prerequisite for checkpoint sync (TODO: backfill)
* tighten checkpoint state conditions
* show error when starting from checkpoint with existing database (not
supported)
* print rest-compatible JSON in ncli/state_sim
* altair/merge support in ncli
* more altair/merge support in ncli_db
* pre-load header to speed up loading
* fix forked block decoding
 2021-11-10 13:39:08 +02:00
kdeme
03a70fbf36 Use nim-ssz-serialization module and rm local ssz code 2021-11-10 13:37:24 +02:00
Ștefan Talpalaru
b1218c6cd1
Version 1.5.4 v1.5.4 2021-11-10 11:55:57 +01:00
Ștefan Talpalaru
782318d3fc
Nim-1.2.14 (#3074) 
* Nim-1.2.14

* bump submodules
 2021-11-10 11:39:43 +01:00
Zahary Karadjov
df31c46cb3 Version 1.5.3 v1.5.3 2021-11-10 11:02:45 +02:00
zah
2c0be052a5
Reduce the logging-related breaking changes (#3070) 
* Reduce the logging-related breaking changes

* Don't disable the stdout log when the `--log-file` option is used
* Rename `--log-stdout` to `--log-format` and hide it
* Deprecate the `--log-file` option
 2021-11-10 11:02:18 +02:00
tersec
5c48982280
a dozen spec URL updates to v1.1.5 (#3078) 2021-11-10 08:12:41 +00:00
tersec
59db4b4f9c
remove intsets usage (#3075) 2021-11-10 02:18:52 +00:00
zah
3545d4d1e1
Address review comments in #3057 (#3069) 
* Address review comments in #3057

* reorder imports in rest_utils

maybe this will help with the mysterious serialization issues

Co-authored-by: Jacek Sieka <jacek@status.im>
 2021-11-09 20:21:36 +01:00
tersec
941eb609ba
update statediffs to work with Altair (#3061) 
* update statediffs to work with Altair
 2021-11-09 16:17:57 +00:00
sacha
9d82606eba
Update logging.md 2021-11-09 15:38:48 +01:00
sacha
db612ae906
Update logging.md 
add warning
 2021-11-09 15:38:32 +01:00
sacha
87fa67045f
Update logging.md 
replace `--log-stdout` with -`-log-format`. remove references to `--log-file`
 2021-11-09 15:21:48 +01:00
Zahary Karadjov
66d60e47ae Bump Chronicles 2021-11-09 13:33:53 +02:00
Zahary Karadjov
200902ccb3 Don't start the Eth1Monitor when there are no validators attached 
This is a temporary revert of the new logic introduced in
d6cd1cd46c357fcbe7e44bdcdb94ebc6d265dcdc, because it leads
to a memory leak in the Eth1Monitor caused by the lack of
pruning of the Eth1Chain when the validator duties are not
executed.
 2021-11-09 12:23:11 +02:00
Zahary Karadjov
ba42b2b316 Correct implementation of the /validator/duties/sync/{epoch} API 
According to the spec, this call should return the positions of
the specified validators within the sync committee. The existing
code was instead returning the indices of the sync sub-committees
where the validator is a member.
 2021-11-09 11:45:00 +02:00
tersec
48eba59971
manually-verified v1.1.5 spec URL updates (#3068) 2021-11-09 08:54:59 +00:00
tersec
2e868dc2ba
mass/mechanical update of 1.1.4 phase0 and altair spec URLs to 1.1.5 (#3067) 2021-11-09 07:40:41 +00:00
tersec
b76326f4b8
use v1.1.5 consensus spec test vectors (#3066) 2021-11-09 06:46:24 +00:00
tersec
95b0ecc5a2
only invalidate {current,previous}_epoch_participation flag cache once (#3063) 2021-11-09 02:44:02 +00:00
tersec
a0f6a19267
bump nim-web3 for Kintsugi engine API changes (#3065) 2021-11-09 02:43:51 +00:00