status-im/nimbus-eth2
2
0
Fork 0
mirror of https://github.com/status-im/nimbus-eth2.git synced 2025-01-10 22:36:01 +00:00
Code Issues Projects Releases Wiki Activity
3,808 Commits 110 Branches 116 Tags 244 MiB
Commit Graph

2227 Commits

Author SHA1 Message Date
Jacek Sieka
95dd846a9b
Make sync horizon configurable (#3113) 
Currently, we don't have a good answer to the question "are we synced
yet" - the sync manager syncs based on the peers it's connected to, but
just because some peer looks like it should be synced from doesn't mean
we're out of sync.

Instead, we use a very silly time-based heuristic - the problem with
that is that the network can go into a rut where nobody produces blocks
- better heuristics would be needed here, but in the meantime, a command
line option can get us out of a tight spot - this PR places such an
option in the client, in the unlikely event it should be needed (most
likely in a testnet).
 2021-11-18 20:35:26 +01:00
Jacek Sieka
f19a497eec
ncli_db: add putState, putBlock (#3096) 
* ncli_db: add putState, putBlock

These tools allow modifying an existing nimbus database for the purpose
of recovery or reorg, moving the head, tail and genesis to arbitrary
points.

* remove potentially expensive `putState` in `BeaconStateDB`
* introduce `latest_block_root` which computes the root of the latest
applied block from the `latest_block_header` field (instead of passing
it in separately)
* avoid some unnecessary BeaconState copies during init
* discover https://github.com/nim-lang/Nim/issues/19094
* prefer `HashedBeaconState` in a few places to avoid recomputing state
root
* fetch latest block root from state when creating blocks
* harden `get_beacon_proposer_index` against invalid slots and document
* move random spec function tests to `test_spec.nim`
* avoid unnecessary state root computation before block proposal
 2021-11-18 13:02:43 +01:00
tersec
9e395011d9
update 22 spec URLs to v1.1.5 (#3111) 2021-11-18 08:08:00 +00:00
tersec
39f6a6534e
document how to run merge local testnet with Nethermind (#3110) 2021-11-17 20:45:39 +01:00
tersec
fde73b5a70
bump nim-web3 to switch payload ID from uint64 to 8-byte buffer (#3103) 2021-11-15 14:55:03 +00:00
tersec
b4d27b36bc
rename eth1Chain to depositsChain (#3094) 2021-11-15 12:01:47 +00:00
tersec
ab742468e8
rename EthAddress to ExecutionAddresss per spec (#3100) 2021-11-15 10:36:07 +00:00
tersec
6f8eff8f13
merge topic sub/unsub infrastructure (#3099) 2021-11-14 09:00:25 +01:00
tersec
8cab7a74e7
remove shared/altair-devnet-3 support (#3098) 2021-11-14 08:57:27 +01:00
tersec
00d066f7dd
add merge gossip validation, except for beaconblocks (#3095) 2021-11-13 22:26:02 +01:00
Jacek Sieka
222674b203
better logging on invalid database (#3097) 
* remove redundant test report
 2021-11-13 17:27:28 +01:00
Jacek Sieka
f3881cbf73
Merge remote-tracking branch 'origin/stable' into unstable 2021-11-13 11:04:32 +01:00
Jacek Sieka
b22d86e161
REST/JSON-RPC: speed up several requests (#3092) 
REST/JSON-RPC and a few more also invalidate caches unnecessarily,
similar to https://github.com/status-im/nimbus-eth2/pull/3089

* avoid copying validator on balance request
 2021-11-12 23:29:28 +01:00
Kim De Mey
2da9309d2a
Remove SSZ files forgotten in move to ssz repo (#3091) 2021-11-12 13:36:35 +01:00
Jacek Sieka
00bbc8e0fe
speed up epoch processing 6x+ (#3089) 
* speed up epoch processing 6x+

This change above all helps contain long replay times on epoch change, reorg
and deep history inspection via REST/RPC

* most effective balances don't actually change due to MAX_EFFECTIVE_BALANCE
* ditto for inactivity scores
* avoid signature check for trusted sync aggregates

pre:
```
./ncli_db --db:mainnet_0/db bench --start-slot=-3200
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed

    3468.621,        0.000,     3468.621,     3468.621,            1, Initialize DB
       0.357,        0.938,        0.171,       52.752,         3155, Load block from database
   15691.471,        0.000,    15691.471,    15691.471,            1, Load state from database
       6.100,        9.469,        0.033,      526.816,         3101, Advance slot, non-epoch
     579.131,        9.523,      566.936,      610.328,          100, Advance slot, epoch
      18.551,       16.317,       12.664,      136.668,         3155, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database load
       0.000,        0.000,        0.000,        0.000,            0, Database store
```

post:
```
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3488.541,        0.000,     3488.541,     3488.541,            1, Initialize DB
       0.369,        1.123,        0.183,       63.208,         3155, Load block from database
   13430.642,        0.000,    13430.642,    13430.642,            1, Load state from database
       6.522,        1.721,        0.034,       36.708,         3101, Advance slot, non-epoch
      89.074,        3.162,       83.573,      101.436,          100, Advance slot, epoch
      18.325,       18.346,       13.005,      145.040,         3155, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database load
       0.000,        0.000,        0.000,        0.000,            0, Database store
```

* Update beacon_chain/spec/state_transition_block.nim

Co-authored-by: zah <zahary@gmail.com>

* avoid copying validator data in accessor

```
    5291.227,        0.000,     5291.227,     5291.227,            1, Initialize DB
       0.436,        0.928,        0.138,       51.438,         3155, Load block from database
   11962.826,        0.000,    11962.826,    11962.826,            1, Load state from database
       6.477,        1.675,        0.037,       34.174,         3101, Advance slot, non-epoch
      76.633,        3.705,       71.106,       98.085,          100, Advance slot, epoch
      18.301,       18.593,       13.208,      149.153,         3155, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database load
       0.000,        0.000,        0.000,        0.000,            0, Database store
```

* work around compiler bug

Co-authored-by: zah <zahary@gmail.com>
 2021-11-11 20:24:29 +01:00
tersec
97ad5d49b0
kintsugi merge vector tests (#3072) 2021-11-10 13:41:02 +02:00
Jacek Sieka
ec650c7fd7
Support starting from altair (#3054) 
* Support starting from altair

* hide `finalized-checkpoint-` - they are incomplete and usage may cause
crashes
* remove genesis detection code (broken, obsolete)
* enable starting ChainDAG from altair checkpoints - this is a
prerequisite for checkpoint sync (TODO: backfill)
* tighten checkpoint state conditions
* show error when starting from checkpoint with existing database (not
supported)
* print rest-compatible JSON in ncli/state_sim
* altair/merge support in ncli
* more altair/merge support in ncli_db
* pre-load header to speed up loading
* fix forked block decoding
 2021-11-10 13:39:08 +02:00
kdeme
03a70fbf36 Use nim-ssz-serialization module and rm local ssz code 2021-11-10 13:37:24 +02:00
Ștefan Talpalaru
b1218c6cd1
Version 1.5.4 2021-11-10 11:55:57 +01:00
Zahary Karadjov
df31c46cb3 Version 1.5.3 2021-11-10 11:02:45 +02:00
zah
2c0be052a5
Reduce the logging-related breaking changes (#3070) 
* Reduce the logging-related breaking changes

* Don't disable the stdout log when the `--log-file` option is used
* Rename `--log-stdout` to `--log-format` and hide it
* Deprecate the `--log-file` option
 2021-11-10 11:02:18 +02:00
tersec
5c48982280
a dozen spec URL updates to v1.1.5 (#3078) 2021-11-10 08:12:41 +00:00
tersec
59db4b4f9c
remove intsets usage (#3075) 2021-11-10 02:18:52 +00:00
zah
3545d4d1e1
Address review comments in #3057 (#3069) 
* Address review comments in #3057

* reorder imports in rest_utils

maybe this will help with the mysterious serialization issues

Co-authored-by: Jacek Sieka <jacek@status.im>
 2021-11-09 20:21:36 +01:00
tersec
941eb609ba
update statediffs to work with Altair (#3061) 
* update statediffs to work with Altair
 2021-11-09 16:17:57 +00:00
Zahary Karadjov
200902ccb3 Don't start the Eth1Monitor when there are no validators attached 
This is a temporary revert of the new logic introduced in
d6cd1cd46c357fcbe7e44bdcdb94ebc6d265dcdc, because it leads
to a memory leak in the Eth1Monitor caused by the lack of
pruning of the Eth1Chain when the validator duties are not
executed.
 2021-11-09 12:23:11 +02:00
Zahary Karadjov
ba42b2b316 Correct implementation of the /validator/duties/sync/{epoch} API 
According to the spec, this call should return the positions of
the specified validators within the sync committee. The existing
code was instead returning the indices of the sync sub-committees
where the validator is a member.
 2021-11-09 11:45:00 +02:00
tersec
48eba59971
manually-verified v1.1.5 spec URL updates (#3068) 2021-11-09 08:54:59 +00:00
tersec
2e868dc2ba
mass/mechanical update of 1.1.4 phase0 and altair spec URLs to 1.1.5 (#3067) 2021-11-09 07:40:41 +00:00
tersec
b76326f4b8
use v1.1.5 consensus spec test vectors (#3066) 2021-11-09 06:46:24 +00:00
tersec
95b0ecc5a2
only invalidate {current,previous}_epoch_participation flag cache once (#3063) 2021-11-09 02:44:02 +00:00
tersec
a0f6a19267
bump nim-web3 for Kintsugi engine API changes (#3065) 2021-11-09 02:43:51 +00:00
tersec
5c296c08b7
17 spec URL updates to v1.1.4 (#3060) 2021-11-08 10:30:52 +00:00
tersec
2c8600e746
mass/mechanical update of 1.1.3 phase0 spec URLs to 1.1.4 in markdown (#3059) 2021-11-08 09:26:18 +00:00
tersec
eb3ad25859
mass/mechanical update of 1.1.3 phase/altair spec URLs to 1.1.4 (#3058) 2021-11-08 06:18:10 +00:00
Dustin Brody
023a6a9387 use v1.1.4 test vectors 2021-11-07 21:42:11 +02:00
Zahary Karadjov
29e5700838 Bugfix: Avoid the aggregation of duplicate signatures when creating sync committee contributions 2021-11-07 21:41:10 +02:00
Jacek Sieka
ea0a191723
Better REST/RPC error messages (#3046) 
* Better REST/RPC error messages
* homogenise block logging (root first)
* homegenise message verification pipeline (verify in
`gossip_verification`, act in `eth2_processor`)
* use `subcommitteeIdx` consistently
* log each sent contribution
* fix block_sim
* fix block topic
* don't recalc root on gossip block validation
* move position loop into sync pool
 2021-11-05 17:39:47 +02:00
Jacek Sieka
a086cf01ac
altair fork handling cleanups (#3050) 
* fix stack overflow crash in REST/debug/getStateV2
* introduce `ForkyXxx` for generic type matching of `Xxx` across
branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) -
`Some` is already used for other types of type classes
* consolidate function naming in BeaconChainDB, use some generics
* import `forks.nim` from other spec modules and move `Forked*` helpers
around to resolve circular imports
* remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout
(less data shuffling between the types)
* fix several cases of states being stored on stack in tests, causing
random failures on some platforms
* remove reading json support from ncli - this should be ported to the
rest json reading instead (doesn't currently work because stack sizes)
 2021-11-05 08:34:34 +01:00
Jacek Sieka
98668dbf16
reuse pre-computed root for sig verification (#3048) 
no point re-hashing arguments
 2021-11-05 05:50:01 +00:00
Etan Kissling
51aef34be7
introduce strong typing for sync committee periods (#3040) 
The sync committee period used to be a plain `uint64`. With the light
client sync relying more heavily on them, it makes sense to introduce
a proper type for them, similar to how they are already used for `Slot`
and `Epoch`. This introduces such a `SyncCommitteePeriod` type.
Furthermore, some usage code dealing with those periods is cleaned up.
 2021-11-02 21:32:34 +01:00
Etan Kissling
1943258c66
extend REST config with more spec constants (#2962) 
So far, the REST config response did not include all spec constants.
The spec for `/eth/v1/config/spec` defines that the response should
include constants for all hard forks known by the beacon node. This
patch extends the corresponding response to include more constants.
 2021-11-02 19:23:31 +01:00
Jacek Sieka
233d756518
Logging and startup improvements (#3038) 
* Logging and startup improvements

Color support for released binaries!

* startup scripts no longer log to file by default - this only affects
source builds - released binaries don't support file logging
* add --log-stdout option to control logging to stdout (colors, json)
* detect tty:s vs redirected logs and log accordingly
* add option to disable log colors at runtime
* simplify several "common" logs, showing the most important information
earlier and more clearly
* remove line numbers / file information / tid - these take up space and
are of little use to end users
  * still enabled in debug builds and tools
* remove `testnet_servers_image` compile-time option
* server images, released binaries and compile-from-source now offer
the same behaviour and features
* fixes https://github.com/status-im/nimbus-eth2/issues/2326
* fixes https://github.com/status-im/nimbus-eth2/issues/1794
* remove instanteneous block speed from sync message, keeping only
average

before:

```
INF 2021-10-28 16:45:59.000+02:00 Slot start                                 topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:884 lastSlot=2384027 wallSlot=2384028 delay=461us84ns peers=0 head=75a10ee5:3348 headEpoch=104 finalized=cd6804ba:3264 finalizedEpoch=102 sync="wwwwwwwwww:0:0.0000:0.0000:00h00m (3348)"
INF 2021-10-28 16:45:59.046+02:00 Slot end                                   topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:821 slot=2384028 nextSlot=2384029 head=75a10ee5:3348 headEpoch=104 finalizedHead=cd6804ba:3264 finalizedEpoch=102 nextAttestationSlot=-1 nextProposalSlot=-1 nextActionWait=n/a
```

after:

```
INF 2021-10-28 22:43:23.033+02:00 Slot start                                 topics="beacnde" slot=2385815 epoch=74556 sync="DDPDDPUDDD:10:5.2258:01h19m (2361088)" peers=37 head=eacd2dae:2361096 finalized=73782:a4751487 delay=33ms687us715ns
INF 2021-10-28 22:43:23.291+02:00 Slot end                                   topics="beacnde" slot=2385815 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 head=eacd2dae:2361096
```

* fix comment

* documentation updates

* mention `--log-file` may be deprecated in the future
* update various docs
 2021-11-02 18:06:36 +01:00
Jacek Sieka
d6cd1cd46c
Remove web3-mode, always keep web3 monitor enabled when given (#3042) 
* Init some components fully before BeaconNode, per component dependency
graph
* remove `--web3-mode` option
* fixes https://github.com/status-im/nimbus-eth2/issues/2685
* reshuffle some beacon node init code
 2021-11-01 15:50:24 +01:00
tersec
36e37bda40
v1.1.3 spec refs URLs (#3036) 2021-10-27 18:40:17 +00:00
Eugene Kabanov
7c9a6b7170
Improve chronos.Future tracking. (#2988) 
* Add `child_id` field.

* Fix json-rpc api call and bump chronos.

* Bump chronos master and fix compilation warnings.

* One more bump of `chronos`.

* add random import

* export rest_utils a bit more

Co-authored-by: Jacek Sieka <jacek@status.im>
 2021-10-27 14:01:11 +02:00
tersec
968a5b3bec
some spec refs to v1.1.3 (#3032) 2021-10-26 18:42:48 +00:00
tersec
8307e9c601
mechanical non-merge v1.1.2 to v1.1.3 spec URL updates (#3030) 2021-10-26 16:44:23 +00:00
Etan Kissling
d32c25ea32
use GeneralizedIndex type for spec constants (#3026) 
Spec constants currently use `uint16` instead of `GeneralizedIndex` to
define generalized indices in a merkle tree. Adjusted to use the latter
instead to avoid problems when adjusting `GeneralizedIndex`'s bit width.
 2021-10-26 12:30:36 +00:00
Eugene Kabanov
7c49d657a6
Fix regression in REST validators and validator_balances requests. (#3022) 
* Fix state validators and validator_balances responses when validator's identifier could not be found in state.

* Add tests.
 2021-10-25 15:52:00 +02:00