Commit Graph

3938 Commits

Author SHA1 Message Date
Jacek Sieka 00bbc8e0fe
speed up epoch processing 6x+ (#3089)
* speed up epoch processing 6x+

This change above all helps contain long replay times on epoch change, reorg
and deep history inspection via REST/RPC

* most effective balances don't actually change due to MAX_EFFECTIVE_BALANCE
* ditto for inactivity scores
* avoid signature check for trusted sync aggregates

pre:
```
./ncli_db --db:mainnet_0/db bench --start-slot=-3200
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed

    3468.621,        0.000,     3468.621,     3468.621,            1, Initialize DB
       0.357,        0.938,        0.171,       52.752,         3155, Load block from database
   15691.471,        0.000,    15691.471,    15691.471,            1, Load state from database
       6.100,        9.469,        0.033,      526.816,         3101, Advance slot, non-epoch
     579.131,        9.523,      566.936,      610.328,          100, Advance slot, epoch
      18.551,       16.317,       12.664,      136.668,         3155, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database load
       0.000,        0.000,        0.000,        0.000,            0, Database store
```

post:
```
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3488.541,        0.000,     3488.541,     3488.541,            1, Initialize DB
       0.369,        1.123,        0.183,       63.208,         3155, Load block from database
   13430.642,        0.000,    13430.642,    13430.642,            1, Load state from database
       6.522,        1.721,        0.034,       36.708,         3101, Advance slot, non-epoch
      89.074,        3.162,       83.573,      101.436,          100, Advance slot, epoch
      18.325,       18.346,       13.005,      145.040,         3155, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database load
       0.000,        0.000,        0.000,        0.000,            0, Database store
```

* Update beacon_chain/spec/state_transition_block.nim

Co-authored-by: zah <zahary@gmail.com>

* avoid copying validator data in accessor

```
    5291.227,        0.000,     5291.227,     5291.227,            1, Initialize DB
       0.436,        0.928,        0.138,       51.438,         3155, Load block from database
   11962.826,        0.000,    11962.826,    11962.826,            1, Load state from database
       6.477,        1.675,        0.037,       34.174,         3101, Advance slot, non-epoch
      76.633,        3.705,       71.106,       98.085,          100, Advance slot, epoch
      18.301,       18.593,       13.208,      149.153,         3155, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database load
       0.000,        0.000,        0.000,        0.000,            0, Database store
```

* work around compiler bug

Co-authored-by: zah <zahary@gmail.com>
2021-11-11 20:24:29 +01:00
Mamy Ratsimbazafy 9f4a55b82b
SSZ markdown test update missing [skip ci] (#3086) 2021-11-10 14:47:02 +01:00
tersec 97ad5d49b0
kintsugi merge vector tests (#3072) 2021-11-10 13:41:02 +02:00
Jacek Sieka ec650c7fd7
Support starting from altair (#3054)
* Support starting from altair

* hide `finalized-checkpoint-` - they are incomplete and usage may cause
crashes
* remove genesis detection code (broken, obsolete)
* enable starting ChainDAG from altair checkpoints - this is a
prerequisite for checkpoint sync (TODO: backfill)
* tighten checkpoint state conditions
* show error when starting from checkpoint with existing database (not
supported)
* print rest-compatible JSON in ncli/state_sim
* altair/merge support in ncli
* more altair/merge support in ncli_db
* pre-load header to speed up loading
* fix forked block decoding
2021-11-10 13:39:08 +02:00
kdeme 03a70fbf36 Use nim-ssz-serialization module and rm local ssz code 2021-11-10 13:37:24 +02:00
Ștefan Talpalaru b1218c6cd1
Version 1.5.4 2021-11-10 11:55:57 +01:00
Ștefan Talpalaru 782318d3fc
Nim-1.2.14 (#3074)
* Nim-1.2.14

* bump submodules
2021-11-10 11:39:43 +01:00
Zahary Karadjov df31c46cb3 Version 1.5.3 2021-11-10 11:02:45 +02:00
zah 2c0be052a5
Reduce the logging-related breaking changes (#3070)
* Reduce the logging-related breaking changes

* Don't disable the stdout log when the `--log-file` option is used
* Rename `--log-stdout` to `--log-format` and hide it
* Deprecate the `--log-file` option
2021-11-10 11:02:18 +02:00
tersec 5c48982280
a dozen spec URL updates to v1.1.5 (#3078) 2021-11-10 08:12:41 +00:00
tersec 59db4b4f9c
remove intsets usage (#3075) 2021-11-10 02:18:52 +00:00
zah 3545d4d1e1
Address review comments in #3057 (#3069)
* Address review comments in #3057

* reorder imports in rest_utils

maybe this will help with the mysterious serialization issues

Co-authored-by: Jacek Sieka <jacek@status.im>
2021-11-09 20:21:36 +01:00
tersec 941eb609ba
update statediffs to work with Altair (#3061)
* update statediffs to work with Altair
2021-11-09 16:17:57 +00:00
sacha 9d82606eba
Update logging.md 2021-11-09 15:38:48 +01:00
sacha db612ae906
Update logging.md
add warning
2021-11-09 15:38:32 +01:00
sacha 87fa67045f
Update logging.md
replace `--log-stdout` with -`-log-format`. remove references to `--log-file`
2021-11-09 15:21:48 +01:00
Zahary Karadjov 66d60e47ae Bump Chronicles 2021-11-09 13:33:53 +02:00
Zahary Karadjov 200902ccb3 Don't start the Eth1Monitor when there are no validators attached
This is a temporary revert of the new logic introduced in
d6cd1cd46c, because it leads
to a memory leak in the Eth1Monitor caused by the lack of
pruning of the Eth1Chain when the validator duties are not
executed.
2021-11-09 12:23:11 +02:00
Zahary Karadjov ba42b2b316 Correct implementation of the /validator/duties/sync/{epoch} API
According to the spec, this call should return the positions of
the specified validators within the sync committee. The existing
code was instead returning the indices of the sync sub-committees
where the validator is a member.
2021-11-09 11:45:00 +02:00
tersec 48eba59971
manually-verified v1.1.5 spec URL updates (#3068) 2021-11-09 08:54:59 +00:00
tersec 2e868dc2ba
mass/mechanical update of 1.1.4 phase0 and altair spec URLs to 1.1.5 (#3067) 2021-11-09 07:40:41 +00:00
tersec b76326f4b8
use v1.1.5 consensus spec test vectors (#3066) 2021-11-09 06:46:24 +00:00
tersec 95b0ecc5a2
only invalidate {current,previous}_epoch_participation flag cache once (#3063) 2021-11-09 02:44:02 +00:00
tersec a0f6a19267
bump nim-web3 for Kintsugi engine API changes (#3065) 2021-11-09 02:43:51 +00:00
sacha 624345bcee
Update logging.md
mainly style edits
2021-11-08 22:05:11 +01:00
Tanguy 8092418746
Bump libp2p (Short agent fix) (#3064)
* cherry-pick libp2p #645

* switch to master
2021-11-08 21:40:23 +01:00
tersec 5c296c08b7
17 spec URL updates to v1.1.4 (#3060) 2021-11-08 10:30:52 +00:00
tersec 2c8600e746
mass/mechanical update of 1.1.3 phase0 spec URLs to 1.1.4 in markdown (#3059) 2021-11-08 09:26:18 +00:00
tersec eb3ad25859
mass/mechanical update of 1.1.3 phase/altair spec URLs to 1.1.4 (#3058) 2021-11-08 06:18:10 +00:00
Dustin Brody 023a6a9387 use v1.1.4 test vectors 2021-11-07 21:42:11 +02:00
Zahary Karadjov 29e5700838 Bugfix: Avoid the aggregation of duplicate signatures when creating sync committee contributions 2021-11-07 21:41:10 +02:00
Ștefan Talpalaru 74f03ddadb
CI: statically check stack size on Linux (#3055) 2021-11-05 18:11:25 +01:00
Jacek Sieka ea0a191723
Better REST/RPC error messages (#3046)
* Better REST/RPC error messages
* homogenise block logging (root first)
* homegenise message verification pipeline (verify in
`gossip_verification`, act in `eth2_processor`)
* use `subcommitteeIdx` consistently
* log each sent contribution
* fix block_sim
* fix block topic
* don't recalc root on gossip block validation
* move position loop into sync pool
2021-11-05 17:39:47 +02:00
Ștefan Talpalaru b09cd71348 build: show stderr by default
Also added: `-d:cwarnings` and `-d:limitStackUsage`
2021-11-05 16:06:48 +02:00
sacha 122055b65e
cp (#3053) 2021-11-05 11:33:28 +01:00
Jacek Sieka a086cf01ac
altair fork handling cleanups (#3050)
* fix stack overflow crash in REST/debug/getStateV2
* introduce `ForkyXxx` for generic type matching of `Xxx` across
branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) -
`Some` is already used for other types of type classes
* consolidate function naming in BeaconChainDB, use some generics
* import `forks.nim` from other spec modules and move `Forked*` helpers
around to resolve circular imports
* remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout
(less data shuffling between the types)
* fix several cases of states being stored on stack in tests, causing
random failures on some platforms
* remove reading json support from ncli - this should be ported to the
rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 08:34:34 +01:00
Jacek Sieka 98668dbf16
reuse pre-computed root for sig verification (#3048)
no point re-hashing arguments
2021-11-05 05:50:01 +00:00
Jacek Sieka 3bfb5ad367
logtrace needs json logs (#3047) 2021-11-03 07:24:03 +01:00
Etan Kissling 51aef34be7
introduce strong typing for sync committee periods (#3040)
The sync committee period used to be a plain `uint64`. With the light
client sync relying more heavily on them, it makes sense to introduce
a proper type for them, similar to how they are already used for `Slot`
and `Epoch`. This introduces such a `SyncCommitteePeriod` type.
Furthermore, some usage code dealing with those periods is cleaned up.
2021-11-02 21:32:34 +01:00
Etan Kissling 1943258c66
extend REST config with more spec constants (#2962)
So far, the REST config response did not include all spec constants.
The spec for `/eth/v1/config/spec` defines that the response should
include constants for all hard forks known by the beacon node. This
patch extends the corresponding response to include more constants.
2021-11-02 19:23:31 +01:00
Jacek Sieka 233d756518
Logging and startup improvements (#3038)
* Logging and startup improvements

Color support for released binaries!

* startup scripts no longer log to file by default - this only affects
source builds - released binaries don't support file logging
* add --log-stdout option to control logging to stdout (colors, json)
* detect tty:s vs redirected logs and log accordingly
* add option to disable log colors at runtime
* simplify several "common" logs, showing the most important information
earlier and more clearly
* remove line numbers / file information / tid - these take up space and
are of little use to end users
  * still enabled in debug builds and tools
* remove `testnet_servers_image` compile-time option
* server images, released binaries and compile-from-source now offer
the same behaviour and features
* fixes https://github.com/status-im/nimbus-eth2/issues/2326
* fixes https://github.com/status-im/nimbus-eth2/issues/1794
* remove instanteneous block speed from sync message, keeping only
average

before:

```
INF 2021-10-28 16:45:59.000+02:00 Slot start                                 topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:884 lastSlot=2384027 wallSlot=2384028 delay=461us84ns peers=0 head=75a10ee5:3348 headEpoch=104 finalized=cd6804ba:3264 finalizedEpoch=102 sync="wwwwwwwwww:0:0.0000:0.0000:00h00m (3348)"
INF 2021-10-28 16:45:59.046+02:00 Slot end                                   topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:821 slot=2384028 nextSlot=2384029 head=75a10ee5:3348 headEpoch=104 finalizedHead=cd6804ba:3264 finalizedEpoch=102 nextAttestationSlot=-1 nextProposalSlot=-1 nextActionWait=n/a
```

after:

```
INF 2021-10-28 22:43:23.033+02:00 Slot start                                 topics="beacnde" slot=2385815 epoch=74556 sync="DDPDDPUDDD:10:5.2258:01h19m (2361088)" peers=37 head=eacd2dae:2361096 finalized=73782:a4751487 delay=33ms687us715ns
INF 2021-10-28 22:43:23.291+02:00 Slot end                                   topics="beacnde" slot=2385815 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 head=eacd2dae:2361096
```

* fix comment

* documentation updates

* mention `--log-file` may be deprecated in the future
* update various docs
2021-11-02 18:06:36 +01:00
sacha 56c6d6b583
update troubleshooting (#3044)
* update troubleshooting

* remove pyrmont references
2021-11-01 17:46:59 +01:00
Jacek Sieka d6cd1cd46c
Remove web3-mode, always keep web3 monitor enabled when given (#3042)
* Init some components fully before BeaconNode, per component dependency
graph
* remove `--web3-mode` option
* fixes https://github.com/status-im/nimbus-eth2/issues/2685
* reshuffle some beacon node init code
2021-11-01 15:50:24 +01:00
Etan Kissling 74f2350a2c
update nim-presto to bfcbece (#3039)
Upstream changes:
- allow reserved Nim keywords as REST params
2021-10-31 19:51:05 +01:00
tersec 36e37bda40
v1.1.3 spec refs URLs (#3036) 2021-10-27 18:40:17 +00:00
Eugene Kabanov 7c9a6b7170
Improve chronos.Future tracking. (#2988)
* Add `child_id` field.

* Fix json-rpc api call and bump chronos.

* Bump chronos master and fix compilation warnings.

* One more bump of `chronos`.

* add random import

* export rest_utils a bit more

Co-authored-by: Jacek Sieka <jacek@status.im>
2021-10-27 14:01:11 +02:00
Etan Kissling 23b5a945ea
merge stable into unstable 2021-10-27 09:58:24 +02:00
Etan Kissling fdbfe19b3e
fix CI for contributions from forked repos (#3028)
This updates the publish-unit-test-result integration with support for
pull requests coming from forked repositories. The update changes the
behaviour so that (a) unit test results are displayed as a comment, and
(b) macOS and Windows builds no longer time out during the CI action.
The official guide for the publish-unit-test-result action has been
followed for this purpose. See the official readme at:
https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.23/README.md#support-fork-repositories-and-dependabot-branches
2021-10-27 00:49:35 +02:00
tersec 968a5b3bec
some spec refs to v1.1.3 (#3032) 2021-10-26 18:42:48 +00:00
tersec 8307e9c601
mechanical non-merge v1.1.2 to v1.1.3 spec URL updates (#3030) 2021-10-26 16:44:23 +00:00