Commit Graph

111 Commits

Author SHA1 Message Date
zah 8967f9cf01
Work-around the sizeof change in behavior introduced in Nim 1.6 (#3462) 2022-03-04 10:52:49 +02:00
tersec f0ada15dac
automated CL spec ref URL updates from v1.1.9 to v1.1.10 (#3455) 2022-03-02 10:00:21 +00:00
Tanguy 1bfbcc48b6
Bump libp2p (#3438) 2022-02-25 13:22:48 +01:00
zah c29aa9d846
Support for Gnosis Chain (#3415)
* Support for Gnosis Chain

`make gnosis-chain-build` will build the Nimbus gnosis chain binary,
stored in `build/nimbus_beacon_node_for_gnosis_chain`.

`make gnosis-chain` will connect to the network.

Other changes:

* Restore compilation with -d:has_genesis_detection
* Removed Makefile target related to testnet0 and testnet1
* Added more debug logging for failed peer handshakes
* Report misconfigured builds which try to embed network metadata
that is incompatible with the currently selected const preset.
* Don't bundle network metadata in minimal builds, as they are not compatible
2022-02-25 10:22:44 +02:00
tersec 7de3f00f35
generic putCorruptState; {Merge=>Bellatrix}BeaconStateNoImmutableValidators (#3427) 2022-02-21 12:55:56 +01:00
tersec 5eecb9a21f
rename no{R=>r}eturn, no{I=>i}init, short{l=>L}og, E{T=>t}h2Node, Beacon{c=>C}hainDB (#3403) 2022-02-16 23:24:44 +01:00
Jacek Sieka c7abc97545
harden and speed up block sync (#3358)
* harden and speed up block sync

The `GetBlockBy*` server implementation currently reads SSZ bytes from
database, deserializes them into a Nim object then serializes them right
back to SSZ - here, we eliminate the deser/ser steps and send the bytes
straight to the network. Unfortunately, the snappy recoding must still
be done because of differences in framing.

Also, the quota system makes one giant request for quota right before
sending all blocks - this means that a 1024 block request will be
"paused" for a long time, then all blocks will be sent at once causing a
spike in database reads which potentially will see the reading client
time out before any block is sent.

Finally, on the reading side we make several copies of blocks as they
travel through various queues - this was not noticeable before but
becomes a problem in two cases: bellatrix blocks are up to 10mb (instead
of .. 30-40kb) and when backfilling, we process a lot more of them a lot
faster.

* fix status comparisons for nodes syncing from genesis (#3327 was a bit
too hard)
* don't hit database at all for post-altair slots in GetBlock v1
requests
2022-02-07 19:20:10 +02:00
Tanguy bcd7b4598c
Tune peering (#3348)
- Request metadata_v2 (altair) by default instead of the v1
- Change the metadata pinger to a 3 failure-then-kick, instead of being time based
- Update kicker scorer to take into account topics which we're not subscribed to, to be sure that we will be able to publish correctly
- Add some metrics to give "fanout" health (in the same spirit of mesh health)
2022-02-01 18:20:55 +01:00
tersec 29e2169585
phase 0 & altair beacon chain and altair validator spec URL updates (#3339) 2022-01-29 13:53:31 +00:00
tersec 89ffa8a1a7
spec URL & copyright year update (#3338) 2022-01-29 01:05:39 +00:00
Jacek Sieka f70aceef37
Harden handling of unviable forks (#3312)
* Harden handling of unviable forks

In our current handling of unviable forks, we allow peers to send us
blocks that come from a different fork - this is not necessarily an
error as it can happen naturally, but it does open up the client to a
case where the same unviable fork keeps getting requested - rather than
allowing this to happen, we'll now give these peers a small negative
score - if it keeps happening, we'll disconnect them.

* keep track of unviable forks in quarantine, to avoid filling it with
known junk
* collect peer scores in single module
* descore peers when they send unviable blocks during sync
* don't give score for duplicate blocks
* increase quarantine size to a level that allows finality to happen
under optimal conditions - this helps avoid downloading the same blocks
over and over in case of an unviable fork
* increase initial score for new peers to make room for one more failure
before disconnection
* log and score invalid/unviable blocks in requestmanager too
* avoid ChainDAG dependency in quarantine
* reject gossip blocks with unviable parent
* continue processing unviable sync blocks in order to build unviable
dag

* docs

* Update beacon_chain/consensus_object_pools/block_pools_types.nim

* add unviable queue test
2022-01-26 13:20:08 +01:00
tersec 2f635d3337
rename *_{MERGE => BELLATRIX} constant names (#3296) 2022-01-18 16:31:05 +00:00
Zahary Karadjov 47f1f7ff1a More efficient reward data persistance; Address review comments
The new format is based on compressed CSV files in two channels:

* Detailed per-epoch data
* Aggregated "daily" summaries

The use of append-only CSV file speeds up significantly the epoch
processing speed during data generation. The use of compression
results in smaller storage requirements overall. The use of the
aggregated files has a very minor cost in both CPU and storage,
but leads to near interactive speed for report generation.

Other changes:

- Implemented support for graceful shut downs to avoid corrupting
  the saved files.

- Fixed a memory leak caused by lacking `StateCache` clean up on each
  iteration.

- Addressed review comments

- Moved the rewards and penalties calculation code in a separate module

Required invasive changes to existing modules:

- The `data` field of the `KeyedBlockRef` type is made public to be used
  by the validator rewards monitor's Chain DAG update procedure.

- The `getForkedBlock` procedure from the `blockchain_dag.nim` module
  is made public to be used by the validator rewards monitor's Chain DAG
  update procedure.
2022-01-18 01:56:56 +02:00
tersec 14aab2c13f
update 10 modules from using merge to bellatrix (#3272) 2022-01-12 15:50:30 +01:00
Jacek Sieka 20e700fae4
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259)
* Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex

Harden the use of `CommitteeIndex` et al to prevent future issues by
using a distinct type, then validating before use in several cases -
datatypes in spec are kept simple though so that invalid data still can
be read.

* fix invalid epoch used in REST
`/eth/v1/beacon/states/{state_id}/committees` committee length (could
return invalid data)
* normalize some variable names
* normalize committee index loops
* fix `RestAttesterDuty` to use `uint64` for `validator_committee_index`
* validate `CommitteeIndex` on ingress in REST API
* update rest rules with stricter parsing
* better REST serializers
* save lots of memory by not using `zip` ...at least a few bytes!
2022-01-09 01:28:49 +02:00
tersec 5878d34117
rename forkDigests.merge to forkDigests.bellatrix (#3245) 2022-01-05 14:24:15 +00:00
Tanguy 511f2d24f0
Tune getLowSubnets (#3241)
* Tune getLowSubnets

* Also aim for dHigh peers in gossipsub

* Apply suggestions from code review

Co-authored-by: Jacek Sieka <jacek@status.im>

Co-authored-by: Jacek Sieka <jacek@status.im>
2022-01-04 14:37:04 +01:00
tersec b81c06edab
rename Beacon{Block,State}Fork.Merge to Bellatrix; update copyright years (#3240) 2022-01-04 09:45:38 +00:00
tersec d20387e910
update copyright years and spec URLs (#3239) 2022-01-04 06:08:19 +00:00
tersec da017d2ca5
update from phase0/altair v1.1.6 URLs to v1.1.8 spec URLs (#3238) 2022-01-04 03:57:15 +00:00
tersec e78d12beb9
support GOSSIP_MAX_SIZE_MERGE blocks; prevent fork choice stutter via aggregate attestations (#3230)
* support GOSSIP_MAX_SIZE_MERGE-sized blocks; prevent fork choice clock stutter via aggregate attestations

* relay max gossip size to libp2p, use tight uncompressed bounds for fixed-size messages

* Update beacon_chain/networking/eth2_network.nim

Co-authored-by: Jacek Sieka <jacek@status.im>

* Update beacon_chain/networking/eth2_network.nim

Co-authored-by: Jacek Sieka <jacek@status.im>

Co-authored-by: Jacek Sieka <jacek@status.im>
2022-01-03 16:20:15 +00:00
Tanguy 4a72def1d5
Bump libp2p (#3207) 2021-12-17 12:39:24 +01:00
Etan Kissling ad26284614
fix typo (`snapshop` -> `snapshot`) (#3192)
This fixes a typo in the word `snapshot` of function name
`extractGenesisValidatorRootFromSnapshop`.
2021-12-14 15:44:34 +00:00
Etan Kissling 342ba68053
add copyright header to streams backends (#3177)
This adds the common copyright header to `libp2p_streams_backend.nim`
and `faststreams_backend.nim`. It was not present before in these file.
2021-12-10 02:28:09 +00:00
tersec cc51f3fd12
v1.1.{5 -> 6} phase 0 and altair spec URL updates (#3157) 2021-12-03 17:40:23 +00:00
tersec 9e395011d9
update 22 spec URLs to v1.1.5 (#3111) 2021-11-18 08:08:00 +00:00
tersec 8cab7a74e7
remove shared/altair-devnet-3 support (#3098) 2021-11-14 08:57:27 +01:00
kdeme 03a70fbf36 Use nim-ssz-serialization module and rm local ssz code 2021-11-10 13:37:24 +02:00
tersec 2e868dc2ba
mass/mechanical update of 1.1.4 phase0 and altair spec URLs to 1.1.5 (#3067) 2021-11-09 07:40:41 +00:00
tersec 5c296c08b7
17 spec URL updates to v1.1.4 (#3060) 2021-11-08 10:30:52 +00:00
tersec eb3ad25859
mass/mechanical update of 1.1.3 phase/altair spec URLs to 1.1.4 (#3058) 2021-11-08 06:18:10 +00:00
Jacek Sieka a086cf01ac
altair fork handling cleanups (#3050)
* fix stack overflow crash in REST/debug/getStateV2
* introduce `ForkyXxx` for generic type matching of `Xxx` across
branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) -
`Some` is already used for other types of type classes
* consolidate function naming in BeaconChainDB, use some generics
* import `forks.nim` from other spec modules and move `Forked*` helpers
around to resolve circular imports
* remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout
(less data shuffling between the types)
* fix several cases of states being stored on stack in tests, causing
random failures on some platforms
* remove reading json support from ncli - this should be ported to the
rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 08:34:34 +01:00
Jacek Sieka d6cd1cd46c
Remove web3-mode, always keep web3 monitor enabled when given (#3042)
* Init some components fully before BeaconNode, per component dependency
graph
* remove `--web3-mode` option
* fixes https://github.com/status-im/nimbus-eth2/issues/2685
* reshuffle some beacon node init code
2021-11-01 15:50:24 +01:00
tersec 8307e9c601
mechanical non-merge v1.1.2 to v1.1.3 spec URL updates (#3030) 2021-10-26 16:44:23 +00:00
Jacek Sieka 421bf936ff
odds and ends (#3015)
* `allSyncCommittees` => `allSyncSubcommittees`
* simplify `_snappy` topic generation (avoid pointless string copies)
* simplify gossip id generator (avoid pointless string copies)
* avoid redundant syncnet ENR updates
* simplify topic validation (allow only validated topics)
2021-10-21 15:09:19 +02:00
nbc-bump-bot[bot] 29b26f3f6b
nim-libp2p auto bump (#2840)
* auto-bump nim-libp2p

* Remove peer info for other peers

Not definitive, just to test the libp2p's unstable branch

* finish up Remove peer info for other peers

* getKey -> getPublicKey

* bump libp2p

* libp2p bump

Co-authored-by: = <Menduist@users.noreply.github.com>
Co-authored-by: Tanguy <tanguy@status.im>
2021-10-21 13:01:29 +02:00
Jacek Sieka 9cf32c3748 clean up sync subcommittee handling
* `SyncCommitteeIndex` -> `SyncSubcommitteeIndex`
* `syncCommitteePeriod` -> `sync_committee_period` (spec spelling)
* tighten period comparisons
* fix assert when validating committee message with non-altair state in
REST api
2021-10-20 22:59:13 +03:00
Jacek Sieka bf6ad41d7d add drop and sync committee metrics
* use storeBlock for processing API blocks
* avoid double block dump
* count all gossip metrics at the same spot
* simplify block broadcast
2021-10-20 18:20:12 +03:00
Jacek Sieka c40cc6cec1 clean up fork enum and field names
* single naming strategy
* simplify some fork code
* simplify forked block production
2021-10-19 11:06:38 +03:00
Jacek Sieka 4f7a8cf79d
register vc duties with subnet tracker (#2949)
* register vc duties with subnet tracker
* fix activation logging during startup
* cache slot signature to avoid duplicate signature work
* schedule aggregation duties one slot at a time to avoid CPU spike at
each epoch
* lower aggregation subnet pre-subscription time to 4 slots (lowers
bandwidth and CPU usage)
* update stability subnets in ENR on startup
* log gossip state
* perform gossip subscriptions just before the next slot starts
* document stuff
* add random include
* don't overwrite subscription state when not subscribed
* log target gossip state
* updating gossip status once is enough
* add test
* remove syncQueueLen - this one is not updated at the end of the sync
and may cause gossip to disconnect itself completely - use a simple head
distance instead
* fix gossip disconnection - if in hysteresis, node.gossipState will be
set to disabled even though we don't disable topic subscriptions
* fix extra duty registration call
2021-10-18 11:11:44 +02:00
tersec 6cc8757930
update 18 spec URLs to v1.1.2; switch from eth2.0-specs to consensus-specs (#2990) 2021-10-14 06:30:21 +00:00
Jacek Sieka 0c4bfb1911
test fixture cleanups (#2976)
* "official" -> "scenarios", like the submodule
* fewer test binaries - various compile hacks have been improved over
time, test suite should follow
* remove obsolete bls tests - there are better test vectors in
nim-blscurve
* remove obsolete mentions of `ssz_testing`
* remove obsolete comments about proc vs globals, unittest2 already uses
proc's
2021-10-12 13:36:52 +02:00
tersec 2ad1b7366a
update 62 spec URLs to v1.1.2 (#2979) 2021-10-12 10:17:37 +00:00
Kim De Mey 7f2b1d69dd
Change concurrent dials from 10 to 20 (#2966) 2021-10-11 09:16:58 +00:00
Jacek Sieka a017e4a817
fix forkDigest 2021-10-06 18:47:11 +03:00
Jacek Sieka 5c80df51e4
remove libp2p todos (#2951)
libp2p has been cleaned up to not raise these exceptions
2021-10-04 18:42:34 +00:00
Zahary Karadjov 22346f3e32 Don't look for syncnets peers before the Altair transition 2021-10-04 18:43:43 +03:00
Etan Kissling 2bbffbde10
abort compile when fork epoch is forgotten (#2939)
There are a few locations in the code that compare the current epoch to
the various FORK_EPOCH constants and branch off into fork-specific code.
When a new fork is introduced, it is sometimes forgotten to update all
of those branch locations. This patch introduces a compile-time check
that ensures that all branches need to be covered exhaustively. This is
done by replacing if-elif structures with case expressions.
2021-10-04 08:31:21 +00:00
tersec 6b3bf7eb7b
merge hardfork database support (#2911)
* merge hardfork database support

* working block_sim

* recreate state transition changes
2021-09-30 01:07:24 +00:00
Tanguy 06df40454f
Improve discovery during fork (#2913) 2021-09-29 14:06:16 +03:00