Commit Graph

1895 Commits

Author SHA1 Message Date
Kim De Mey 0ec90b26a5
Update ENR record with metadata attnets at each attestation subnet cycle (#2148) 2020-12-09 10:13:51 +01:00
tersec 8b8b25ddac
always check whether gossip should be enabled in onslotstart (#2162) 2020-12-08 18:11:54 +01:00
Jacek Sieka f10f29d829
avoid some futures and copies in processing pipeline (#2157)
`addLastNoWait` does the same thing with less task scheduling and
copying
2020-12-08 09:59:40 +01:00
Dustin Brody 3b06e0f657 directly construct cstring in err() expressions 2020-12-07 22:24:06 +02:00
tersec 72d00e33b8
always wait for 2/3 into slot for aggregate attestations (#2158) 2020-12-07 20:32:34 +01:00
Eugene Kabanov 81c25ca4cd
Disconnect peers on status handshake timeout. (#2149) 2020-12-07 19:47:07 +01:00
Eugene Kabanov 46c2740097
Documentation for Validators API. (#2147)
* Recover proper validator API call and remove incorrect one.
Add more examples to API documentation.
2020-12-07 14:51:14 +02:00
Dustin Brody 32a18769e6 remove waitFor in attestation subnet cycling 2020-12-07 14:48:04 +02:00
tersec 26ea76bbdf
don't lag aggregated attestations by a slot (#2150)
* don't lag aggregated attestations by a slot

* don't use aggregation topic at all

* use aggregates again, but with aggressively low ATTESTATION_PROPAGATION_SLOT_RANGE; seems to hold on to LH 1.0 nodes

* clean up scaffolding and double ATTESTATION_PROPAGATION_SLOT_RANGE to 16

* increase ATTESTATION_PROPAGATION_SLOT_RANGE to 24

* increase ATTESTATION_PROPAGATION_SLOT_RANGE to 28 and isolate in only used function due to customization; remove TRAILING_DISTANCE machinery
2020-12-07 12:56:49 +01:00
Zahary Karadjov 7363bf3c7c
Fix a logical typo in beacon_api/validator.getStatus 2020-12-03 21:24:01 +02:00
Zahary Karadjov d302c1d8ee
v1.0.2 2020-12-03 19:17:32 +02:00
Zahary Karadjov f4a3a36731
More specific error message when block validation fails 2020-12-03 11:51:42 +02:00
Ștefan Talpalaru be107df7f1 status bar: actually display it every second, after updating its data 2020-12-03 11:41:40 +02:00
cheatfate 42dd59852b Fix compilation problem. 2020-12-03 11:40:32 +02:00
cheatfate 0904ee5609 get_v1_validator_attestation_data() implementation. 2020-12-03 11:40:32 +02:00
cheatfate f6285d8381 Fix proposeSignedBlock() become sync. 2020-12-03 11:40:08 +02:00
cheatfate 088b001388 Fix get_v1_beacon_states_stateId_committees_epoch() to use optional arguments properly.
Optimize get_v1_beacon_pool_attestations() a bit.
2020-12-03 11:40:08 +02:00
cheatfate 21ef0f3a57 Fixes for get_v1_beacon_states_stateId_validators() and get_v1_beacon_states_stateId_validators_validatorId(). 2020-12-03 11:40:08 +02:00
cheatfate 9afafb54f5 get_v1_beacon_states_stateId_validator_balances() implementation. 2020-12-03 11:40:08 +02:00
cheatfate 5dade7b441 post_v1_beacon_blocks() implementation. 2020-12-03 11:40:08 +02:00
cheatfate 5f392893c0 Change approach for get_v1_beacon_pool_attestations() to use hexadecimal strings. 2020-12-03 11:40:08 +02:00
cheatfate 9e7f8b37c7 get_v1_beacon_pool_attestations() implementation. 2020-12-03 11:40:08 +02:00
cheatfate 7e4ff7a740 Implementation of post_v1_beacon_pool_attester_slashings() and post_v1_beacon_pool_proposer_slashings().
Remove doAssert.
2020-12-03 11:40:08 +02:00
cheatfate 6cae25701b Implementation of beacon_pool_attester_slashings(), beacon_pool_proposer_slashings() and beacon_pool_voluntary_exits(). 2020-12-03 11:40:08 +02:00
Zahary Karadjov 027d2547cd Fix a block proposal issue caused by incorrect merkle proofs
The key change here is that `addChunksAndGenMerkleProofs` is called
with all pending deposits instead of just the deposits included in
the block. The later was effectively producing merkle proofs against
a different root.
2020-12-03 11:25:52 +02:00
Zahary Karadjov 43662be3f8
v1.0.1 2020-12-02 14:31:13 +02:00
tersec d5e3712609
log errors in more detail when block production fails (#2135)
* log errors in more detail when block production fails

* pass through block header error

* make everything cstring
2020-12-02 13:23:10 +01:00
Sacha Saint-Leger 94c66f15a9
Wanted peers log message fix (#2114)
* Edit Peer count low wanted peers log

* remove prometheus folder

* remove prometheus.yml

* Add current peers to low peer count log

* change to 25% peer count notice, 12.5% warning

Co-authored-by: kdeme <kim.demey@gmail.com>
2020-12-02 10:47:23 +01:00
Zahary Karadjov 92655f936a Add backoff for the deposit logs requests as well; More debug logs 2020-12-02 01:43:54 +02:00
Zahary Karadjov 7225a6d6ed Retry all web3 requests up to 3 times with exponential backoff 2020-12-02 01:43:54 +02:00
cheatfate f3b8c777a7 Strip whitespace. 2020-12-02 01:43:36 +02:00
Zahary Karadjov 4feb0a308e Fix #2125 (ETH status bar display); Bump LibP2P 2020-12-02 00:03:59 +02:00
Dustin Brody 3e4b49462b detect excessively long graffiti strings 2020-12-01 15:39:46 +02:00
Dustin Brody 68c91d1d1b don't wait until after the first slot to enable gossip 2020-12-01 15:39:03 +02:00
Zahary Karadjov 2fd71a9aac Reduce the rate of requesting deposits to lower the risk of triggering the Infura rate limits 2020-12-01 15:38:50 +02:00
Zahary Karadjov 4eaf29875a Less risk for Eth1-induced delays in block proposal 2020-12-01 15:38:50 +02:00
Zahary Karadjov 38f7558e50 Work around a strange codegen issue to fix local sim in CI; Bump LibP2P 2020-12-01 15:38:00 +02:00
Zahary Karadjov 3bdda3dd46
Hotfix: use the mainnet bootstrap nodes without specifying --network=mainnet explicitly 2020-12-01 10:44:30 +02:00
Zahary Karadjov 4328576e18
Hotfix: 'deposits import' was ignoring its arguments in Linux builds 2020-12-01 00:59:57 +02:00
Zahary Karadjov 8327240458
v1.0.0-stateofus 2020-11-29 23:50:13 +02:00
Zahary Karadjov ac9bdde543
Don't rely on a metric value for the ETH display in the status bar 2020-11-29 23:35:39 +02:00
zah cabb07a186 Apply suggestions from code review
Co-authored-by: Sacha Saint-Leger <sacha@status.im>
2020-11-29 23:08:07 +02:00
Zahary Karadjov 3c0dfc2fbe Implement the 'deposits exit' command; Remove 'deposits create' 2020-11-29 23:08:07 +02:00
Jacek Sieka a2364ce1bc
add getGossipSubPeers RPC (#2107) 2020-11-29 20:07:20 +01:00
Zahary Karadjov ae19ab72c0 Implement #2067 2020-11-29 18:27:26 +02:00
Jacek Sieka c3bfe15a35
increase default max peers (#2106)
* increase default max peers

also avoid reconnection when opening stream as this might induce
a loop

* Use dial without addresses

* dial back max peers a little
2020-11-29 14:43:41 +01:00
cheatfate 7731109320 Remove getSpecPreset() because of crash and add missing calls into api.md 2020-11-29 11:01:55 +02:00
Zahary Karadjov bf2673abc4
Restore the ETH display in the status bar 2020-11-28 20:53:51 +02:00
Jacek Sieka 8685eb7042 add validator balance metrics 2020-11-28 16:22:09 +02:00
Eugene Kabanov db030eb944
Setup agent/proto version identifiers in libp2p. (#2079)
* Supply `nim-libp2p` with `agent-string` and `proto-string` as configured parameters.

* Remove protoString and rng check.

* Fix compilation problem.
2020-11-28 08:00:36 +01:00
Jacek Sieka e7f2735271
fix broken metrics during replay (#2090)
* move metrics out of state transition
* add validator count metric
* remove expensive beacon_current_validators, beacon_previous_validators
metrics (they should be reimplemented with cache), add cheap
beacon_active_validators to approximate
* remove unused validator count metrics
* tidy imports/defects
2020-11-27 23:16:13 +01:00
tersec 2421d338c1
node.network.metadata.attnets is only secondary source of truth (#2089) 2020-11-27 15:54:13 +01:00
Jacek Sieka d16e127daf
Peer stuff (#2084)
* Revert "Revert "Full "node" RPC calls implementation and fixes to peer lifetime states. (#2065)" (#2082)"

This reverts commit 7cc3dc8027.

* fix nil disconnectedFut crash

* fixes

don't resetPeer, it causes peer miscounts

* disconnect disconnecting peers

...when there's a race.

* avoid connection spamming

* never decrease SeenTable timeout
* only recover ENR for known peers

* seen only when really disconnected
2020-11-26 20:23:45 +01:00
Giovanni Petrantoni 9b9aa93cd4
Seenttl by spec (#2083)
* add rpc call to get chronos futures at runtime

* add seenTTL gossip parameter

Co-authored-by: Jacek Sieka <jacek@status.im>
2020-11-26 13:08:22 +01:00
Eugene Kabanov 687cbaf94c
Adjust number of sync workers from 20 to 10. (#2077)
Adjust watch task min-pause-time to 1.minute.
Slowed down pause-time recovery by factor 3/4 instead of 1/2.
2020-11-26 09:19:27 +01:00
Jacek Sieka 7cc3dc8027
Revert "Full "node" RPC calls implementation and fixes to peer lifetime states. (#2065)" (#2082)
This reverts commit d041287a4f.
2020-11-26 09:05:23 +01:00
Eugene Kabanov d041287a4f
Full "node" RPC calls implementation and fixes to peer lifetime states. (#2065)
* Initial commit.

* Fix log lines and compilation error.

* Add get_v1_node_peers() implementation.

* Fix peer's lifetime states.

* Use the most recent multiaddress.

* Fix assign NewPeerScore again.
Fix compilation error with last seen address.
Fix Peer upgraded log line place.

* syncing, health, peer_count, peer_id and fixes for identity.

* Fix compilation problems.

* Move object declaration to callsigs.
Fix identity addresses fields.

* Finish node RPC calls.

* Avoid leak of lifetime future.

* Bump chronos.

* Fix json generator problem.
2020-11-26 08:09:59 +01:00
kdeme e69b5ff473 Add a record create and print command 2020-11-25 18:32:59 +02:00
cheatfate 3a13750734 Try to obtain deposit contract address from config.
Attempt to fix json generation.
2020-11-25 18:30:44 +02:00
cheatfate 68f4ad64a4 Disable exception if depositAddress is not available. 2020-11-25 18:30:44 +02:00
cheatfate af3862da48 Make depositAddress public in eth1_monitor.
Fix representation of config.
Add deposit_contract implementation.
2020-11-25 18:30:44 +02:00
cheatfate 8eafbb4e09 Add config_spec implementation. 2020-11-25 18:30:44 +02:00
Zahary Karadjov 3594fa2a22
Version 1.0.0-rc1 2020-11-25 03:13:58 +02:00
Zahary Karadjov b9e4fef616
Check that the selected data dir is compatible with the selected network 2020-11-25 03:13:58 +02:00
Jacek Sieka e1e4871287 fix time formatting and pre-genesis ops 2020-11-25 01:51:46 +02:00
Zahary Karadjov 151d116f02
Fix off-by-one error in Eth1 pruning 2020-11-25 01:51:17 +02:00
Zahary Karadjov a995cdd3d9
Bugfix: fix an incorrect balance increase during genesis state generation 2020-11-24 23:30:26 +02:00
Zahary Karadjov 594ef713a0
Increase the web3 timeouts 2020-11-24 23:28:20 +02:00
zah 372c9b798c
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056)
* Handle some web3 timeouts better

* Add support for developer .env files

* Eth1 improvements; Mainnet genesis state

Notable changes:

* The deposits table have been removed from the database. The client
  will no longer process all deposits on start-up.

* The network metadata now includes a "state snapshot" of the deposit
  contract. This allows the client to skip syncing deposits made prior
  to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont
  and Mainnet.

* The Eth1 monitor won't be started unless there are validators attached
  to the node.

* The genesis detection code is now optional and disabled by default

* Bugfix: The client should not produce blocks that will fail validation
  when it hasn't downloaded the latest deposits yet

* Bugfix: Work around the database corruption affecting Pyrmont nodes

* Remove metadata for Toledo and Medalla
2020-11-24 22:21:47 +01:00
tersec 040093a945
medalla -> pyrmont (#2062) 2020-11-23 17:46:51 +01:00
Giovanni Petrantoni 6f476ca380
Gossip11 params (#2047)
* add rpc call to get chronos futures at runtime

* add eth2 spec gossip parameters

* bump libp2p

* fix url, remove 1.0

Co-authored-by: Jacek Sieka <jacek@status.im>
2020-11-23 10:57:03 +01:00
Jacek Sieka 672915e170
work around long pyrmont startup time (#2060)
* also fixes unnecessary copy/memory alloc when loading DbSeq entries
2020-11-21 18:53:40 +01:00
Zahary Karadjov 00aedddf2a
Version 0.6.6 2020-11-20 22:31:03 +02:00
Jacek Sieka dbcc0686ff delay pruning of cache for finalized epoch (fixes #2049) 2020-11-20 20:57:50 +02:00
Zahary Karadjov 316a19af5f Address review comments 2020-11-20 17:13:51 +02:00
Zahary Karadjov d232f16b40 Add more eth1-related logging 2020-11-20 17:13:51 +02:00
Zahary Karadjov e22248bca1 Tentative fix for the Pyrmont Eth1 voting issue 2020-11-20 17:13:51 +02:00
tersec 3c4cf95d0e
remove v0.12.3 and medalla support (#2038) 2020-11-20 14:49:49 +01:00
tersec 54c388b7b4
close slashing protection database (#2050) 2020-11-20 14:23:55 +01:00
tersec 1d7fb2ed0c
remove {.inline.} pragmas (#2033)
* remove {.inline.} pragmas

* re-add inline on bitseqs functions and tweak inlining threshold

* remove macOS/LLVM inlining setting; revert non-init/module-local/tests inline pragma removal
2020-11-20 11:00:22 +01:00
Zahary Karadjov 883e18da81
Hotfix: don't crash on web3 provider errors during initialization 2020-11-20 10:26:47 +02:00
Jacek Sieka 771c1d092a
hotfix fork choice score application (#2041)
Depending on the order of nodes in fork choice, it seems that the break
here could cause some scores not to be applied correctly
2020-11-19 15:11:08 +01:00
Zahary Karadjov e370e80aee
Add more Pyrmont bootstrap nodes 2020-11-18 14:40:45 +02:00
Zahary Karadjov 6b6e9b4d77
v0.6.4 Script and instructions for joining Pyrmont; WIP page in the book 2020-11-18 00:27:22 +02:00
Zahary Karadjov 11e1a9e8e8
A hacky work-around for a web3 issue that may cause the client to go into a loop of failing requests 2020-11-18 00:27:21 +02:00
Zahary Karadjov ebfacf597c
Review and fix some usages of Future.cancel; Bump Chronos 2020-11-18 00:27:21 +02:00
Ștefan Talpalaru 9c5cef346b set file and dir permissions 2020-11-17 23:28:26 +02:00
Jacek Sieka 4479c0a9f1
fix some libp2p channel hangs (#2032)
* fix some libp2p channel hangs

* make sure no concurrent reads are issues by awaiting cancellations
* bumps with fixes

* merge bump
2020-11-17 16:07:05 +01:00
tersec 9e716b32bd
address some XDeclaredButNotUsed hints (#2028) 2020-11-17 11:14:53 +01:00
Jacek Sieka a6b188bfd4
misc fixes (#2027)
* log when database is loading (to avoid confusion)
* generate network keys later during startup
* fix quarantine not scheduling chain of parents for download and
increase size to one epoch
* log validator count, enr and peerid more clearly on startup
2020-11-16 20:15:43 +01:00
tersec 21c4ce8fd4
remove superfluous TODOs/not-really-TODOs, type conversion, imports (#2025) 2020-11-16 17:10:51 +01:00
cheatfate 19707e89c2 Change random network key log line and include PeerID. 2020-11-16 17:41:23 +02:00
Jacek Sieka 5b011b65ae add rpc call to get chronos futures at runtime 2020-11-16 12:31:39 +02:00
tersec e2f161dbf7
fix attestation sending schedules to avoid timing attack (#1995) 2020-11-16 10:44:18 +01:00
Zahary Karadjov b022dc4d1f Use O(n) algorithm in initialize_beacon_state_from_eth1; Avoid unnecessary merkle proofs generation 2020-11-15 21:40:40 +02:00
Zahary Karadjov 8012102704 Bugfix: the client can miss the genesis event in the absence of new deposits 2020-11-15 21:40:40 +02:00
Zahary Karadjov 9f28a464fb
[skip ci] Update the version number 2020-11-14 23:26:46 +02:00
kdeme d02d8f8d51 Drop UseDiscv51 flag and only support discovery v5.1 2020-11-14 19:12:30 +02:00
Zahary Karadjov 80ca7a2d9f Implement #1768 2020-11-12 22:29:43 +02:00
Zahary Karadjov 17d35e1fd9 Allow the node to start when it fails to initialize the Eth1 monitor
* Avoid hangs when wss:// is specified for a non-secure HTTP server
* Produce an ERROR when the web3 provider is unsupported, but still launch the node
2020-11-12 22:29:43 +02:00
Zahary Karadjov 5e45e7429e Rewrite all HTTP(S) web3 URLs to WebSocket URls 2020-11-12 22:29:43 +02:00
cheatfate 87c92ea05f Address review comments.
Remove `-8.0` bucket because `-4.0` is a limit.
2020-11-12 21:52:01 +02:00
cheatfate dbb058682c Add beacon_attestation_sent_delay metric. 2020-11-12 21:52:01 +02:00
Zahary Karadjov 8ad45de65b
Add makefile targets for Pyrmont 2020-11-12 21:51:15 +02:00
tersec b9e122a60c
address some TODOs (#2005) 2020-11-12 19:24:07 +00:00
Eugene Kabanov e035b7d0c9
Fix status rpc request to use RESP_TIMEOUT. (#2002) 2020-11-12 15:30:12 +00:00
tersec 754d65c159
non-spec aggregate validation check now in spec; update genesis times (#2001) 2020-11-12 15:29:32 +00:00
tersec a0621d52f6
bump remaining rc.0 spec ref (#2003) 2020-11-12 13:58:41 +00:00
cheatfate f642f71457 Calculate how much time syncing takes, and show it at statusbar. 2020-11-12 14:16:12 +02:00
Jacek Sieka 332e248d91
rpc: setLogLevel (#1991)
* rpc: setLogLevel

also change default rpc port to match makefile and manual

* set topic-level formats too

and make feature less secret
2020-11-12 11:46:02 +01:00
Jacek Sieka 43c6c5a0b3
Gossip message id should decompress data (#1998)
* also fix missing return on empty message
2020-11-12 11:45:28 +01:00
Jacek Sieka 5644cfae8f
increase max db object size (fixes #1996) 2020-11-12 08:42:45 +01:00
Jacek Sieka c5710d97fc
default to gossipsub 1.1 (#1992) 2020-11-11 13:30:18 +01:00
Zahary Karadjov 389c11743a
Review TODO items and self-assign the most important ones 2020-11-10 20:41:04 +02:00
Jacek Sieka 6e40ca5e15 Use separate cache for getting epochref
...else tmpState gets overwritten when producing blocks and requesting
epochRef for finalized state.
2020-11-10 17:40:32 +02:00
Eugene Kabanov ea4d94c65c
Add guard task to monitor synchronization worker Future. (#1972)
* Add guard task to monitor synchronization worker Future.

* Change log events from debug to warning level.
2020-11-10 14:47:26 +01:00
tersec 7eaae89263
keep AllTests-mainnet.md consistent; ensure don't send overly large messages aren't sent (#1973) 2020-11-09 18:11:09 +00:00
tersec a010ef2061
update tests from v1.0.0-rc.0 to v1.0.0 (#1976) 2020-11-09 17:44:18 +00:00
Zahary Karadjov bd1047b715 0.6.0 release fixes
* Updated README
* No double v in nimbus_beacon_node --version
* No md5 checksums
2020-11-09 17:09:49 +02:00
tersec 271df8b604
bump 1.0.0rc-0 spec refs to 1.0.0 (#1974) 2020-11-09 14:18:55 +00:00
Jacek Sieka 61a3209c24
remove version quotes 2020-11-09 13:37:58 +01:00
Zahary Karadjov 94e9ce3c1f Add metadata for the toledo network 2020-11-09 11:38:52 +02:00
Zahary Karadjov 4b603d7f5a Change the default log level to INFO 2020-11-09 11:38:52 +02:00
Zahary Karadjov e9b9cd75ee Rename binaries; Mimic the original repo layout in the distribution 2020-11-09 11:38:52 +02:00
Ștefan Talpalaru bf58915500
release fixes (#1966) 2020-11-07 08:46:53 +01:00
Jacek Sieka d2476b643a
bump version (#1964)
* bump version

* update version format to "standard" style

* remove zinken/attacknet (both networks deprecated)

* add mainnet warning
2020-11-07 08:45:55 +01:00
kdeme fb6ce5c804 Enable discv5.1 for general beacon_node and inspector builds 2020-11-07 01:27:02 +02:00
Zahary Karadjov ac1072b4bb Add ./run-medalla-beacon-node script 2020-11-07 01:25:02 +02:00
Zahary Karadjov 4a710252d8 Check the web3 provider for network compatibility 2020-11-07 01:25:02 +02:00
Zahary Karadjov 24f8b7d483
[skip ci] Clean up some accidentally commited code 2020-11-06 11:53:44 +02:00
Zahary Karadjov a8a66fe3f6 Fix #1784; Handle Infura HTTPS URLs 2020-11-06 11:15:32 +02:00
Zahary Karadjov f596a24c4c Mainnet deposit contract monitoring 2020-11-06 11:15:32 +02:00
Jacek Sieka 95f5f76180
Datatype cleanup (#1953)
* clear up spec todo

* test fix

* remove unnecessary toSszType

* type

* one more
2020-11-04 21:52:47 +00:00
Jacek Sieka fc7885b27e Store block summary in database
This introcudes a cache for block summaries, useful for instantiating
the block dag on startup, bringing medalla startup times down from
minutes to seconds.

This is something of a temporary band-aid that would be obsoleted by a
finalized block store.
2020-11-04 11:28:55 +02:00
Zahary Karadjov 7823c25ba8
Hotfix for a failing assert during block production 2020-11-04 11:06:58 +02:00
Sacha Saint-Leger 36b54fe091 Update eth2_network.nim
More informative and friendlier log message (from the user's perspective). Also helps avoid the potential for confusion with the validator key.
2020-11-03 23:23:32 +02:00
Zahary Karadjov 6ee80651c7 Rename mainchain_monitor to eth1_monitor 2020-11-03 23:23:10 +02:00
Zahary Karadjov 2f29e3e7f3 Further reduce the number of RPC requests; Significantly faster ETH1 syncing 2020-11-03 23:23:10 +02:00
Zahary Karadjov 14b2d4324d openarray -> openArray 2020-11-03 23:23:10 +02:00
Zahary Karadjov 18639c3eff Don't require requests that might fail on non-archive Geth nodes 2020-11-03 23:23:10 +02:00
Zahary Karadjov 79f1b6117e Remove dead code from the beacon node binary; Fix 'make test' 2020-11-03 23:23:10 +02:00
Zahary Karadjov 07206ca264 An efficient algorithm for incremental generation of multiple merkle proofs 2020-11-03 23:23:10 +02:00
Zahary Karadjov d913273dcc Fix #1823 2020-11-03 23:23:10 +02:00
Jacek Sieka 937decb0e9
Revert "Gossip11 post audit (#1917)" (#1947)
This reverts commit 63173ab2c1.

It appears the cluster is having trouble staying connected - since the culprit is unknown, this is a first step on the way to what was stable.

Notably, this does not fully revert libp2p itself, merely the gossip version.
2020-11-03 11:45:40 +01:00
Sacha Saint-Leger 9255945fb0
Update mainchain_monitor.nim (#1945)
Make it clear that this failure concerns the eth1 chain
2020-11-02 21:16:24 +01:00
Ștefan Talpalaru 002a3dfa54
graceful shutdown: move the hook registration (#1940)
...closer to where it starts having an effect.
2020-11-02 19:02:27 +01:00
Jacek Sieka 2f049dc6e7
Fix init condition for fork choice (#1941)
... or we choose the wrong head on startup, sometimes.
2020-11-02 18:51:08 +01:00
Jacek Sieka ca63d48b82
tone down validator exit logging (#1939)
Validators exiting is normal, no need to scream about it

* avoid reallocating seq on big exit queue
* avoid fetching state cache when updating head (it's rarely needed)
* remove incorrectly implemented live validator counts (avoids memory
allocs)
2020-11-02 18:34:23 +01:00
Sacha Saint-Leger 7294d98ed1
Update validator_duties.nim (Improve node out of sync message) (#1933)
* Update validator_duties.nim

Improve node out of sync message

* Update validator_duties.nim
2020-11-02 08:52:16 +01:00
Zahary Karadjov 67113373f9 Work around BLST build issues 2020-10-30 23:56:04 +02:00
Zahary Karadjov f76679810b Fix #1855; Add support for wallet recovery passwords 2020-10-30 23:56:04 +02:00
Giovanni Petrantoni 63173ab2c1
Gossip11 post audit (#1917)
* bump libp2p gossip

* use gossip 1.1

* use proper libp2p commit hash

* switch to master libp2p, has audit fixes now
2020-10-30 14:26:35 +00:00
Eugene Kabanov 3f5c7c36bc
Some syncing fixes (#1919)
* Add exponential rewind on MissingParent.

* Try to avoid peers which are useless for syncing.
Fix forward sync restart at proper point.
Fix getLocalWallSlot() to not return slots from the future.

* Fix incorrect logs.

* Fix logging.
Enable peer's status messages log on DEBUG level.

* Fix watch task to monitor operation progress, but not local head progress.

* Add more logging information.
Remove recurring failures detection mechanism.
2020-10-30 13:33:52 +01:00
Jacek Sieka 81f4fe0783
limit fork choice initialization horizon (#1910)
This helps manage the long startup time on chains that are not
finalizing for long periods of time
2020-10-29 12:09:03 +01:00
Jacek Sieka 020a32ffa4
minor api cleanups (#1913)
* readd getNodeVersion
* implement get_v1_config_fork_schedule
* fix formatting of several types
2020-10-28 19:51:38 +01:00
tersec 19c51e5900
use MAXIMUM_GOSSIP_CLOCK_DISPARITY for BeaconBlock verification (#1918) 2020-10-28 14:04:21 +01:00
Jacek Sieka ff3f8741de
handle fork choice selecting a head unknown to chaindag (#1907)
* move head slot metric to head selection
* avoid redundant head selection in validator duties
2020-10-28 07:55:36 +00:00
Jacek Sieka 6050eebec0
document not-nil issues 2020-10-27 13:01:45 +01:00
Eugene Kabanov eee01a32d5
Regression fix of eth2_network_simulation on Windows. (#1900)
* Concentrate all sensitive writeFile/createPath calls in one place.
Fix eth2_network_simulation for Windows.

* Remove artifacts.

* fix import

Co-authored-by: Jacek Sieka <jacek@status.im>
2020-10-27 12:04:17 +01:00
Eugene Kabanov c82ff24b5c
Syncing fixes (#1909)
* Fix continuous sync queue rewinds on slow PCs.
Fix recurring disconnects on low peer score.

* Calculate average syncing speed, not the current one.
Move speed calculation to different task.

* Address review comments.
2020-10-27 10:25:28 +01:00
Jacek Sieka ee2ebc96a8
move rpc api to own folder, mimic upstream structure (#1905)
also implements a few more endpoints
2020-10-27 10:00:57 +01:00
Mamy Ratsimbazafy 677f099110
BLS signature v4 (#1912) 2020-10-27 09:37:04 +01:00
Kim De Mey 8102d5dc4d
Bump eth2-testnets for more discv5.1 bootstrap nodes on medalla (#1911)
And improve logging UX a bit by ignoring comments or newlines
2020-10-27 08:44:46 +01:00
Kim De Mey 756b62fdf5
Bump nim-eth to get UseDiscv51 flag (#1895)
* Bump nim-eth to get UseDiscv51 flag

* Switch medalla to discovery v5.1, other targets to v5.0

* Bump nim-eth for better discv5.1 logging

* Bump eth2-testnets for updated medalla bootnodes
2020-10-23 18:59:58 +02:00
Jacek Sieka 7c0b4d28d2 speed up reward/penalty calculation
Calculating rewards/penalties is slow due to how we compute sets of
attestations validators then use the sets for inclusion checks, to see
who attested. The dominant function during validated block processing /
epoch processing is hash set building and lookup.

This PR inverts the flow by removing the sets and creating a single
large validator status list, then applying all relevant state
attestations, then updating rewards and penalties.

This provides a 10x speedup to epoch processing which in turn speeds up
both empty slot and block processing - for example, on startup, we
replay all non-finalized blocks to prime fork choice - the same when
validating attestations or replaying states on reorg.
2020-10-23 19:23:36 +03:00
Jacek Sieka 499e5ca991
misc memory and perf fixes (#1899)
* misc memory and perf fixes

* use EpochRef for attestation aggregation
* compress effective balances in memory (medalla unfinalized: 4gb ->
1gb)
* avoid hitting db when rewinding to head or clearance state
* avoid hitting db when blocks can be applied to in-memory state -
speeds up startup considerably
* avoid storing epochref in fork choice
* simplify and speed up beacon block creation flow - avoids state reload
thanks to head rewind optimization
* iterator-based committee and attestation participation help avoid lots
of small memory allocations throughout epoch transition (40% speedup on
epoch processing, for example during startup)

* add constant for threshold
2020-10-22 12:53:33 +02:00
tersec decfd66a78
bump libp2p: seq[byte] message IDs and improve reject/ignore handling (#1898)
* bump libp2p: use seq[byte] message IDs and improve reject/ignore handling

* bump libp2p to pick up debug message details
2020-10-21 20:03:12 +00:00
zah 9810a31447
Fix #1838 (#1892) 2020-10-20 15:01:21 +02:00
tersec a136c2e95a
bump libp2p; integrate pubsub.ValidationResult into extended validation (#1893) 2020-10-20 12:31:20 +00:00
tersec e87e0ef179
update to 1.0.0-rc.0 message ID; unexport funcs; enable test in mainnet; remove unused subscribe-with-nondummy-handler function; fix gossip received metric (#1884) 2020-10-20 08:54:11 +00:00
tersec c446eb7cff
update v1.0.0-rc.0 preset spec references (#1891)
* update ve1.0.0-rc.0 preset spec references

* remove runtime preset ETH1_FOLLOW_DISTANCE from preset files; remove two CI build items to try to keep Travis from timing out
2020-10-19 13:53:23 +00:00
tersec 5b13a5ba10
update attestation extended validation to v1.0.0-rc.0 (#1889)
* update attestation extended validation to v1.0.0-rc.0

* attestation block and target must be within same epoch

* remove duplicate attestation epoch/target epoch check

* in spec now
2020-10-19 09:25:06 +00:00
Jacek Sieka df43b8aa8b
save some more states after all (#1887)
Don't save states when replaying history, but do save states when
applying new blocks (!)
2020-10-18 15:47:39 +00:00
Zahary Karadjov dbc90e998a Add Disc v5.1 bootstrap nodes for Zinken
Also prepares the Makefiles for supporting Medalla
through a separate binary
2020-10-18 12:44:52 +03:00
Zahary Karadjov d98be078c3 Address #1690 Min password entropy 2020-10-16 19:34:26 +03:00
Zahary Karadjov 9899578e1c Update get_eth1_data from v0.11.1 to v1.0.0-rc 2020-10-16 11:46:53 +03:00
Dustin Brody 9a543e0af7 partial hotfix for #1879 crash 2020-10-16 11:46:19 +03:00
Zahary Karadjov 948d15ea95 Follow-up on #1687 Don't ask for password on invalid keystores 2020-10-16 11:45:39 +03:00
Viktor Kirilov db70f066ba fixed all crashes which can be triggered through the validator RPC API - fixes #1680 2020-10-15 20:19:43 +03:00
Viktor Kirilov 7ed885f144 configuring a writer for the dynamic log output device for the VC binary as well (same as in the BN) 2020-10-15 20:19:10 +03:00
cheatfate 7ad59b1852 Bump nim-stew. 2020-10-15 20:18:33 +03:00
cheatfate 1742a82ca7 Proper fix for writeFile() places.
Comment unused/insecure procedure.
2020-10-15 20:18:33 +03:00
cheatfate 84fe294c00 Fix protection of data folder and security files using Windows ACL. 2020-10-15 20:18:33 +03:00
Zahary Karadjov 5f6bdc6709 Store all deposit-derived data in memory 2020-10-15 20:15:51 +03:00
Zahary Karadjov 7a577b2cef More tests for getBlockRange 2020-10-15 20:15:51 +03:00
Zahary Karadjov 080609eee1 Address #1366 Avoid uint64 overflow in getBlockRange when skipStep is large 2020-10-15 20:15:51 +03:00
Zahary Karadjov 2152dc6136 Simplify the mainchain monitor 2020-10-15 20:15:51 +03:00
Zahary Karadjov ce1fda1195 Store the deposits and the immutable validator data in Sqlite 2020-10-15 20:15:51 +03:00
Zahary Karadjov e69af00e3a Don't penalize flooding peers; Just rate limit them 2020-10-15 20:15:51 +03:00
Zahary Karadjov 4d66914f5a Fix the test suite 2020-10-15 20:15:51 +03:00
Zahary Karadjov f0892ce934 Fix compilation problems on Windows 2020-10-15 20:15:51 +03:00
Zahary Karadjov e6320e5881 Address #1584 Don't keep all deposits in memory (persist them to disk) 2020-10-15 20:15:51 +03:00
Zahary Karadjov bc8acdb9de Switch to an in-house clearScreen to avoid problems with exception tracking 2020-10-15 20:15:51 +03:00
Zahary Karadjov 69e498dc00 Address #1689 and #1517 Usability and security improvements in wallet creation 2020-10-15 20:15:51 +03:00
Zahary Karadjov 02a3770803 Adds a fuzzing test for the validator client CLI
Also removes an obsolete way to add validators through the command line
2020-10-15 20:15:51 +03:00
Zahary Karadjov 8ce0fc3a89 Address #1687
This addresses the issues by detecting and rejecting keystores with
incorrect PBKDF2 and SCrypt params. It also bumps the version of
nim-json-serialization to include a bugfix for incorrect parsing
of json files featuring comments.
2020-10-15 20:15:51 +03:00
Zahary Karadjov cd949a2b81 Address #1691 More pervasive handling of Ctrl+C 2020-10-15 20:15:51 +03:00
Zahary Karadjov 8a6281aad2 Simple cost model for sync requests; Penalize peers perfoming flooding or invalid requests 2020-10-15 20:15:51 +03:00
Jacek Sieka 6b9419e547
fix db growth on attestation processing (#1860)
It turns out that we often save lots of states in the database that are
the result of empty slot processing only - here, we make sure to only
save a state if a block follows - this fixes several issues:

* empty slot states are not always pruned leading to state database size
explosion
* storing states is (very) slow which slows down processing in general,
so we should only do it when it's likely to be useful
* attestation processing doesn't get stuck on saving random states that
won't appear in the chain history
2020-10-15 14:28:44 +02:00
Viktor Kirilov 4e73d10f86
removed 2 buggy RPC API endpoints - there are 2 well functioning ones in validator_api.nim anyway (#1867) 2020-10-15 14:26:59 +02:00
tersec da59f45a90
update spec/datatypes spec refs from v0.12.3 to v1.0.0 (#1866) 2020-10-14 10:00:27 +00:00
tersec 513ba72b9a
add v1.0.0-rc.0 support behind compile-time flag (#1852)
* add v1.0.0-rc.0 support behind compile-time flag

* keep runtime presets consistent
2020-10-13 17:21:25 +00:00
tersec 3ee2dd8da4
p2p-interface spec ref bump (except non-updated places) (#1862) 2020-10-12 14:37:14 +00:00
tersec 1994ffe5a0
update 130+ spec references from v0.12.3 to v1.0.0-rc1 (#1854) 2020-10-12 08:59:24 +00:00
Zahary Karadjov 00a8a68671
Address #1695
Better error messages when the beacon node is asked to listen on a
reserved port (0) or an already taken one.
2020-10-09 16:39:03 +03:00
tersec d69d89866e
fix logtrace in local network sim (#1840) 2020-10-09 10:58:54 +02:00
tersec b79e5f8af5
update nim-beacon-chain to nimbus-eth2 in beacon_chain/, ncli/, tests/, and README.md (#1843) 2020-10-08 19:02:05 +00:00
tersec c0aa2f1e58
update process_slashings() to use PROPORTIONAL_SLASHING_MULTIPLIER (#1841) 2020-10-08 18:00:07 +00:00
Eugene Kabanov 3bd7ab4c20
Do not reward empty responses. (#1827)
Request status from peers every StatusExpirationTime.
2020-10-08 14:50:48 +02:00
tersec f08f44b9a2
in exit pool, bundle receive messages into beaconblocks (#1812)
* in exit pool, filter out already-packaged messages; bundle remaining messages into beaconblocks

* filter messages at block construction time

* allow adding up to intended capacity of buffers, beyond per-block limits

* document rationale/design for filtering mechanism
2020-10-07 16:57:21 +00:00
Zahary Karadjov b46c8410a6 Retire Spadina and prepare for the Zinken network 2020-10-07 18:12:21 +03:00
Mamy Ratsimbazafy 6b9090398e
[JSON-RPC] Prevent silent uint conversion (#1819)
* Prevent silent uint conversion

* bump json-rpc
2020-10-07 15:02:54 +02:00
Zahary Karadjov deaddc1fc0 Address review comments 2020-10-07 09:32:03 +03:00
Zahary Karadjov aed291128a Add support for starting from weak subjectivity checkpoints
Also removes the `genesis.ssz` file stored in the data folder.
The `medalla-fast-sync` target has been adapted to use the new features.
2020-10-07 09:32:03 +03:00
Zahary Karadjov 2c06492407 Address #1728 2020-10-06 15:46:22 +03:00
Zahary Karadjov 950bcd0f6d Address #1729: Min password length 2020-10-06 15:46:22 +03:00
Zahary Karadjov 78953fd9b5 Address #1729: NFKD Normalization 2020-10-06 15:46:22 +03:00
Zahary Karadjov fec4b5014d Address #1694 2020-10-06 15:46:22 +03:00
Zahary Karadjov 47d4899911 Wallet creation does not allow overwriting an existing wallet 2020-10-06 15:46:22 +03:00
Zahary Karadjov 9d23094c68 Address #1692 2020-10-06 15:46:22 +03:00
Eugene Kabanov 27fc8bcc80
Address #1538 and do not penalize peers for empty response. (#1811) 2020-10-06 14:10:02 +02:00
Mamy Ratsimbazafy 6bb6b66766
Partial LTO (#1789)
* initial partial LTO example

* Add select private key related files to ignore LTO

* Apply to keystores
2020-10-06 13:38:18 +02:00
Mamy Ratsimbazafy c92d228ec8
Fix non-split attesting protection (#1804)
* address #1679 - att protection in non-split mode

* AttachedValidator public_key -> pubkey

* Always build with slashing protection

* Remove now unnecessary forwarding impl proc
2020-10-06 10:51:33 +02:00
tersec 5354a0a3d5
fix #1585 and restore storeLEBytes on big-endian systems (#1796)
* fix #1585 and restore storeLEBytes on big-endian systems

* switch to using stew/endians2

* create static endians2-like shim in bitseqs
2020-10-06 08:00:11 +00:00
Viktor Kirilov 8c59b92081
fixed #1663 - Interger overflow in compute_start_slot_at_epoch through RPC (#1810)
* fixed #1663 - Interger overflow in compute_start_slot_at_epoch through RPC

* changed the way the overflow check is done - took the approach from PR #1797 - see the comment in PR #1810 for more details
2020-10-05 21:58:30 +00:00
cheatfate 00a500ef3c Fix rebase issues. 2020-10-05 22:19:50 +03:00
cheatfate f091c8d4df Add chronicles.formatIt for PublicKey.
Fix logs.
Rename checkFilePermissions to checkSensitiveFilePermissions.
2020-10-05 22:19:50 +03:00
cheatfate dd8b73c85c Remove `keypath` and set version to 1 for network keystorage. 2020-10-05 22:19:50 +03:00
cheatfate 9441e912cb Merge different places of keyboard input to keyboardCreatePassword() and keyboardGetPassword().
Fix Windows problem with english words array.
Add checksum for english words array verification.
2020-10-05 22:19:50 +03:00
cheatfate 868ce14762 Fix compilation issue. 2020-10-05 22:19:50 +03:00
cheatfate 82228fe471 Fix last createDir. 2020-10-05 22:19:50 +03:00
cheatfate 86139839f1 Add permissions checks and handling to wallets and bls keystores. 2020-10-05 22:19:50 +03:00
cheatfate fc09c2b3d4 Fix compilation problem. 2020-10-05 22:19:50 +03:00
cheatfate 1d76683a8b Fix os dependency. 2020-10-05 22:19:50 +03:00
cheatfate 20f695515a Fix localtestnet simulation. 2020-10-05 22:19:50 +03:00
cheatfate add22a20e1 Update local_testnet and simulation scripts to use netkey-file and insecure-netkey-password.
Add more logging
2020-10-05 22:19:50 +03:00
cheatfate e1182f8000 Add insecure password for automated testing.
Fix checkDataDir to run before setupLogging.
2020-10-05 22:19:50 +03:00
cheatfate e43af4e978 Fix CLI option. 2020-10-05 22:19:50 +03:00
cheatfate 40f2b74f73 Add keystore management and interactive password handling. 2020-10-05 22:19:50 +03:00
cheatfate ae72c08573 Fix utils problem. 2020-10-05 22:19:50 +03:00
cheatfate cec4ad2717 Refactor keystore.nim to include network key storage too.
Use constant-time hex encoding/decoding procedures.
Add tests for network key storage.
2020-10-05 22:19:50 +03:00
cheatfate c5c788a9db Secure network key file and data directory. 2020-10-05 22:19:50 +03:00
Zahary Karadjov d9738b43b3
Revert "Address issue #1538."
This reverts commit 1be3fd53b1.
2020-10-05 19:25:47 +03:00
cheatfate 1be3fd53b1 Address issue #1538. 2020-10-05 13:23:25 +03:00
cheatfate 67a93f558b Address issue #1714. 2020-10-05 13:22:45 +03:00
Jacek Sieka 99afafecd7
fix quadratic seq assignment in fork choice (#1805)
this would reallocate the attestation queue on every attestation and
other call to update_time, causing quite the overhead (~10% cpu spent
when gossiping)
2020-10-03 23:43:27 +02:00
Mamy Ratsimbazafy b57693ec0d
Logging update (#1795)
* Fix discovery log message trigger

* Bump chronicles - include https://github.com/status-im/nim-chronicles/pull/89 for better NOTICE/WARNING color
2020-10-03 08:35:45 +02:00
tersec 5e95fd7468
Revert "update to v0.12.3 message ID for Spadina launch (#1762)" (#1801)
This reverts commit a2270a5f27.
2020-10-02 19:50:21 +00:00
Zahary Karadjov f5340998b9 Add a fuzzing test for the beacon node CLI parser 2020-10-02 12:42:32 +03:00
tersec ee114deb82
fix #1782 by checking for empty attester index sets (#1786) 2020-10-01 21:05:22 +02:00
Mamy Ratsimbazafy 0280d6c73e
Revisiting log levels (#1788)
* Update log level - https://github.com/status-im/nim-beacon-chain/issues/1779 https://github.com/status-im/nim-beacon-chain/issues/1785

* Address review comments

* Document the logging strategy [skip ci]
2020-10-01 20:56:42 +02:00
tersec 13d4831664
fix #1783 by not assuming that an attached validator will attest (#1787) 2020-10-01 14:48:24 +00:00
tersec 7eaaab908c
fix output of proposer slashing test fixture (#1780)
* fix output of proposer slashing test fixture

* run make test

* a few more v0.12.3 spec refs
2020-09-30 13:12:03 +00:00
Giovanni Petrantoni 0c921b7bc0
Multinet fixes (#1774)
* add missing hardcoded preset value

* wip

* wip
2020-09-29 10:12:57 +02:00
Kim De Mey f6b8e26567
Fix typos [skip ci] (#1771) 2020-09-28 20:43:09 +02:00
Viktor Kirilov 2e6fd8976a
Ctrl+C now stops the BN when waiting for genesis (#1761)
* Ctrl+C now stops the BN when waiting for genesis

* proper shutdown when waiting for genesis if Ctrl+C is used
2020-09-28 17:19:57 +02:00
tersec a2270a5f27
update to v0.12.3 message ID for Spadina launch (#1762)
* update to v0.12.3 message ID for Spadina launch

* remove base64 import
2020-09-28 17:07:10 +02:00
Kim De Mey 23bec99341
Let also discovery used listen-address cli option instead of always any address (#1658) 2020-09-27 22:00:24 +02:00
Jacek Sieka c472d5368b
avoid attestation pool copy in check (#1755) 2020-09-25 19:51:44 +02:00
Mamy Ratsimbazafy 94120ad011
Slash protection fixes (#1757)
* Address #1698

* Add test for #1699
2020-09-25 19:39:06 +02:00
Jacek Sieka 7837646079
anonymize libp2p messages (#1756)
* anonymize libp2p messages

* bump
2020-09-25 18:40:30 +02:00
Zahary Karadjov b6262bfe46
Bugfix: prevent an error during deposit data generation 2020-09-25 17:11:36 +03:00
Eugene Kabanov 1bf8d3af33
Disconnect peers with low score. (#1747)
* Disconnect peers with low score.

* Change PeerScoreLow value.

* Add spec url for DisconnectionReason.
2020-09-25 15:43:45 +02:00
tersec f96ad87d28
switch another 50+ spec refs from v0.12.2 to v0.12.3 (#1749) 2020-09-25 11:52:50 +00:00
tersec 6cf7e837ce
refactor exit pool and gossipsub-subscribe to exit pool message types (#1733)
* refactor exit pool and gossipsub-subscribe to all exit pool message types

* remove unused loop counter
2020-09-24 19:05:49 +02:00
tersec 6398a43cc1
update 120+ beacon_chain and validator spec refs from v0.12.2 to v0.12.3 (#1740) 2020-09-24 19:04:10 +02:00
Jacek Sieka b3a9afa0b1
libp2p: limit max gossip writes (#1739)
* libp2p: limit max gossip writes

* bump
2020-09-24 19:03:17 +02:00
tersec 0eb53f2802
avoid unpacking phase 1 tests to reduce Azure CI disk usage (#1736) 2020-09-24 17:16:00 +02:00
Viktor Kirilov 410a72667b fixed interop with geth after updating nim-web3 2020-09-24 16:07:57 +03:00
Eugene Kabanov 2cd0c3adaa
Fix condition. (#1734) 2020-09-24 14:14:29 +02:00
Viktor Kirilov b9deff43ce fixed build after rebasing on top of the recently introduced slashing protection 2020-09-24 11:31:14 +03:00
Viktor Kirilov e926db22b9 initial implementation of attestation aggregation for the validator API 2020-09-24 11:31:14 +03:00
Jacek Sieka f0dbebfd3f
avoid storing empty slot states (#1720)
with the improved empty slot processing, these provide relatively little
benefit, but take up lots of storage that's difficult to free
2020-09-24 09:02:03 +02:00
Mamy Ratsimbazafy 4e23b0ef23
Fix https://github.com/status-im/nim-beacon-chain/issues/1701 + raise concerns in keystore secrets protection (#1726) 2020-09-24 07:27:56 +02:00
Eugene Kabanov 08795b3f5d
Fix tight loop at the end of sync process. (#1731) 2020-09-23 17:58:02 +02:00
Jacek Sieka e1c177cdd1
bump libp2p (#1721)
gossipsub 1.1 can be enabled with -d:nbc_gossipsub_11
2020-09-22 19:34:34 +02:00
Zahary Karadjov 73724b2c4b Add attacknet-beta1-mc-0 2020-09-22 18:51:51 +03:00
Zahary Karadjov 10cfd1ec37 Adds the Spedina testnet; Upgrades the spec presets to spec v0.12.3
Other changes:

* Removed all references to Altona
* Refactored the Makefile to avoid code duplication
2020-09-22 18:51:51 +03:00
Eugene Kabanov 654b8d66bf
Peer management (#1707)
* addPeer() and addPeerNoWait() now returns PeerStatus, not bool.
Minor refactoring of PeerPool.
Fix tests.

* Refactor PeerPool.
Add lenSpace.
Add tests for lenSpace.
PeerPool.add procedures now return different error codes.
Fix SyncManager break/continue problem.
Fix connectWorker break/continue problem.
Refactor connectWorker and discoveryLoop.
Fix incoming/outgoing blocking problem.

* Refactor discovery loop.
Add checkPeer.

* Fix logic and compilation bugs.

* Adjust position of debugging log.

* Fix issue with maximum peers in PeerPool.
Optimize node record decoding.

* fix discoveryLoop.

* Remove aliases and fix tests using aliases.
2020-09-21 18:02:27 +02:00
tersec 3190c695b0
minimal v0.12.3 update (#1716) 2020-09-21 15:58:35 +00:00
Jacek Sieka fc10f5121a protect against data after initial request
spec requires that channel is closed

also, avoid some unnecessary futures
2020-09-18 21:34:07 +03:00
zah 7c688bffda
Address #1586 (#1596) 2020-09-18 18:34:13 +02:00
Mamy Ratsimbazafy ea4ec6a785
Upgrade blst (#1682)
* Bump BLST

* Test for https://github.com/supranational/blst/issues/22 regression

* Use SHA256 from BLST + bump nim-blscurve to reenable fno-tree-vectorize

* SHA256 on non-blst platforms import fixes

* import fixes again

* can't prefix with nimcrypto

* address review comment [skip ci]

* {.noInit.} on the digests
2020-09-18 16:55:55 +02:00
tersec e106549efe
keep REJECT/IGNORE of messages failing validation for libp2p scoring (#1676)
* keep REJECT/IGNORE status of messages failing validation for libp2p scoring

* fix test suite
2020-09-18 13:53:09 +02:00
Dmitriy Ryajov 2f89e2ab4e
drop subscribePeer, it's called from pubsub now (#1677) 2020-09-17 11:40:21 +02:00
Jacek Sieka dcf8a6b05d
improve slot processing speeds (#1670)
about 40% better slot processing times (with LTO enabled) - these don't
do BLS but are used
heavily during replay (state transition = slot + block transition)

tests using a recent medalla state and advancing it 1000 slots:

```
./ncli slots --preState2:state-302271-3c1dbf19-c1f944bf.ssz --slot:1000
--postState2:xx.ssz
```
pre:

```

All time are ms
Average,       StdDev,          Min,          Max,      Samples,
Test
Validation is turned off meaning that no BLS operations are performed
39.236,        0.000,       39.236,       39.236,            1,
Load state from file
0.049,        0.002,        0.046,        0.063,          968,
Apply slot
256.504,       81.008,      213.471,      591.902,           32,
Apply epoch slot
28.597,        0.000,       28.597,       28.597,            1,
Save state to file
```

cast:
```
All time are ms
Average,       StdDev,          Min,          Max,      Samples,
Test
Validation is turned off meaning that no BLS operations are performed
37.079,        0.000,       37.079,       37.079,            1,
Load state from file
0.042,        0.002,        0.040,        0.090,          968,
Apply slot
215.552,       68.763,      180.155,      500.103,           32,
Apply epoch slot
25.106,        0.000,       25.106,       25.106,            1,
Save state to file
```

cast+rewards:
```
All time are ms
Average,       StdDev,          Min,          Max,      Samples,
Test
Validation is turned off meaning that no BLS operations are performed
40.049,        0.000,       40.049,       40.049,            1,
Load state from file
0.048,        0.001,        0.045,        0.060,          968,
Apply slot
164.981,       76.273,      142.099,      477.868,           32,
Apply epoch slot
28.498,        0.000,       28.498,       28.498,            1,
Save state to file
```

cast+rewards+shr
```
All time are ms
Average,       StdDev,          Min,          Max,      Samples,
Test
Validation is turned off meaning that no BLS operations are performed
12.898,        0.000,       12.898,       12.898,            1,
Load state from file
0.039,        0.002,        0.038,        0.054,          968,
Apply slot
139.971,       68.797,      120.088,      428.844,           32,
Apply epoch slot
24.761,        0.000,       24.761,       24.761,            1,
Save state to file

```
2020-09-16 20:59:33 +00:00
Mamy Ratsimbazafy 52548f079b
Opt-in Slashing protection + interchange (#1643)
* Slashing protection + interchange initial commit

* Restrict the when UseSlashingProtection dance in other modules

* Integrate slashing tests in other all_tests

* Add attestation slashing protection support

* Add a message that mention if built with/without slashing protection

* no op the initialization proc

* test slashing protection in Jenkins (temp)

* where to configure NIMFLAGS in Jenkins ...

* Jenkins -> ensure Built with slashing protection

* Add slashing protection complete import

* use Opt.get(otherwise)

* Don't use negation in proc name

* Turn slashing protection on by default
2020-09-16 13:30:03 +02:00
Eugene Kabanov 6e463257f4
PeerPool fixes. (#1654)
* Refactor peer_pool.
Fix eth2_network peer counters.
Fix PeerPool do not allow to add more peers when empty space available.

* Remove unused imports.

* Add test for a bug.

* Fix eth2_network disconnect should deletePeer not release.
More PeerPool refactoring.
2020-09-16 13:00:11 +03:00
tersec 6d8130dc49
close block_sim database; remove code duplication in exit_pool (#1656) 2020-09-16 09:16:23 +02:00
Eugene Kabanov 9abdbdabd8
Fix sync_manager.nim not rewarding peers for good responses. (#1660) 2020-09-16 09:15:06 +02:00
tersec 7e10b9850b
enable topic unsubscribing and attestation subnet cycling (#1646)
* enable topic unsubscribing and attestation subnet cycling

* remove refences to states

* waitFor -> await

* revert exit pool changes for a cleaner PR
2020-09-15 12:40:43 +00:00
Jacek Sieka c76305f824
fix some todo (#1645)
* remove some superfluous gcsafes
* remove getTailState (unused)
* don't store old epochrefs in blocks
* document attestation pool a bit
* remove `pcs =` cruft from log
2020-09-14 14:50:03 +00:00
tersec 9f21bbd666
[WIP] skeleton of attester slashing pool & validators (#1639)
* skeleton of attester slashing pool & validators

* add skeleton for proposer slashings and voluntary exits; rename pool to more inclusive exit pool to stay consistent with all three; ensure is initialized by beacon_node so is safe to merge, even if it doesn't do much yet
2020-09-14 16:26:31 +02:00
Ștefan Talpalaru 60b8905852
reorder startup steps (#1642)
so the Ctr+C hook and the metrics server go before the potentially slow
`BeaconNode.init()`
2020-09-14 08:24:31 +02:00
tersec aca1a318f2
cleanly close kvstore databases and bump nim-eth (#1630)
* cleanly close kvstore databases

* close databases for all subcommands and during error conditions
2020-09-12 05:35:58 +00:00
Kim De Mey 5fff800cf8
Bump nim-eth to use lrucache for discovery sessions (#1622) 2020-09-11 17:46:48 +00:00
tersec 48893f1c2e
add ncli_db subcommand to prune database of unnecessary blocks and states (#1593)
* add ncli_db subcommand to prune database of unnecessary blocks, states, and state roots

* tweak comments

* reduce default aggressiveness in pruning old states

* move copyPrunedDatabase() to ncli_db, as it's not generally useful as part of beacon_chain_db and doesn't use any internal interfaces
2020-09-11 15:20:34 +02:00
Eugene Kabanov c7c9b9d5f1
Syncing V2 (#1602)
* Syncing workers now not bound to peers.
Sync status is now printed in statusbar.

* Add `SyncQueue.outSlot` to statusbar too.

* Add `inRangeEvent` and `rangeAge` parameter.

* Fix rangeAge is not depends on SyncQueue latest slot.
Fix syncManager to start from latest local head slot.

* Add notInRange event.

* Remove suspects field.
2020-09-11 14:46:01 +02:00
Jacek Sieka 775683cf2e
await validator duties (#1626)
Validator duties proceed slot-by-slot - we should not start a new
validator duty iteration before the previous one is gone or we might run
into consistency and voting issues
2020-09-11 09:50:00 +00:00
Jacek Sieka 8a5a261fcd
Quick fix to prune some states, pending smarter state storage (#1624)
* Quick fix to prune some states, pending smarter state storage

Adverse effects might include slow rewinds - typically the protocol
doesn't ask for pre-finalized states but RPC might

* document issue, add test

* fix cache miss log
2020-09-11 10:03:50 +02:00