Commit Graph

1351 Commits

Author SHA1 Message Date
Dmitriy Ryajov 87f983c639 use split out pubsub 2020-08-17 17:24:36 +03:00
tersec 612881b95d
refactor topic (un)subscribing/validating to collate each (#1510)
* refactor topic (un)subscribing/validating to collate each

* fix comment

* tweak comment
2020-08-17 14:07:29 +02:00
tersec f34eddb6e9
fix get_unslashed_attesting_indices() and add official EF rewards tests for it (#1514) 2020-08-17 01:09:27 +00:00
tersec bc6eefe31e
add --enable-logtrace argument to launch_local_testnet (#1502)
* add --enable-logtrace argument to launch_local_testnet

* scan for all available logfiles

* remove specific filename references

* update v0.11.3 spec ref to v0.12.2
2020-08-16 11:12:19 +02:00
Mamy Ratsimbazafy 454b9d0724
Bump nim-blscurve (#1491)
* Bump BLSCurve

* Use unified aggregation API

* use new blscurve with unified aggregate API

* bump

* fix toRaw

* replace state_sim combine with AggregateSignature

* Fix 32-bit

* Fix 32-bit for real and test deactivating ccache for fno-tree-lopp-vectorize flag

* change compilation switches to narrow down Linux issue

* Use -fno-tree-vectorize to disable both tree-loop-vectorize and tree-slp-vectorize

* blscurve now disables both Loop and SLP vectorization

* Add tests for the miracl/milagro fallback

* Travis has max log size of 4MB

* Test with Miracl in the finalization test

* fix state_sim log level

* Coment out the slow fallback tests
2020-08-15 19:33:58 +02:00
tersec 611c5097cc
use cache in process_voluntary_exit() (#1507) 2020-08-14 12:42:59 +00:00
Dustin Brody 3d121d9734 remove quadratic deposit Merkle tree initialization 2020-08-14 12:33:58 +03:00
Jacek Sieka a1bd44f4b0
small spec cleanups (#1501)
* clean up logging a bit
* return error on indexed attestation check
2020-08-13 13:47:06 +00:00
Zahary Karadjov c765c5ae2d
Bugfix: Correct wallet by UIID search in 'deposits create' 2020-08-13 14:32:22 +03:00
Jacek Sieka 58d77153fc
fix invalid state root being written to database (#1493)
* fix invalid state root being written to database

When rewinding state data, the wrong block reference would be used when
saving the state root - this would cause state loading to fail by
loading a different state than expected, preventing blocks to be
applied.

* refactor state loading and saving to consistently use and set
StateData block
* avoid rollback when state is missing from database (as opposed to
being partially overwritten and therefore in need of rollback)
* don't store state roots for empty slots - previously, these were used
as a cache to avoid recalculating them in state transition, but this has
been superceded by hash tree root caching
* don't attempt loading states / state roots for non-epoch slots, these
are not saved to the database
* simplify rewinder and clean up funcitions after caches have been
reworked
* fix chaindag logscope
* add database reload metric
* re-enable clearance epoch tests

* names
2020-08-13 11:50:05 +02:00
tersec ab34584f23
initial dynamic subscribe/unsubscribe for attestations to/from subnets (#1462)
* initial dynamic subscribe/unsubscribe for attestations to/from subnets

* implement random stability subnet and clean up

* switch from HashSet[uint64] to set[uint8]

* refactor subnet logic out from beacon_node and actual (un)subscribing

* only try to subscribe to marginally different subnets

* add assertions

* maintain ENR subnets

* assert that beacon_node and eth2_network have consistent view of subscribed subnets

* disable actual cycling
2020-08-12 17:48:31 +00:00
tersec af3355e0f8
create local testnet mode for eth2_network (#1494) 2020-08-12 14:16:59 +00:00
Eugene Kabanov 711f1f88ee
Use one single async queue and loop for processing blocks. (#1487)
* Initial commit

* Fix compilation problem.

* Address review comments.
2020-08-12 11:29:11 +02:00
Jacek Sieka 5da25e76be
avoid rewind in fork choice application (#1489) 2020-08-12 04:49:52 +00:00
Jacek Sieka 8b0f2cc96f
share validator keys in EpochRef (#1486) 2020-08-11 21:39:53 +02:00
tersec 22c1ef5a8d
split subscribe into non-validating subscribe and addValidator (#1485)
* split subscribe into non-validating subscribe and addValidator

* stop exporting get_committee_assignments
2020-08-11 15:08:44 +00:00
Zahary Karadjov 224ebdfd72 A simple metric for measuring the delay in the onSecond timer 2020-08-10 23:53:55 +03:00
Zahary Karadjov 30a8ec410d More spec compliant blocksByRange requests
* Eliminate possibilities for range errors and overflows
* Handle more properly invalid requests for furute slots
* Eliminate the confusing surrounding the MAX_REQUEST_BLOCKS constant

Addresses https://github.com/status-im/nim-beacon-chain/issues/1366
2020-08-10 22:09:13 +03:00
Ștefan Talpalaru 7763df95a4
storeBlock() duration metric (#1480) 2020-08-10 19:10:43 +02:00
Jacek Sieka 2b4526e743
bls: avoid exception flow on cache miss (#1479) 2020-08-10 14:51:23 +00:00
tersec fe1a7922c8
update attestation aggregation validation to spec refs of v0.12.2 (#1481) 2020-08-10 14:49:18 +00:00
Jacek Sieka 10da7fe9da remove eth from default status bar
not viable at higher validator counts - linear scan + forced public key
init makes it extremely slow
2020-08-10 17:01:53 +03:00
Jacek Sieka 2a36949913
use epochcache for attesting (#1478) 2020-08-10 15:21:31 +02:00
Jacek Sieka 280e72f3c9
remove snappy RPC support (#1477)
removed in 0.12.2 - the flow, in particular when the other peer doesn't
support snappy, is hard to follow because of the trial-and-error
approach - removing it simplifies things and removes some of the
hard-to-read parts of the thunking etc
2020-08-10 15:18:17 +02:00
Jacek Sieka 936440fccd
use libp2p peer events to track peer (#1468)
this resolves some peer counting issues that were happening because the
lifetime future in PeerInfo was unreliable (multiple PeerInfo instances
existed per peer)

In addition, this solves another race condition: when connecting to a
peer and later dialling that protocol, it is not certain that the same
connection will be used if there's a concurrent incoming peer connection
ongoing - better not make too many assumptions about who sent statuses
when.
2020-08-10 12:58:34 +02:00
Jacek Sieka 585c410d90 remove randompeers
unused, requires importing `random` which we're trying to avoid
2020-08-10 11:48:33 +03:00
Eugene Kabanov 55fcece0b2
SyncManager fix to process blocks one by one. (#1464)
* Allow sync manager process blocks one by one.

* Log storeBlock() and updateHead() duration.

* Calculate duration only for blocks added without any error.

* Fix float compilation error.

* Fix duration.

* Fix SyncQueue tests.
2020-08-10 09:15:50 +02:00
Jacek Sieka 3b6a8a692d
cleanup unused chaindag epoch features
these are somewhat obsoleted by the more extensive use of EpochRef
2020-08-07 19:49:52 +02:00
Eugene Kabanov 38bf8ccbec
Implement tracing of lags in the logs. (#1465) 2020-08-07 16:22:58 +00:00
Jacek Sieka 84a501d1ff
remove one cache, add another (#1449)
* remove one cache, add another

This cache removes the need for rewinding in most attestation validation
flow since the attestations come from one of two epochs and must be
targetting a viable block.

Additionally, it also removes all state caches which are less likely to
be used over-all - more metrics are needed to track the rewinding.

On risk is that when chains don't finalize, we'll have lots of epochrefs
in memory meaning lots of validator key databases, most being exactly
the same. This can be addressed in any number of ways. Some of the
memory usage is mitigated by the fact that we previously had lots of big
state caches and now we're keeping only keys instead.

* cleanups

* doc
2020-08-06 19:48:47 +00:00
Dmitriy Ryajov c5077af4bc
decreate amount of concurent dials (#1460) 2020-08-06 19:21:12 +00:00
Zahary Karadjov 9861eb1152
Use the same keystore directory names as Lighthouse
Rationale: this makes moving keys between the clients eaiser

Other changes:

* Restore building with custom presets
  (defaultRuntimePreset is not a template in this mode)
2020-08-06 21:50:19 +03:00
Jacek Sieka f4c16ed0db
eh cleanups (#1458)
current exception sometimes buggy in nim
2020-08-06 18:47:39 +00:00
Zahary Karadjov b902fddd19 Allow loading keystores produced by Lighthouse
The spec allows the description to be set to 'null'
2020-08-06 17:33:57 +03:00
tersec 81b3c0ea40
update spec refs to v0.12.2 (#1457)
* update spec refs to v0.12.2 and change a .len.uint64 to .lenu64

* pull back from any non-pure-comment changes, since Jenkins is being wonky
2020-08-06 13:05:13 +00:00
Viktor Kirilov 5bbeb38f2d fixes the BN/VC communication - properly getting the attestation duties & also fixed start.sh 2020-08-06 15:29:05 +03:00
Zahary Karadjov 009e9a6a41 Fix 'make test' 2020-08-06 14:49:58 +03:00
protolambda e90c5440e8 make eth1 distance runtime configurable 2020-08-06 14:49:58 +03:00
Jacek Sieka 221f372dbc use peer id in a number of places 2020-08-05 19:34:59 +03:00
Zahary Karadjov b427c7249f More logging and metrics (incoming gossip blocks; outgoing aggregated attestations) 2020-08-05 19:28:35 +03:00
Jacek Sieka deaeb62de3
clean up quarantine 2020-08-05 16:19:55 +02:00
Viktor Kirilov 983d931a83 initializing the pubkeys properly in the rest of the places for the BN/VC communication to work 2020-08-05 16:37:55 +03:00
Viktor Kirilov 154c6bf8ee fix for the BN/VC communication - related to the lazy initialization of validator public keys 2020-08-05 16:37:55 +03:00
Jacek Sieka 15b99e4c11
cache beacon proposer indices (#1440)
also clear old epochrefs as they're growing unwieldy

in particular, this speeds up gossip block validation by avoiding the
rewind
2020-08-05 08:28:43 +02:00
tersec dd0a91de0c
allow a slot's worth of clock disparity in attestation validation (#1442) 2020-08-04 20:10:29 +00:00
Dustin Brody 8618cbdf38 keep representations consistent for tests 2020-08-04 23:00:33 +03:00
Dustin Brody e8c31e4ce8 just in case 2020-08-04 23:00:33 +03:00
Dustin Brody 5e3af9a458 fix validator cache keys 2020-08-04 23:00:33 +03:00
Dustin Brody 2c7d207a72 make test passes 2020-08-04 23:00:33 +03:00
Dustin Brody 9c65c50249 use case statement to avoid runtime doAssert 2020-08-04 23:00:33 +03:00
Dustin Brody c142de4b7f be more consistent about pubkeys fed to verify_foo_signature() not being separately initialized, while pubkeys, generally, used for matching purposes, elsewhere explicitly initialized 2020-08-04 23:00:33 +03:00
Dustin Brody db397cc5f3 lazily initialize validator public keys, which were a third or more of sync CPU usage 2020-08-04 23:00:33 +03:00
Jacek Sieka ac78e75bf8
lear missing on orphan add in quarantine (#1441) 2020-08-04 19:49:25 +00:00
Jacek Sieka 70df0ad057 don't mark quarantined blocks as missing 2020-08-04 22:37:06 +03:00
Jacek Sieka 48aca60e81
add target block to quarantine when missing (#1438) 2020-08-04 19:00:55 +00:00
tersec a979568d74
switch mapIt/find to in-place linear search (#1436) 2020-08-04 17:15:27 +00:00
Jacek Sieka cf8cd8321b
Revert "Lazy crypto [alt design #1369] (#1371)" (#1435)
This reverts commit 023f7f4518.
2020-08-04 17:15:13 +00:00
Jacek Sieka c6674de5d2 use epoch ref to update fork choice
this dramatically speeds up startup in long periods of non-finality
2020-08-04 20:00:31 +03:00
Jacek Sieka 196bc6ea3b
remove redundant attestation check (#1433)
only viable blocks (descendants of the finalized checkpoint) are in the
dag already
2020-08-04 15:52:46 +00:00
cheatfate b70462acf8 Fix request manager is not penalizing peers which are failed to deliver blocks. 2020-08-04 18:14:51 +03:00
Zahary Karadjov 4deed117a9
Hotfix: Allow 'make madella' to work with the latest Medalla metadata 2020-08-04 17:55:37 +03:00
Ștefan Talpalaru b4e434849b
makeBannerAndConfig(): disable ProveInit warning 2020-08-04 15:00:55 +02:00
Ștefan Talpalaru a1d3532f49
move stateSnapshotContents from BeaconNodeConf (#1425) 2020-08-04 12:35:20 +02:00
Zahary Karadjov d45ee66c21
Remove a confusing warning on Keystore import 2020-08-04 12:17:06 +03:00
Viktor Kirilov d7d1e072ba
fix for the BN/VC communication since the #1371 PR (#1416) 2020-08-04 08:15:26 +00:00
tersec df80071bcf
update attestation and block validation to v0.12.2; clean up getAncestorAt()/get_ancestor() (#1417)
* update attestation validation to v0.12.2; clean up getAncestorAt()/get_ancestor()

* update beacon block validation to v0.12.2
2020-08-03 19:47:42 +00:00
Jacek Sieka d22a2cec2b
Start libp2p before writing ENR file (#1418)
this makes sure that all libp2p transports are open for business when
the file hits the ground
2020-08-03 19:35:27 +02:00
Dmitriy Ryajov 52d9d269d7
bump libp2p (delayed send, without hooks) (#1413)
* use `switch.isConnected`

* libp2p

* add timeout to publish

* use isConnected

* adjust timeouts

* latest libp2p master

* do not drop peers
2020-08-03 16:43:22 +00:00
tersec 5b13d8f3fa
synchronize test summaries with make test (and do a few small cleanups) (#1415) 2020-08-03 12:07:25 +00:00
tersec 1c6936a35c
add attestations received before referred-to blocks to unresolved list (#1410) 2020-08-03 10:49:15 +00:00
tersec f2dd57db1e
increase allowed missing validators in verifyFinalization mode (#1414) 2020-08-03 05:31:35 +00:00
Zahary Karadjov 8c2a9a4950
Switch back to the old undepleted Infura key; Bake in the Medalla genesis state 2020-08-03 03:59:45 +03:00
Zahary Karadjov d052623381
Fix a trivial issue with 'make medalla' and add more logging 2020-08-03 00:19:25 +03:00
Zahary Karadjov be2c2e27f3 Cosmetic fixes 2020-08-02 23:00:43 +03:00
Zahary Karadjov c882b7c2f3 Add Scrypt support in the Keystores 2020-08-02 23:00:43 +03:00
Zahary Karadjov 1aba7aed6d Updated Keystore test vectors 2020-08-02 23:00:43 +03:00
Zahary Karadjov c293254ded Add 'deposits import' command; Switch to NJS when loading the keystores and improve the data validation 2020-08-02 23:00:43 +03:00
Zahary Karadjov d58668157a Use consistent naming for the synonyms 'deposits-dir' and 'validators-dir' 2020-08-02 23:00:43 +03:00
Zahary Karadjov 2a94f6eca6 medalla-deposit target capable of executing mass deposits 2020-08-02 23:00:43 +03:00
tersec 85febf072e
assert on missing active validators' attestations when verifying finalization (#1387)
* assert on missing > 10 supposed-to-be-active validators' attestations in finalization verification mode

* clean up state_sim imports
2020-08-01 18:24:25 +00:00
Viktor Kirilov 0a96e5f564
renamed CandidateChains to ChainDagRef and made the Quarantine type a ref type so there is a single instance in the beacon node (#1407) 2020-07-31 14:49:06 +00:00
tersec e0a6f58abe
convert 10 v0.12.1 spec refs to v0.12.2 (#1406) 2020-07-31 09:59:14 +00:00
Eugene Kabanov 65fa1c1124
Ignoring json serialization errors. (#1403)
Add ability to use logs from jenkins.
2020-07-31 09:43:11 +00:00
Viktor Kirilov c032366547
removed the BlockPool type and all of the proxy functions around it (#1401)
* removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type

* fixed the build after the rebase
2020-07-30 21:18:17 +02:00
Jacek Sieka c5fecd472f
more fork-choice fixes (#1388)
* more fork-choice fixes

* use target block/epoch to validate attestations
* make addLocalValidators sync
* add current and previous epoch to cache before doing state transition
* update head state using clearance state as a shortcut, when possible
* use blockslot for fork choice balances
* send attestations using epochref cache

* fix invalid finalized parent being used

also simplify epoch block traversal

* single error handling style in fork choice

* import fix, remove unused async
2020-07-30 17:48:25 +02:00
tersec 55607f107f
add metric to track finalization delay (#1399) 2020-07-30 16:27:34 +02:00
tersec 254a510c0d
mark .borrow. procs as noSideEffect (#1402) 2020-07-30 14:47:59 +02:00
tersec 99f2d8e06c
update 14 v0.12.1 spec refs to v0.12.2 (#1400) 2020-07-30 09:47:57 +00:00
Mamy Ratsimbazafy 023f7f4518
Lazy crypto [alt design #1369] (#1371)
* Lazy loading of crypto objects

* Try to fix incorrect field access by hiding fields but no luck. SSZ/Chronicles/macro bug?

* Fix incorrect blsValue access. was "aggregate" not "chronicles"

* Fix tests that rely on the internal BLSValue representation
2020-07-29 18:13:05 +00:00
Eugene Kabanov 371ea7d99b
Implement log tracing of attestation send and receive messages (attempt 2) (#1394)
* Implement log tracing of attestation send and receive messages.

* Cosmetic changes.

* Fix for latest changes in nim-json-serialization
2020-07-29 17:20:20 +02:00
tersec d97cc35d30
switch 14 v0.12.1 spec refs to v0.12.2 spec refs (#1395) 2020-07-29 12:47:03 +00:00
tersec 1220cc05ad
Revert "Implement log tracing of attestation send and receive messages. (#1361)" (#1389)
This reverts commit ac12af16bf.
2020-07-28 17:45:31 +00:00
tersec a07dab300c
verifyFinalization checks for justification being consistent with later finalization (#1386) 2020-07-28 17:35:32 +00:00
Viktor Kirilov 0ffcbd10e5 All logs should be stored in appropriate node's folder - related to #1349 2020-07-28 20:14:17 +03:00
Eugene Kabanov ac12af16bf
Implement log tracing of attestation send and receive messages. (#1361)
* Implement log tracing of attestation send and receive messages.

* Cosmetic changes.
2020-07-28 17:06:03 +00:00
Zahary Karadjov b801dd65b9 Makefile targets for Medalla 2020-07-28 19:27:45 +03:00
Zahary Karadjov 0c60e452e7 Allow the deposit contract deployment block to be specified as a number (needed for Medalla) 2020-07-28 18:13:01 +03:00
tersec b45de824a4
convert 22 v0.12.1 spec refs to v0.12.2 and remove 1 unnecessary spec ref (#1384) 2020-07-28 14:55:46 +00:00
Jacek Sieka 157ddd2ac4
Fork choice fixes 5 (#1381)
* limit attestations kept in attestation pool

With fork choice updated, the attestation pool only needs to keep track
of attestations that will eventually end up in blocks - we can thus
limit the horizon of attestations that we keep more aggressively.

To get here, we expose getEpochRef which gets metadata about a
particular epochref, and make sure to populate it when a block is added
- this ensures that state rewinds during block addition are minimized.

In addition, we'll use the target root/epoch when validating
attestations - this helps minimize the number of different states that
we need to rewind to, in general.

* remove CandidateChains.justifiedState

unused

* remove BlockPools.Head object

* avoid quadratic quarantine loop

* fix
2020-07-28 13:54:32 +00:00
Zahary Karadjov 6ccfff7d58
Allow network metadata without bootstrap nodes 2020-07-28 13:46:22 +03:00