76 Commits

Author	SHA1	Message	Date
tersec	da017d2ca5	update from phase0/altair v1.1.6 URLs to v1.1.8 spec URLs (#3238)	2022-01-04 03:57:15 +00:00
Jacek Sieka	6b60a774e0	Lazy aggregated batch verification (#3212) ... A novel optimisation for attestation and sync committee message validation: when batching, we look for signatures of the same message and aggregate these before batch-validating: this results in up to 60% fewer signature verifications on a busy server, leading to a significant reduction in CPU usage. * increase batch size slightly which helps finding more aggregates * add metrics for batch verification efficiency * use simple `blsVerify` when there is only one signature to verify in the batch, avoiding the RNG	2021-12-29 15:28:40 +01:00
Jacek Sieka	069bccd51b	batch-verify sync messages for a small perf boost (#3151) ... * batch-verify sync messages for a small perf boost Generally reuses the same structure as attestation and aggregate verification * normalize `signatures` and `signature_batch` to use the same pattern of verification * normalize parameter names, order etc for signature stuff in general * avoid calling `blsSign` directly - instead, go through `signatures` consistently	2021-12-09 14:56:54 +02:00
Jacek Sieka	1a8b7469e3	move quarantine outside of chaindag (#3124) ... * move quarantine outside of chaindag The quarantine has been part of the ChainDAG for the longest time, but this design has a few issues: * the function in which blocks are verified and added to the dag becomes reentrant and therefore difficult to reason about - we're currently using a stateful flag to work around it * quarantined blocks bypass the processing queue leading to a processing stampede * the quarantine flow is unsuitable for orphaned attestations - these should also should be quarantined eventually Instead of processing the quarantine inside ChainDAG, this PR moves re-queueing to `block_processor` which already is responsible for dealing with follow-up work when a block is added to the dag This sets the stage for keeping attestations in the quarantine as well. Also: * make `BlockError` `{.pure.}` * avoid use of `ValidationResult` in block clearance (that's for gossip)	2021-12-06 10:49:01 +01:00
tersec	cc51f3fd12	v1.1.{5 -> 6} phase 0 and altair spec URL updates (#3157)	2021-12-03 17:40:23 +00:00
Jacek Sieka	aa1dea03cd	speed up gossip and sync block validation (#3143) ... * avoid recomputing hash for block signature check * check block slot match before hitting the database	2021-12-01 10:52:40 +01:00
Jacek Sieka	9c2f43ed0e	Speed up altair block processing 2x (#3115) ... * Speed up altair block processing >2x Like #3089, this PR drastially speeds up historical REST queries and other long state replays. * cache sync committee validator indices * use ~80mb less memory for validator pubkey mappings * batch-verify sync aggregate signature (fixes #2985) * document sync committee hack with head block vs sync message block * add batch signature verification failure tests Before: ``` ../env.sh nim c -d:release -r ncli_db --db:mainnet_0/db bench --start-slot:-1000 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 5830.675, 0.000, 5830.675, 5830.675, 1, Initialize DB 0.481, 1.878, 0.215, 59.167, 981, Load block from database 8422.566, 0.000, 8422.566, 8422.566, 1, Load state from database 6.996, 1.678, 0.042, 14.385, 969, Advance slot, non-epoch 93.217, 8.318, 84.192, 122.209, 32, Advance slot, epoch 20.513, 23.665, 11.510, 201.561, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` After: ``` 7081.422, 0.000, 7081.422, 7081.422, 1, Initialize DB 0.553, 2.122, 0.175, 66.692, 981, Load block from database 5439.446, 0.000, 5439.446, 5439.446, 1, Load state from database 6.829, 1.575, 0.043, 12.156, 969, Advance slot, non-epoch 94.716, 2.749, 88.395, 100.026, 32, Advance slot, epoch 11.636, 23.766, 4.889, 205.250, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * add comment	2021-11-24 13:43:50 +01:00
tersec	9e395011d9	update 22 spec URLs to v1.1.5 (#3111)	2021-11-18 08:08:00 +00:00
Jacek Sieka	a086cf01ac	altair fork handling cleanups (#3050) ... * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)	2021-11-05 08:34:34 +01:00
tersec	2b2846b468	implement forked merge state/block support (#2890) ... * implement forked state/block support * merge support for containsOrphan; import cleanup; 80-column lines * add merge block header operations and slot sanity fixture * add epoch state transition tests; implement is_valid_gas_limit(), is_merge_block(), is_execution_enabled(), and compute_timestamp_at_slot() * implement process_execution_payload() and add merge deposit operations tests * add merge block sanity tests * add merge case to syncCommitteeParticipants * v1.1.0-beta.5 updates * reduce getTestStates-based memory usage; don't try to REST-serialize ExecutionPayload transactions without underlying support * add execution payload tests; switch var to let in tests/official/	2021-09-27 14:22:58 +00:00
tersec	9725d15a3e	update spec references from eth2.0-specs to consensus-specs and to v1.1.0-beta.2 (#2822)	2021-08-26 10:21:52 +02:00
Jacek Sieka	a7a65bce42	disentangle eth2 types from the ssz library (#2785) ... * reorganize ssz dependencies This PR continues the work in https://github.com/status-im/nimbus-eth2/pull/2646, https://github.com/status-im/nimbus-eth2/pull/2779 as well as past issues with serialization and type, to disentangle SSZ from eth2 and at the same time simplify imports and exports with a structured approach. The principal idea here is that when a library wants to introduce SSZ support, they do so via 3 files: * `ssz_codecs` which imports and reexports `codecs` - this covers the basic byte conversions and ensures no overloads get lost * `xxx_merkleization` imports and exports `merkleization` to specialize and get access to `hash_tree_root` and friends * `xxx_ssz_serialization` imports and exports `ssz_serialization` to specialize ssz for a specific library Those that need to interact with SSZ always import the `xxx_` versions of the modules and never `ssz` itself so as to keep imports simple and safe. This is similar to how the REST / JSON-RPC serializers are structured in that someone wanting to serialize spec types to REST-JSON will import `eth2_rest_serialization` and nothing else. * split up ssz into a core library that is independendent of eth2 types * rename `bytes_reader` to `codec` to highlight that it contains coding and decoding of bytes and native ssz types * remove tricky List init overload that causes compile issues * get rid of top-level ssz import * reenable merkleization tests * move some "standard" json serializers to spec * remove `ValidatorIndex` serialization for now * remove test_ssz_merkleization * add tests for over/underlong byte sequences * fix broken seq[byte] test - seq[byte] is not an SSZ type There are a few things this PR doesn't solve: * like #2646 this PR is weak on how to handle root and other dontSerialize fields that "sometimes" should be computed - the same problem appears in REST / JSON-RPC etc * Fix a build problem on macOS * Another way to fix the macOS builds Co-authored-by: Zahary Karadjov <zahary@gmail.com>	2021-08-18 20:57:58 +02:00
Jacek Sieka	7a622e8505	rework spec imports (#2779) ... The spec imports are a mess to work with, so this branch cleans them up a bit to ensure that we avoid generic sandwitches and that importing stuff generally becomes easier. * reexport crypto/digest/presets because these are part of the public symbol set of the rest of the spec types * don't export `merge` types from `base` - this causes circular deps * fix circular deps in `ssz/spec_types` - this is the first step in disentangling ssz from spec * be explicit about phase0 vs altair - longer term, `altair` will become the "natural" type set, then merge and so on, so no point in giving `phase0` special preferential treatment	2021-08-12 13:08:20 +00:00
Jacek Sieka	9697b73e71	forkedbeaconstate_helpers -> forks (#2772) ... Simpler module name for stuff that covers forks * check that runtime config matches database state * also include some assorted altair cleanups * use "standard" genesis fork in local testnet to work around missing runtime config support	2021-08-10 22:46:35 +02:00
tersec	b1d5609171	remove false OnBlockAdded dependency on phase0 HashedBeaconState (#2661) ... * remove false OnBlockAdded dependency on phase.HashedBeaconState * introduce altair data types into block_clearance; update some alpha.6 spec refs to alpha.7; add get_active_validator_indices_len ForkedHashedBeaconState wrapper * switch many modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 and remove rm'd G2_POINT_AT_INFINITY * switch more modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 * remove unnecessary phase0-only wrapper of get_attesting_indices(); allow signatures_batch to process either fork; remove O(n^2) nested loop in process_inactivity_updates(); add altair support to getAttestationsforTestBlock() * add Altair versions of asSigVerified(), asTrusted(), and makeBeaconBlock() * fix spec URL to be Altair for Altair makeBeaconBlock()	2021-06-21 08:35:24 +00:00
tersec	146fa48454	use ForkedHashedBeaconState in StateData (#2634) ... * use ForkedHashedBeaconState in StateData * fix FAR_FUTURE_EPOCH -> slot overflow; almost always use assign() * avoid stack allocation in maybeUpgradeStateToAltair() * create and use dispatch functions for check_attester_slashing(), check_proposer_slashing(), and check_voluntary_exit() * use getStateRoot() instead of various state.data.hbsPhase0.root * remove withStateVars.hashedState(), which doesn't work as a design anymore * introduce spec/datatypes/altair into beacon_chain_db * fix inefficient codegen for getStateField(largeStateField) * state_transition_slots() doesn't either need/use blocks or runtime presets * combine process_slots(HBS)/state_transition_slots(HBS) which differ only in last-slot htr optimization * getStateField(StateData, ...) was replaced by getStateField(ForkedHashedBeaconState, ...) * fix rollback * switch some state_transition(), process_slots, makeTestBlocks(), etc to use ForkedHashedBeaconState * remove state_transition(phase0.HashedBeaconState) * remove process_slots(phase0.HashedBeaconState) * remove state_transition_block(phase0.HashedBeaconState) * remove unused callWithBS(); separate case expression from if statement * switch back from nested-ref-object construction to (ref Foo)(Bar())	2021-06-11 20:51:46 +03:00
Jacek Sieka	d859bc12f0	write uncompressed validator keys to database (#2639) ... * write uncompressed validator keys to database Loading 150k+ validator keys on startup in compressed format takes a lot of time - better store them in uncompressed format which makes behaviour just after startup faster / more predictable. * refactor cached validator key access * fix isomorphic cast to work with non-var instances * remove cooked pubkey cache - directly use database cache in chaindag as well (one less cache to keep in sync) * bump blscurve, introduce loadValid for known-to-be-valid keys	2021-06-10 10:37:02 +03:00
Jacek Sieka	abe0d7b4ae	singe validator key cache ... Instead of keeping a validator key list per EpochRef, this PR introduces a single shared validator key list in ChainDAG, and cleans up some other ChainDAG and key-related issues. The PR does not introduce the validator key list in the state transition - this is because we batch-check all signatures before entering the spec code, thus the spec code never hits the cache. A future refactor should _probably_ remove the threadvar altogether. There's a few other small fixes in here that make the flow easier to read: * fix `var ChainDAGRef` -> `ChainDAGRef` * fix `var QuarantineRef` -> `QuarantineRef` * consistent `dag` variable name * avoid using threadvar pubkey cache in most cases * better error messages in batch signature checking	2021-06-01 20:43:44 +03:00
tersec	c06ffc7804	proposed structure for altair (#2323) ... * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader	2021-05-28 15:25:58 +00:00
tersec	0b0bfd1de0	use StateData in place of BeaconState outside state transition code (#2551) ... * use StateData in place of BeaconState outside state transition code * propagate more StateData usage * remove withStateVars().state * wrap get_beacon_committee(BeaconState, ...) as gbc(StateData, ...) * switch makeAttestation() to use StateData * use StateData wrapper/dispatcher for get_committee_count_per_slot() * convert AttestationCache.init(), weak subjectivity functions, and updateValidatorMetrics() * add get_shuffled_active_validator_indices(StateData) and get_block_root_at_slot(StateData) * switch makeAttestationData() to StateData * sync AllTests-mainnet.md after rebase	2021-05-21 09:23:28 +00:00
Jacek Sieka	7dba1b37dd	remove attestation/aggregate queue (#2519) ... With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message	2021-04-26 22:39:44 +02:00
tersec	79bb0d5379	only deserialize attestation and aggregation gossiped signatures once (#2472) ... * only deserialize attestation and aggregation gossiped signatures once * re-indent some aggregate checks into block scope * spelling * remove debugging assertion * put part of gossip validation back into block context * attestation pool test signature loading isn't so unsafe, and exportRaw isn't free * remove more development doAsserts; don't exportRaw in loops	2021-04-09 14:59:24 +02:00
Mamy Ratsimbazafy	6b13cdce36	Batch attestations (#2439) ... * batch attestations * Fixes (but now need to investigate the chronos 0 .. 4095 crash similar to https://github.com/status-im/nimbus-eth2/issues/1518 * Try to remove the processing loop to no avail :/ * batch aggregates * use resultsBuffer size for triggering deadline schedule * pass attestation pool tests * Introduce async gossip validators. May fix the 4096 bug (reentrancy issue?) (similar to sync unknown blocks #1518) * Put logging at debug level, add speed info * remove unnecessary batch info when it is known to be one * downgrade some logs to trace level * better comments [skip ci] * Address most review comments * only use ref for async proc * fix exceptions in eth2_network * update async exceptions in gossip_validation * eth2_network 2nd pass * change to sleepAsync * Update beacon_chain/gossip_processing/batch_validation.nim Co-authored-by: Jacek Sieka <jacek@status.im> Co-authored-by: Jacek Sieka <jacek@status.im>	2021-04-02 16:36:43 +02:00
Mamy Ratsimbazafy	c47d636cb3	Split Eth2Processor in prep for batching (#2396) ... * Split Eth2Processor in gossip and consensus part and materialize the shared block queue * Update initialization in test_sync_manager	2021-03-11 11:10:57 +01:00
tersec	97f7284e51	bump spec refs from v1.0.0 to v1.0.1 and update copyright years (#2357)	2021-02-25 13:37:22 +00:00
Mamy Ratsimbazafy	70a03658e3	Block validation flow v2 + Batch (serial) sig verification (#2250) ... * bump nim-blscurve * Outline the block validation flow * introduce the SigVerified types, pass the tests * Split clearance/quarantine to prepare for batch crypto verif * Add a batch signature collector * Make clearance use SigVerified block and split verification between crypto and state transition * Always use signedBeaconBlock for the onBlockAdded callback * RANDAO signing_root is the epoch instead of the full block * Support skipping BLS for testing * Fix compilation of the validator client * Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c) * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561819858 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828025 * onBlockAdded callback should use TrustedSignedBeaconBlock https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561837261 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828946 * Use the application RNG: https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561815336 * Improve codegen of conversion zero-cost) * Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch) * Graceful handle rogue pubkeys and signatures now that those are lazy-loaded	2021-01-25 20:45:48 +02:00