Commit Graph

100 Commits

Author SHA1 Message Date
Jacek Sieka 1301600341
Trusted blocks (#1227)
* cleanups

* fix ncli state root check flag
* add block dump to ncli_db
* limit ncli_db benchmark length
* tone down finalization logs

* introduce trusted blocks

We only store blocks whose signature we've verified in the database - as
such, there's no need to check it again, and most importantly, no need
to deserialize the signature when loading from database.

50x startup time improvement, 200x block load time improvement.

* fix rewinding when deposits have invalid signature
* speed up ancestor iteration by avoiding copy
* avoid deserializing signatures for trusted data
* load blocks lazily when rewinding (less memory used)

* chronicles workarounds

* document trustedbeaconblock
2020-06-25 12:23:10 +02:00
Dustin Brody db870fead4 update 20 beacon chain protocol spec refs 2020-06-18 08:39:28 +00:00
tersec ee9f4a2e3f
remove skipMerkleValidation and skipBlockParentRootValidation (#1197) 2020-06-18 07:56:47 +02:00
Dustin Brody ffca27b45f update 24 v0.11.x spec refs to v0.12.1 2020-06-17 12:11:03 +00:00
Dustin Brody ec85a8bdbf update a dozen unchanged spec refs to v0.12.1 2020-06-17 09:25:54 +00:00
Jacek Sieka 5c25d23ef1 eth2hash -> eth2digest
hash in nim is the insecure hash-map helper - might as well use `digest`
consistently to mark the difference
2020-06-16 13:30:21 +00:00
Jacek Sieka 89e4819ce9
collect signature production and verificaiton in one place (#1179)
* collect signature production and verificaiton in one place

Signatures are made over data and domain - here we collect all such
activities in one place.

Also:
* security: fix cast-before-range-check
* log block/attestation verification consistently
* run block verification based on `getProposer` in its own history
* clean up some unused stuff

* import

* missing raises
2020-06-16 07:45:04 +02:00
Dustin Brody 8a2c796e83 switch 12 unchanged spec references from 0.11.x to 0.12.1; line-wrap and remove pointless-to-counterproductive "return" 2020-06-15 10:10:31 +00:00
Dustin Brody 1d1e90adf4 Update 36 spec refs from 0.11.x to 0.12.1 2020-06-12 19:41:03 +00:00
Dustin Brody 8f7541959a allow compile-time switching between v0.11.3 and v0.12.1 2020-06-09 18:39:05 +00:00
Dustin Brody cc13153fc6 initial 0.12.1 beacon chain spec 2020-06-09 18:39:05 +00:00
Jacek Sieka 56ffb696be
reorder ssz (#1099)
* reorder ssz

* split into hash_trees and ssz_serialization, roughly, for hashing and
IO
* move bitseqs into ssz (from stew)
* clean up imports

* docs, imports
2020-06-03 15:52:02 +02:00
Jacek Sieka 07393c8de6
Ssz cache (#1084)
* wip: cache

* cache lists and arrays of complex objects (5x block processing speed
on ncli_db)

trivial baseline cache that stores tree in flat memory structure

* support array of uint64

* work around type issues

* more type compiler bug workarounds

* cache balances, more type fixes

* index type

* ncli_db: add validation flag, better ux

* int64 fixes

* test fix

* "oops"

```
647.913,        0.000,      647.913,      647.913,            1,
Initialize DB
0.540,        0.402,        0.340,        9.451,          619,
Load block from database
40.268,        0.000,       40.268,       40.268,            1,
Load state from database
0.498,        0.150,        0.343,        0.930,          596,
Apply block
3.548,       11.005,        0.729,       54.022,           23,
Apply epoch block
```

* support all basic types

* cleanups

* a few more cleanups
2020-06-03 07:57:17 +02:00
tersec b7bb62d48b
use shuffled validator cache in more places; cleanups (#1095) 2020-06-03 07:42:08 +02:00
Jacek Sieka f1dcee2749
wip: cache
* cache lists and arrays of complex objects (5x block processing speed
on ncli_db)

trivial baseline cache that stores tree in flat memory structure
2020-06-01 14:52:50 +02:00
Dustin Brody 0929d90d93 unexport candidate_chains.init; some spec version bumps 2020-05-26 05:06:37 +00:00
Dustin Brody 74e35c464f halt processing if one of the operation functions fails (failed deposits due to BLS signature issues return as not-failed, so it remains consistant with that 2020-05-24 08:20:13 +00:00
Dustin Brody 67ba11cc1a use template to more cleanly implement for_ops 2020-05-24 08:20:13 +00:00
Dustin Brody 95ee0218db switch too-strict/against-spec operations processing to use spec-based process_operations 2020-05-24 08:20:13 +00:00
Jacek Sieka f06df1cea6 remove some copies
* in makeBeaconBlock - use rollback instead
* in tests - this helps state_sim give more accurate data and makes it
30% faster
* fix some usages of raw BeaconState
2020-05-22 17:15:35 +00:00
Dustin Brody 3fa2f70853 update all remaining beacon chain spec references from v0.11.1 and v0.11.2 to v0.11.3 2020-05-20 08:48:18 +00:00
Dustin Brody 6c941b1ff7 spec v0.11.3 update 2020-05-19 17:16:55 +00:00
Zahary Karadjov a99977b772 Progress towards #991; Enable the distinct ssz.List type 2020-05-19 15:53:03 +03:00
Ștefan Talpalaru c572f61129
bump vendor/nim-metrics 2020-05-09 01:13:57 +02:00
Jacek Sieka c74ba5c0c6
ssz: move ref support outside (#943)
* ssz: move ref support outside

Instead of allocating ref's inside SSZ, move it to separate helper:

* makes `ref` allocations explicit
* less magic inside SSZ
* `ref` in nim generally means reference whereas SSZ was loading as
value - if a type indeed used references it would get copies instead of
references to a single value on roundtrip which is unexpected

TODO: EF tests would benefit from some refactoring since they all do the
same thing practically..

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2020-04-29 22:12:07 +02:00
Jacek Sieka 28d6cd2524
avoid memory allocations and copies when loading states (#942)
* rolls back some of the ref changes
* adds utility to calculate stack sizes
* works around bugs in nim exception handling and rvo
2020-04-28 10:08:32 +02:00
Zahary Karadjov 4e9fa51ae9 Introduce BeaconNodeRef and use it in all the right places 2020-04-26 13:04:53 +03:00
Zahary Karadjov fdcbfdff05 Pass the test suite with a BeaconState ref type 2020-04-26 13:04:53 +03:00
Zahary Karadjov 740b76d152 Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim
BEWARE! This commit will trigger a stack overflow during local sim
2020-04-26 13:04:53 +03:00
tersec 65353bab76
add another check for inconsistent aggregation and committee length (#927)
* add another check for inconsistent aggregation and committee length, since ncli_transition bypasses process_attestation(...)/check_attestation(...) and calls almost directly into process_epoch(...)

* bump validator functions to v0.11.1 spec references

* bump some spec references to v0.11.1

* poke
2020-04-24 09:16:40 +02:00
Jacek Sieka ed74770451
spec: regulate exceptions (#913)
* spec: regulate exceptions

* a few more simple raises
2020-04-22 07:53:02 +02:00
tersec 480c11f8aa
fix remaining block sanity test vectors (#902)
* fix remaining block sanity test vectors

* bump epoch processing spec refs to v0.11.1
2020-04-18 00:09:34 +02:00
Dustin Brody 164d171bd9
revert previous commit 2020-04-17 20:27:42 +02:00
Dustin Brody ef92529feb
fix remaining block sanity test vectors 2020-04-17 20:25:18 +02:00
Dustin Brody 3df9a22ff7 fix remaining attester slashing tests from EF phase0 spec tests by checking for uniqueness 2020-04-17 16:01:32 +00:00
Zahary Karadjov 672f690bf6 Implement most of the v0.11.1 networking changes 2020-04-16 01:01:18 +03:00
Jacek Sieka afa08c8e3c
crypto: cleanup (#882)
* crypto: cleanup

* fix several Defect-on-user-input
* make crypto interface more similar to secp
* use `crypto.nim` in all of nbc

* digest: raises

* fix

* vendor
2020-04-11 10:51:07 +02:00
tersec 31bf8fa408
remove all but one UnusedImport warning (#881)
* remove all but one UnusedImport warning

* bump a few more spec version references from v0.10.1 to v0.11.1

* more v0.10.1 spec reference updates/removals

* yet more v0.10.1 spec reference updates
2020-04-10 16:06:24 +02:00
Jacek Sieka 0f47c76b50
reenable block bls verification (#875)
* finalizing state_transition

* cleanup

Co-authored-by: Dustin Brody <tersec@users.noreply.github.com>
2020-04-09 09:41:02 +00:00
tersec daabb1b5b2
complete all (non-semantic, comment-only) 0.11.0 -> 0.11.1 beacon chain spec updates, mark all beacon chain v0.11.0 spec references as v0.11.1, and remove now unnecessary/unused UpdateFlags vars from 4 test fixtures (#841) 2020-03-30 23:40:24 +00:00
tersec f5f939bd31
0.11.1 beacon chain spec update (#836)
* initial 0.11.1 spec commit; no test regressions and finalizes in eth2_network_simulation

* with BLS 0.10/0.11 available, stop skipping attester slashing, proposer slashing, and voluntary exist operations fixture tests

* switch param orders to group state.{fork, genesis_validators_root}; bump spec/datatypes spec version for network purposes

* mark attestation construction and broadcast and some minimal/mainnet constants as 0.11.1-compatible; remove phase 1 sharding constants from minimal which don't exist in that preset
2020-03-30 11:31:44 +00:00
Jacek Sieka 689bcf71c4 clean up block creation
* consistently use state at new block slot to produce block
* factor out signature funcs
* fix missing block application test
2020-03-22 11:15:07 +00:00
Mamy Ratsimbazafy ccace9a034
Fix shortlog crypto (#818)
* fix shortlogs of crypto types

* Valid signature without a "real: " prefix tag

* remove 0x prefix
2020-03-19 17:18:48 +01:00
Dustin Brody 313109c743 align process_block(...) name with spec; fix beacon chain proposer index generation when some validators aren't active to fix the two regressing 0.11.0 block sanity tests 2020-03-17 12:06:56 +02:00
Dustin Brody 6068dd960b initial 0.11.0 spec version update 2020-03-17 12:06:56 +02:00
Mamy André-Ratsimbazafy bfd4df912e Update to BLS v0.10.1 2020-03-13 12:20:41 +00:00
Jacek Sieka 9a3db7a81f
[WIP] Fake bls at runtime (#735)
* Initial implementation of runtime bls skipping.

Add libnfuzz skipBLSValidation handling, check that it propagates.

* Rename skipBLSValidation -> skipBlsValidation, start skipStateRootValidation
* Replace skipValidation flags with more granular flags.

Also added skipBlockParentRootValidation flag
Mainly replaced with skipBlsValidation but also StateRoot or
BlockParentRootValidation flags where appropriate.

* Adjust interop test to pass when skipping merkle validation.
* Stop skipping validation for mainchain_monitor.
* Remove comment.
* Also skipMerkleValidation for test_beacon_chain_db.
2020-03-05 13:52:10 +01:00
Dustin Brody 1ffc2df23d add a couple new deposit tests; fix the false-positive BLS verifications while keeping all but two tests working, despite mismatched 0.9/0.10 BLS standards; better-factor the skipping of BLS validation and Merkle tree validation 2020-02-04 18:39:38 +00:00
Dustin Brody e4922cb177 implement compute_signing_root() and mark some functions as 0.10.1-compatible 2020-02-04 12:24:30 +00:00
Dustin Brody 3efbe36c30 additional 0.10.1 updates where already-implemented code is already done 2020-01-27 12:50:54 +00:00