Commit Graph

483 Commits

Author SHA1 Message Date
tersec 8ebd496fbe
Altair transition tests (#2624)
* Working Altair transition tests

* with fixed upstream test vectors, remove state root workaround

* switch upgrade_to_altair() to returning a reference

* remove test_state_transition

* fix invalid fork state/block combinations error messages

* avoid memory copies by reintroducing state_transition_slots(var SomeHashedBeaconState)
2021-06-04 10:38:00 +00:00
Jacek Sieka abe0d7b4ae singe validator key cache
Instead of keeping a validator key list per EpochRef, this PR introduces
a single shared validator key list in ChainDAG, and cleans up some other
ChainDAG and key-related issues.

The PR does not introduce the validator key list in the state transition
- this is because we batch-check all signatures before entering the spec
code, thus the spec code never hits the cache.

A future refactor should _probably_ remove the threadvar altogether.

There's a few other small fixes in here that make the flow easier to
read:

* fix `var ChainDAGRef` -> `ChainDAGRef`
* fix `var QuarantineRef` -> `QuarantineRef`
* consistent `dag` variable name
* avoid using threadvar pubkey cache in most cases
* better error messages in batch signature checking
2021-06-01 20:43:44 +03:00
tersec a808a6755a
add Altair fork tests for upgrade_to_altair() (#2620)
* add Altair fork tests for upgrade_to_altair()

* Deposit processing ensures all keys are valid
2021-05-31 12:54:53 +00:00
tersec c06ffc7804
proposed structure for altair (#2323)
* proposed structure for hf1

* refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim

* hf1 is Altair

* some syncing with alpha 2

* adjust epoch processing to disambiguate access to RewardFlags

* relocate StateData to stay consistent with meaning phase 0 StateData

* passes v1.1.0 alpha 5 SSZ consensus object tests

* Altair block header test fixtures work

* fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work

* deposit operation Altair test fixtures work

* slot sanity and all but a couple epoch transition tests switched to Altair

* attestation Altair test fixtures work

* Altair block sanity test fixtures work

* add working altair sync committee tests

* improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
tersec 0b0bfd1de0
use StateData in place of BeaconState outside state transition code (#2551)
* use StateData in place of BeaconState outside state transition code

* propagate more StateData usage

* remove withStateVars().state

* wrap get_beacon_committee(BeaconState, ...) as gbc(StateData, ...)

* switch makeAttestation() to use StateData

* use StateData wrapper/dispatcher for get_committee_count_per_slot()

* convert AttestationCache.init(), weak subjectivity functions, and updateValidatorMetrics()

* add get_shuffled_active_validator_indices(StateData) and get_block_root_at_slot(StateData)

* switch makeAttestationData() to StateData

* sync AllTests-mainnet.md after rebase
2021-05-21 09:23:28 +00:00
Jacek Sieka 97f4e1fffe
Db1 cont (#2573)
* Revert "Revert "Upgrade database schema" (#2570)"

This reverts commit 6057c2ffb4.

* ssz: fix loading empty lists into existing instances

Not a problem earlier because we didn't reuse instances

* bump nim-eth

* bump nim-web3
2021-05-17 18:37:26 +02:00
tersec 6057c2ffb4
Revert "Upgrade database schema" (#2570)
This reverts commit 22ddf74752.
2021-05-17 06:34:44 +00:00
Jacek Sieka 22ddf74752 Upgrade database schema
The `kvstore` design we're using now turns out to not be the best way to
use `sqlite` - in particular, there are some significant benefits to
using rowid in certain situations and to keep data in separate tables.

With this branch, there are massive improvements in startup time
(seconds instead of minutes) and state/block storage and pruning times
(milliseconds instead of seconds) - these improvements can in particular
be seen on slow drives and translate directly into better attestation
performance.

* update kvstore to new keyspace design
* remove `DirStoreRef` and the hidden `--state-db-kind` option - this
was an experiment to store large blobs in files, but with the new
kvstore, there's no compelling reason to do so
* remove `DbMap` - unused and would need updating for new keyspace
design
* introduce separate tables for each data type (blocks, states etc)
* remove "WITHOUT ROWID" pessimization for tables with large blobs
* close DbSeq statements explicitly (and earlier)
* store beacon block summaries in separate table, without SSZ
compression and load them all with single query on startup
* stop storing backwards compat full states
* mark genesis beacon block as trusted
* avoid faststreams when loading SSZ data
* remove `DisagreementBehavior` (unused)
2021-05-14 20:05:23 +03:00
tersec e0f4d28116
rename initialize_beacon_state to initialize_beacon_state_from_eth1 (#2536) 2021-05-04 12:19:11 +02:00
Dustin Brody 7f42d38219 rename initialize_beacon_state{_from_eth1,}; suppress warnings when doppelganger detection disabled 2021-04-28 00:12:41 +03:00
Jacek Sieka f1f424cc2d attestation processing speedups
* avoid creating indexed attestation just to check signatures - above
all, don't create it when not checking signatures ;)
* avoid pointer op when adding attestation to pool
* better iterator for yielding attestations
* add metric / log for attestation packing time
2021-04-14 21:51:17 +03:00
Jacek Sieka 7165e0ac31
Reset cached indices when resetting cache on SSZ read (#2480)
* Reset cached indices when resetting cache on SSZ read

When deserializing into an existing structure, the cache should be
cleared - goes for json also. Also improve error messages.
2021-04-08 13:11:04 +03:00
tersec 36311bfc05
incorporate proposals into nextActionWait; switch some proc to func (#2438) 2021-03-24 10:05:04 +00:00
tersec 97f7284e51
bump spec refs from v1.0.0 to v1.0.1 and update copyright years (#2357) 2021-02-25 13:37:22 +00:00
Dustin Brody 490e4c7666 don't used sorted() in get_indexed_attestation() 2021-01-30 00:23:02 +02:00
tersec 1bdbf099cc
use IntSet rather than HashSet[ValidatorIndex] (#2267)
* use IntSet rather than HashSet[ValidatorIndex]

* add bounds check before uint64 -> int conversion

* use intsets in block transitions

* remove superfluous Nim issue explanation/reference
2021-01-26 12:52:00 +01:00
Jacek Sieka 5ca10d3de2 avoid copying keys for trusted signatures
this removes `is_valid_indexed_attestion` from performance benchmarks
when replaying blocks
2021-01-25 19:48:02 +02:00
Jacek Sieka 5713a3ce4c
performance fixes (#2259)
* performance fixes

* don't mark tree cache as dirty on read-only List accesses
* store only blob in memory for keys and signatures, parse blob lazily
* compare public keys by blob instead of parsing / converting to raw
* compare Eth2Digest using non-constant-time comparison
* avoid some unnecessary validator copying

This branch will in particular speed up deposit processing which has
been slowing down block replay.

Pre (mainnet, 1600 blocks):

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3450.269,        0.000,     3450.269,     3450.269,            1, Initialize DB
       0.417,        0.822,        0.036,       21.098,         1400, Load block from database
      16.521,        0.000,       16.521,       16.521,            1, Load state from database
      27.906,       50.846,        8.104,     1507.633,         1350, Apply block
      52.617,       37.029,       20.640,      135.938,           50, Apply epoch block
```

Post:

```
    3502.715,        0.000,     3502.715,     3502.715,            1, Initialize DB
       0.080,        0.560,        0.035,       21.015,         1400, Load block from database
      17.595,        0.000,       17.595,       17.595,            1, Load state from database
      15.706,       11.028,        8.300,      107.537,         1350, Apply block
      33.217,       12.622,       17.331,       60.580,           50, Apply epoch block
```

* more perf fixes

* load EpochRef cache into StateCache more aggressively
* point out security concern with public key cache
* reuse proposer index from state when processing block
* avoid genericAssign in a few more places
* don't parse key when signature is unparseable
* fix `==` overload for Eth2Digest
* preallocate validator list when getting active validators
* speed up proposer index calculation a little bit
* reuse cache when replaying blocks in ncli_db
* avoid a few more copying loops

```
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3279.158,        0.000,     3279.158,     3279.158,            1, Initialize DB
       0.072,        0.357,        0.035,       13.400,         1400, Load block from database
      17.295,        0.000,       17.295,       17.295,            1, Load state from database
       5.918,        9.896,        0.198,       98.028,         1350, Apply block
      15.888,       10.951,        7.902,       39.535,           50, Apply epoch block
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```

* clear full balance cache before processing rewards and penalties

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    3947.901,        0.000,     3947.901,     3947.901,            1, Initialize DB
       0.124,        0.506,        0.026,      202.370,       363345, Load block from database
      97.614,        0.000,       97.614,       97.614,            1, Load state from database
       0.186,        0.188,        0.012,       99.561,       357262, Advance slot, non-epoch
      14.161,        5.966,        1.099,      395.511,        11524, Advance slot, epoch
       1.372,        4.170,        0.017,      276.401,       363345, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```
2021-01-25 13:04:18 +01:00
Jacek Sieka e7f2735271
fix broken metrics during replay (#2090)
* move metrics out of state transition
* add validator count metric
* remove expensive beacon_current_validators, beacon_previous_validators
metrics (they should be reimplemented with cache), add cheap
beacon_active_validators to approximate
* remove unused validator count metrics
* tidy imports/defects
2020-11-27 23:16:13 +01:00
Zahary Karadjov a995cdd3d9
Bugfix: fix an incorrect balance increase during genesis state generation 2020-11-24 23:30:26 +02:00
tersec 9e716b32bd
address some XDeclaredButNotUsed hints (#2028) 2020-11-17 11:14:53 +01:00
Zahary Karadjov b022dc4d1f Use O(n) algorithm in initialize_beacon_state_from_eth1; Avoid unnecessary merkle proofs generation 2020-11-15 21:40:40 +02:00
tersec b9e122a60c
address some TODOs (#2005) 2020-11-12 19:24:07 +00:00
tersec 271df8b604
bump 1.0.0rc-0 spec refs to 1.0.0 (#1974) 2020-11-09 14:18:55 +00:00
Jacek Sieka 95f5f76180
Datatype cleanup (#1953)
* clear up spec todo

* test fix

* remove unnecessary toSszType

* type

* one more
2020-11-04 21:52:47 +00:00
Zahary Karadjov 14b2d4324d openarray -> openArray 2020-11-03 23:23:10 +02:00
Jacek Sieka ca63d48b82
tone down validator exit logging (#1939)
Validators exiting is normal, no need to scream about it

* avoid reallocating seq on big exit queue
* avoid fetching state cache when updating head (it's rarely needed)
* remove incorrectly implemented live validator counts (avoids memory
allocs)
2020-11-02 18:34:23 +01:00
Jacek Sieka 7c0b4d28d2 speed up reward/penalty calculation
Calculating rewards/penalties is slow due to how we compute sets of
attestations validators then use the sets for inclusion checks, to see
who attested. The dominant function during validated block processing /
epoch processing is hash set building and lookup.

This PR inverts the flow by removing the sets and creating a single
large validator status list, then applying all relevant state
attestations, then updating rewards and penalties.

This provides a 10x speedup to epoch processing which in turn speeds up
both empty slot and block processing - for example, on startup, we
replay all non-finalized blocks to prime fork choice - the same when
validating attestations or replaying states on reorg.
2020-10-23 19:23:36 +03:00
Jacek Sieka 499e5ca991
misc memory and perf fixes (#1899)
* misc memory and perf fixes

* use EpochRef for attestation aggregation
* compress effective balances in memory (medalla unfinalized: 4gb ->
1gb)
* avoid hitting db when rewinding to head or clearance state
* avoid hitting db when blocks can be applied to in-memory state -
speeds up startup considerably
* avoid storing epochref in fork choice
* simplify and speed up beacon block creation flow - avoids state reload
thanks to head rewind optimization
* iterator-based committee and attestation participation help avoid lots
of small memory allocations throughout epoch transition (40% speedup on
epoch processing, for example during startup)

* add constant for threshold
2020-10-22 12:53:33 +02:00
tersec e87e0ef179
update to 1.0.0-rc.0 message ID; unexport funcs; enable test in mainnet; remove unused subscribe-with-nondummy-handler function; fix gossip received metric (#1884) 2020-10-20 08:54:11 +00:00
tersec 3ee2dd8da4
p2p-interface spec ref bump (except non-updated places) (#1862) 2020-10-12 14:37:14 +00:00
tersec 1994ffe5a0
update 130+ spec references from v0.12.3 to v1.0.0-rc1 (#1854) 2020-10-12 08:59:24 +00:00
tersec b79e5f8af5
update nim-beacon-chain to nimbus-eth2 in beacon_chain/, ncli/, tests/, and README.md (#1843) 2020-10-08 19:02:05 +00:00
tersec c0aa2f1e58
update process_slashings() to use PROPORTIONAL_SLASHING_MULTIPLIER (#1841) 2020-10-08 18:00:07 +00:00
Zahary Karadjov aed291128a Add support for starting from weak subjectivity checkpoints
Also removes the `genesis.ssz` file stored in the data folder.
The `medalla-fast-sync` target has been adapted to use the new features.
2020-10-07 09:32:03 +03:00
Mamy Ratsimbazafy 0280d6c73e
Revisiting log levels (#1788)
* Update log level - https://github.com/status-im/nim-beacon-chain/issues/1779 https://github.com/status-im/nim-beacon-chain/issues/1785

* Address review comments

* Document the logging strategy [skip ci]
2020-10-01 20:56:42 +02:00
tersec f96ad87d28
switch another 50+ spec refs from v0.12.2 to v0.12.3 (#1749) 2020-09-25 11:52:50 +00:00
tersec 6398a43cc1
update 120+ beacon_chain and validator spec refs from v0.12.2 to v0.12.3 (#1740) 2020-09-24 19:04:10 +02:00
tersec b3b578501a
mark comments documenting functions as a whole as such (#1613) 2020-09-08 08:54:55 +00:00
tersec e4a43f7628
address issue #1580 (#1600)
* address issue #1580

* Update beacon_chain/spec/beaconstate.nim

Co-authored-by: Jacek Sieka <jacek@status.im>
2020-09-02 16:16:25 +00:00
Jacek Sieka 9244ae7a38 more speedups
* evaluate block attestations under the epochref of the block - this is
what the state transition function does
* avoid copying attestation seq unnecessarily
* avoid unnecessary hashset for unslashed indices
2020-08-19 14:51:04 +03:00
Jacek Sieka a1bd44f4b0
small spec cleanups (#1501)
* clean up logging a bit
* return error on indexed attestation check
2020-08-13 13:47:06 +00:00
tersec 81b3c0ea40
update spec refs to v0.12.2 (#1457)
* update spec refs to v0.12.2 and change a .len.uint64 to .lenu64

* pull back from any non-pure-comment changes, since Jenkins is being wonky
2020-08-06 13:05:13 +00:00
Dustin Brody 8618cbdf38 keep representations consistent for tests 2020-08-04 23:00:33 +03:00
Dustin Brody e8c31e4ce8 just in case 2020-08-04 23:00:33 +03:00
Dustin Brody 2c7d207a72 make test passes 2020-08-04 23:00:33 +03:00
Dustin Brody c142de4b7f be more consistent about pubkeys fed to verify_foo_signature() not being separately initialized, while pubkeys, generally, used for matching purposes, elsewhere explicitly initialized 2020-08-04 23:00:33 +03:00
Dustin Brody db397cc5f3 lazily initialize validator public keys, which were a third or more of sync CPU usage 2020-08-04 23:00:33 +03:00
tersec a979568d74
switch mapIt/find to in-place linear search (#1436) 2020-08-04 17:15:27 +00:00
Jacek Sieka cf8cd8321b
Revert "Lazy crypto [alt design #1369] (#1371)" (#1435)
This reverts commit 023f7f4518.
2020-08-04 17:15:13 +00:00
tersec df80071bcf
update attestation and block validation to v0.12.2; clean up getAncestorAt()/get_ancestor() (#1417)
* update attestation validation to v0.12.2; clean up getAncestorAt()/get_ancestor()

* update beacon block validation to v0.12.2
2020-08-03 19:47:42 +00:00
tersec 99f2d8e06c
update 14 v0.12.1 spec refs to v0.12.2 (#1400) 2020-07-30 09:47:57 +00:00
Mamy Ratsimbazafy 023f7f4518
Lazy crypto [alt design #1369] (#1371)
* Lazy loading of crypto objects

* Try to fix incorrect field access by hiding fields but no luck. SSZ/Chronicles/macro bug?

* Fix incorrect blsValue access. was "aggregate" not "chronicles"

* Fix tests that rely on the internal BLSValue representation
2020-07-29 18:13:05 +00:00
Jacek Sieka 157ddd2ac4
Fork choice fixes 5 (#1381)
* limit attestations kept in attestation pool

With fork choice updated, the attestation pool only needs to keep track
of attestations that will eventually end up in blocks - we can thus
limit the horizon of attestations that we keep more aggressively.

To get here, we expose getEpochRef which gets metadata about a
particular epochref, and make sure to populate it when a block is added
- this ensures that state rewinds during block addition are minimized.

In addition, we'll use the target root/epoch when validating
attestations - this helps minimize the number of different states that
we need to rewind to, in general.

* remove CandidateChains.justifiedState

unused

* remove BlockPools.Head object

* avoid quadratic quarantine loop

* fix
2020-07-28 13:54:32 +00:00
tersec dad3dd5809
update 111 v0.12.1 beacon chain spec refs to v0.12.2 (#1380) 2020-07-27 18:49:46 +02:00
Jacek Sieka 48cebc7157
spec cleanups (#1379)
* add helper for beacon committee length (used for quickly validating
attestations)
* refactor some attestation checks to do cheap checks early
* validate attestation epoch before computing active validator set
* clean up documentation / comments
* fill state cache on demand
2020-07-27 16:04:44 +00:00
tersec 20a2525390
v0.12.2 beacon chain protocol update (#1378) 2020-07-27 12:59:57 +02:00
tersec 90708a8287
Prefer converting int` to uint64 and switch foo.len.uint64 to .len64 (#1375)
* avoid converting from uint64 to int, and where most feasible, int type conversion at all

* .len.uint64 -> .len64

* fix 32-bit compilation

* try keeping state_sim loop variable/bounds as int for 32-bit Azure

* len64 -> lenu64
2020-07-26 20:55:48 +02:00
Jacek Sieka e0a18a3105
cache beacon committee size calculation (#1363)
* cache beacon committee size calculation

this fixes a bug in get_validator_churn_limit as well

* fix

* make committee counts consistently uint64

mixing feels like the worst of the two worlds
2020-07-23 19:01:07 +02:00
Jacek Sieka fb2f742972
Fork choice fixes 2 (#1356)
* fork choice cleanup

* enable v2 pruning
* prefer `get_current_epoch`
* fix finalization check to use correct epoch

* small cleanups

* add `count_active_validators`
* remove misleading logs
* fix justified checkpoint slot calculation in rpc
2020-07-22 23:01:44 +02:00
tersec 4a9a7be271
faster syncing (#1348)
* maybe faster syncing

* 80-character lines

* remove instrumentation debugEchos; fix target attestation epoch in attestation pool validation

* use the epoch-granularity matching in attestation.addResolved(...)
2020-07-22 09:51:45 +02:00
tersec 83abbcb917
drop get_attesting_indices()/get_unslashed_attesting_indices() from 15% to 1% of workload at block_sim at 100k validators (#1351) 2020-07-21 18:35:43 +02:00
Jacek Sieka 8b01284b0e
cache block hash (#1329)
hash_tree_root was turning up when running beacon_node, turns out to be
repeated hash_tree_root invocations - this pr brings them back down to
normal.

this PR caches the root of a block in the SignedBeaconBlock object -
this has the potential downside that even invalid blocks will be hashed
(as part of deserialization) - later, one could imagine delaying this
until checks have passed

there's also some cleanup of the `cat=` logs which were applied randomly
and haphazardly, and to a large degree are duplicated by other
information in the log statements - in particular, topics fulfill the
same role
2020-07-16 15:16:51 +02:00
tersec 26e893ffc2
restore EpochRef and flush statecaches on epoch transitions (#1312)
* restore EpochRef and flush statecaches on epoch transitions

* more targeted cache invalidation

* remove get_empty_per_epoch_cache(); implement simpler but still faster get_beacon_proposer_index()/compute_proposer_index() approach; add some abstraction layer for accessing the shuffled validator indices cache

* reduce integer type conversions

* remove most of rest of integer type conversion in compute_proposer_index()
2020-07-15 12:44:18 +02:00
zah 0be77f9cbc
Restore compilation with custom presets (#1309)
This also assigns precise types to the constants in the minimal
and mainnet presets in order to reduce the chance of compilation
errors when custom presets are used (previously, only the custom
presets have precisely assigned types for the constants).
2020-07-13 16:44:58 +02:00
Zahary Karadjov 3ec6a02b12
Merge devel and resolve conflicts 2020-07-10 02:02:40 +03:00
tersec 61b0b5af17
update most remaining non-fork-choice spec refs, updating code where necessary (#1292)
* update most of the remaining non-fork-choice spec refs, updating code where necessary

* revert presumably harmless compute_signing_root() change, but this way, keep things really unchanged outside inspector
2020-07-09 11:43:27 +00:00
Mamy Ratsimbazafy 3cdae9f6be
Dual headed fork choice [Revolution] (#1238)
* Dual headed fork choice

* fix finalizedEpoch not moving

* reduce fork choice verbosity

* Add failing tests due to pruning

* Properly handle duplicate blocks in sync

* test_block_pool also add a test for duplicate blocks

* comments addressing review

* Fix fork choice v2, was missing integrating block proposed

* remove a spurious debug writeStackTrace

* update block_sim

* Use OrderedTable to ensure that we always load parents before children in fork choice

* Load the DAG data in fork choice at init if there is some (can sync witti)

* Cluster of quarantined blocks were not properly added to the fork choice

* Workaround async gcsafe warnings

* Update blockpoool tests

* Do the callback before clearing the quarantine

* Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads

* Make it work with latest devel - Altona readyness

* Add a recovery mechanism when forkchoice desyncs with blockpool

* add the current problematic node to the stack

* Fix rebase indentation bug (but still producing invalid block)

* Fix cache at epoch boundaries and lateBlock addition
2020-07-09 11:29:32 +02:00
Dustin Brody 4140b3b9d9 update 29 spec refs to v0.12.1 2020-07-08 20:49:25 +00:00
Zahary Karadjov 318b225ccd
Merge devel and resolve the conflicts 2020-07-08 15:36:03 +03:00
Dustin Brody fc8502c54e halve memory usage from state caches 2020-07-08 10:21:41 +00:00
Zahary Karadjov c4af4e2f35
Working test suite with run-time presets 2020-07-08 02:02:14 +03:00
Zahary Karadjov 441ae9b58c
Introduce a RuntimePreset object for the overridable testnet preset values 2020-07-03 22:29:23 +03:00
tersec a92276d510
adopt Result[void, string] in place of some bool return signatures (#1275)
* adopt Result[void, string] in place of some bool return signatures

* string -> cstring to reduce memory allocations; ensure all err() strings are constants, with contextual information from higher-level callers

* logScope usage fixes

* homogenize err() reporting convention

* invalid signature in deposit isn't an error
2020-07-03 17:03:14 +00:00
Zahary Karadjov 31e31bb30c Switch to monitoring of Blocks; More accurate genesis detection and faster Eth1 syncing 2020-06-29 02:18:48 +03:00
zah 578ea2013a
implement spev v0.12.1 in initialize_beacon_state_from_eth1 (#1236) 2020-06-25 16:24:55 +02:00
Jacek Sieka 1301600341
Trusted blocks (#1227)
* cleanups

* fix ncli state root check flag
* add block dump to ncli_db
* limit ncli_db benchmark length
* tone down finalization logs

* introduce trusted blocks

We only store blocks whose signature we've verified in the database - as
such, there's no need to check it again, and most importantly, no need
to deserialize the signature when loading from database.

50x startup time improvement, 200x block load time improvement.

* fix rewinding when deposits have invalid signature
* speed up ancestor iteration by avoiding copy
* avoid deserializing signatures for trusted data
* load blocks lazily when rewinding (less memory used)

* chronicles workarounds

* document trustedbeaconblock
2020-06-25 12:23:10 +02:00
Mamy Ratsimbazafy 902093f57c
Revert "Dual headed fork choice [Reloaded] (#1223)" (#1234)
This reverts commit 6836d41ebd.
2020-06-25 11:36:03 +02:00
Mamy Ratsimbazafy 6836d41ebd
Dual headed fork choice [Reloaded] (#1223)
* Dual headed fork choice

* fix finalizedEpoch not moving

* reduce fork choice verbosity

* Add failing tests due to pruning

* Properly handle duplicate blocks in sync

* test_block_pool also add a test for duplicate blocks

* comments addressing review
2020-06-24 20:24:36 +02:00
Dustin Brody 62a7e7fede fix is_valid_indexed_attestation() to check for too-high attestation indices 2020-06-22 14:15:51 +00:00
Dustin Brody 4ecbc655ea switch 11 beaconstate.nim spec refs from v0.11.x to v0.12.1 2020-06-19 11:54:37 +00:00
tersec dc1a565b3f
support v0.12.1 attestation topics in beacon node/inspector subscribing (#1187)
* support v0.12.1 attestation topics in beacon node and inspector subscribing

* bump is_valid_merkle_branch() spec ref
2020-06-18 15:10:25 +02:00
tersec ee9f4a2e3f
remove skipMerkleValidation and skipBlockParentRootValidation (#1197) 2020-06-18 07:56:47 +02:00
Jacek Sieka 5c25d23ef1 eth2hash -> eth2digest
hash in nim is the insecure hash-map helper - might as well use `digest`
consistently to mark the difference
2020-06-16 13:30:21 +00:00
Jacek Sieka 89e4819ce9
collect signature production and verificaiton in one place (#1179)
* collect signature production and verificaiton in one place

Signatures are made over data and domain - here we collect all such
activities in one place.

Also:
* security: fix cast-before-range-check
* log block/attestation verification consistently
* run block verification based on `getProposer` in its own history
* clean up some unused stuff

* import

* missing raises
2020-06-16 07:45:04 +02:00
Dustin Brody 1d1e90adf4 Update 36 spec refs from 0.11.x to 0.12.1 2020-06-12 19:41:03 +00:00
Jacek Sieka 42832cefa8
Small fixes (#1165)
* random fixes

* create dump dir on startup
* don't crash on failure to write dump
* fix a few `uint64` instances being used when indexing arrays - this
should be a compile error but isn't due to compiler bugs
* fix standalone test_block_pool compilation
* add signed block processing in ncli

* reuse cache entry instead of allocating a new one

* allow for small clock disparities when validating blocks
2020-06-12 18:43:20 +02:00
Mamy Ratsimbazafy ce897fe83f
[Split fork choice PR] Derisk-ed attestation checks changes (#1154)
* Derisked attestation pool improvements

* tune down frequent logs

* VoteTracker logging
2020-06-10 08:58:12 +02:00
Jacek Sieka 56ffb696be
reorder ssz (#1099)
* reorder ssz

* split into hash_trees and ssz_serialization, roughly, for hashing and
IO
* move bitseqs into ssz (from stew)
* clean up imports

* docs, imports
2020-06-03 15:52:02 +02:00
tersec b7bb62d48b
use shuffled validator cache in more places; cleanups (#1095) 2020-06-03 07:42:08 +02:00
Zahary Karadjov a8003e7e38 More complete integration of the List type; Detect more invalid inputs; Enable more tests 2020-05-24 19:00:34 +03:00
Dustin Brody b9817867f5 audit & update spec references for some slashing and rewards and penalty functions; check and update all remaining v0.11.0 spec references to v0.11.3 [skip ci] 2020-05-20 14:57:08 +00:00
Dustin Brody 3fa2f70853 update all remaining beacon chain spec references from v0.11.1 and v0.11.2 to v0.11.3 2020-05-20 08:48:18 +00:00
Zahary Karadjov a99977b772 Progress towards #991; Enable the distinct ssz.List type 2020-05-19 15:53:03 +03:00
tersec 8432932c11
update to spec v0.11.2 to keep compatible with Schlesi (#1036)
* update to spec v0.11.2 to keep compatible with Schlesi

* update spec/state_transition_epoch spec references to v0.11.2

* bump other spec refs
2020-05-18 23:05:27 +00:00
Jacek Sieka fb2e0ddbec
sync fixes (#1005)
* sync fixes

* fix Status message finalized info
* work around sync starting before initial status exchange
* don't fail block on deposit signature check failure (fixes #989)
* print ForkDigest and Version nicely
* dump incoming blocks
* fix crash when libp2p peer connection is closed
* update chunk size to 16 to work around missing blocks when syncing

* bump libp2p

* bump libp2p

* better deposit skip message
2020-05-11 18:08:52 +00:00
tersec 093d298b2b
Increase finalization and finalization checking robustness (#990)
* fix some warnings related to beacon_node splitting; reimplement finalization verification more robustly; improve attestation pool block selection logic

* re-add missing import

* whitelist allowed state transition flags and make rollback/restore naming more consistent

* restore usage of update flags passed into skipAndUpdateState(...) in addition to the potential verifyFinalization flag

* switch rest of rollback -> restore
2020-05-09 12:43:15 +00:00
tersec c548b90a0e
default preset update; rm process_slots(BeaconState); some BLS skipping disabled (#963)
* re-enable BLS validation for block sanity tests

* update default preset to mainnet; remove process_slots(state: var BeaconState)

* bump spec ref to v0.11.1

* bump all v0.11.1 spec references except the actually changed is_valid_indexed_attestation(...) in spec/beaconstate to v0.11.2

* remove stack-risky/low-ROI doAssert
2020-05-04 21:07:18 +00:00
tersec cd939d4606
fix spec/beaconstate compile error in the presence of other PRs' SSZ tightening (#952) 2020-04-30 09:01:55 +00:00
tersec 3a56ddc5c4
begin removing plain BeaconState versions of state transition functions (#951)
* remove near-duplicate code paths: process_slot(), process_slots(), and state_transition() for BeaconState are now wrappers around the HashedBeaconState versions

* convert tests/test_state_transition.nim to use HashedBeaconState

* convert mocking infrastructure and spec_block/epoch_processing tests to use HashedBeaconBlock, and remove thus unused process_slot*(state: var BeaconState)
2020-04-30 08:44:19 +02:00
Zahary Karadjov 4e9fa51ae9 Introduce BeaconNodeRef and use it in all the right places 2020-04-26 13:04:53 +03:00
tersec 65353bab76
add another check for inconsistent aggregation and committee length (#927)
* add another check for inconsistent aggregation and committee length, since ncli_transition bypasses process_attestation(...)/check_attestation(...) and calls almost directly into process_epoch(...)

* bump validator functions to v0.11.1 spec references

* bump some spec references to v0.11.1

* poke
2020-04-24 09:16:40 +02:00
Jacek Sieka ed74770451
spec: regulate exceptions (#913)
* spec: regulate exceptions

* a few more simple raises
2020-04-22 07:53:02 +02:00
tersec 480c11f8aa
fix remaining block sanity test vectors (#902)
* fix remaining block sanity test vectors

* bump epoch processing spec refs to v0.11.1
2020-04-18 00:09:34 +02:00
Dustin Brody 164d171bd9
revert previous commit 2020-04-17 20:27:42 +02:00
Dustin Brody ef92529feb
fix remaining block sanity test vectors 2020-04-17 20:25:18 +02:00
Dustin Brody 3df9a22ff7 fix remaining attester slashing tests from EF phase0 spec tests by checking for uniqueness 2020-04-17 16:01:32 +00:00
Dustin Brody 05d1d1246a bump a couple more eth2 phase0 spec refs to v0.11.1 2020-04-17 16:37:52 +03:00
Dustin Brody d559f4ee89 create distinct CommitteeIndex type 2020-04-15 09:59:49 +00:00
Dustin Brody 2771deadfc re-add test_interop to all_tests and mark several v0.10.1 phase 0 spec references as v0.11.1 2020-04-07 13:16:55 +03:00
Dustin Brody e7de8aae20 fix/restore 0.11.1 BeaconState genesis interop: (a) use GENESIS_FORK_VERSION properly in BeaconState init; (b) use GENESIS_FORK_VERSION to be compatible with zcli/zrnt initial eth1 deposits; (c) let List[foo] work with 64-bit ints, so BeaconState.validators SSZ-serializes properly; and (d) update test_interop to use new spec/crypto API 2020-04-07 13:16:55 +03:00
Dustin Brody 60df05a420 mark several v0.10.1 spec references as v0.11.1-compatible 2020-04-04 13:43:04 +00:00
tersec ea59f2ecf2
comments-only changes: update a bunch of v0.10.1 spec references to v0.11.0 and explain rationale for extended validation design in code (#847) 2020-04-01 11:41:39 +00:00
tersec cd388bc9bb
extended validation (#812)
* initial extended validation setup

* flesh out all TODO items for attestation and beaconblock verification

* fix finalization and add chronicles debugging messages

* directly use blockPool.headState rather than pointlessly updating it and document this constraint

* fix logic relating to first-attestation checking; support validating blocks across multiple forks
2020-03-31 18:39:02 +00:00
tersec daabb1b5b2
complete all (non-semantic, comment-only) 0.11.0 -> 0.11.1 beacon chain spec updates, mark all beacon chain v0.11.0 spec references as v0.11.1, and remove now unnecessary/unused UpdateFlags vars from 4 test fixtures (#841) 2020-03-30 23:40:24 +00:00
tersec f5f939bd31
0.11.1 beacon chain spec update (#836)
* initial 0.11.1 spec commit; no test regressions and finalizes in eth2_network_simulation

* with BLS 0.10/0.11 available, stop skipping attester slashing, proposer slashing, and voluntary exist operations fixture tests

* switch param orders to group state.{fork, genesis_validators_root}; bump spec/datatypes spec version for network purposes

* mark attestation construction and broadcast and some minimal/mainnet constants as 0.11.1-compatible; remove phase 1 sharding constants from minimal which don't exist in that preset
2020-03-30 11:31:44 +00:00
Jacek Sieka 689bcf71c4 clean up block creation
* consistently use state at new block slot to produce block
* factor out signature funcs
* fix missing block application test
2020-03-22 11:15:07 +00:00
Dustin Brody cae1c0fd1a address beacon_node TODO regarding missing proposer index; mark initiate_validator_exit(...), is_eligible_for_activation_queue(...), Attestation, DepositMessage, BeaconBlockBody, PendingAttestation, Eth1Data, and integer_squareroot(...) as 0.11.0 2020-03-17 12:06:56 +02:00
Dustin Brody ebcde664e5 mark decrease_balance(...), get_validator_churn_limit(...), network spec version, Checkpoint, compute_epoch_at_slot(...), get_active_validator_indices(...), and get_committee_count_at_slot(...) as 0.11.0 2020-03-17 12:06:56 +02:00
Dustin Brody 6068dd960b initial 0.11.0 spec version update 2020-03-17 12:06:56 +02:00
Mamy André-Ratsimbazafy bfd4df912e Update to BLS v0.10.1 2020-03-13 12:20:41 +00:00
Jacek Sieka 9a3db7a81f
[WIP] Fake bls at runtime (#735)
* Initial implementation of runtime bls skipping.

Add libnfuzz skipBLSValidation handling, check that it propagates.

* Rename skipBLSValidation -> skipBlsValidation, start skipStateRootValidation
* Replace skipValidation flags with more granular flags.

Also added skipBlockParentRootValidation flag
Mainly replaced with skipBlsValidation but also StateRoot or
BlockParentRootValidation flags where appropriate.

* Adjust interop test to pass when skipping merkle validation.
* Stop skipping validation for mainchain_monitor.
* Remove comment.
* Also skipMerkleValidation for test_beacon_chain_db.
2020-03-05 13:52:10 +01:00
Dustin Brody a8567cafb5 mark out where/how AttestationData and Attestation construction are honest validator-compliant 2020-02-21 19:44:03 +00:00
Dustin Brody 1ffc2df23d add a couple new deposit tests; fix the false-positive BLS verifications while keeping all but two tests working, despite mismatched 0.9/0.10 BLS standards; better-factor the skipping of BLS validation and Merkle tree validation 2020-02-04 18:39:38 +00:00
Dustin Brody e4922cb177 implement compute_signing_root() and mark some functions as 0.10.1-compatible 2020-02-04 12:24:30 +00:00
Dustin Brody 45dd12cf3f update process_deposit() to actually check is_valid_merkle_branch() unless skipValidation specified 2020-01-30 09:31:08 +00:00
Dustin Brody 60b1775879 update check_attestation/process_attestation() to 0.10.1 and fill in missing check 2020-01-28 12:07:33 +02:00
Dustin Brody 3efbe36c30 additional 0.10.1 updates where already-implemented code is already done 2020-01-27 12:50:54 +00:00
Mamy Ratsimbazafy 1ba2cebd85 v0.10.1 (test vectors + unchanged parts) (#696)
* update to 0.10.1

* SSZ Generic and nbench uses the v0.10.1 fixtures

* Tests + spec links: v0.10.0 -> v0.10.1

* Add v0.10.1 TODO in get_latest_attesting_balance (forkchoice)

* SSZ Bytes are now ByteList

* Remove nim-result submodules that was leftover/added by mistake in the branch
2020-01-27 10:56:32 +00:00
Dustin Brody c301e5d42f mark 49 0.9.2/0.9.4 spec functions/data types as 0.10.0 2020-01-15 12:34:16 +00:00
Dustin Brody 46de726e3d mark all compatible 0.9.3 functions and data structures as 0.9.4 2019-12-20 17:24:42 +00:00
Mamy Ratsimbazafy 106352aff3
Nbench - Flexible benchmarking of Nimbus internals (#641)
* nbench PoC

* Remove the yaml files from the example scenarios

* update README with current status

* Add an alternative implementation that uses defer

* Forgot to add the old proc body

* slots-processing

* allow benching state_transition failures

* Add Attestations processing (workaround confutils bug:
- https://github.com/status-im/nim-confutils/issues/10
- https://github.com/status-im/nim-confutils/issues/11
- https://github.com/status-im/nim-confutils/issues/12

* Add CLI command in the readme

* Filter report and add notes about CPU cycles

* Report averages

* Add debugecho style time/cycle print

* Report when we skip BLS and state root verification

* Update to 0.9.3

* Generalize scenario parsing

* Support all block processing scenarios

* parallel bench runner PoC

* gitBetter load issues reporting (the load issues were invalid signature and expected to fail)
2019-12-20 17:14:43 +01:00
Dustin Brody 417f96213d fix genesis interop by fixing deposit signature handling; add TRACE-level compilation to CI 2019-12-20 15:26:48 +00:00
Dustin Brody ea4afd7454 mark 19 more 0.9.2-compatible entities as 0.9.3-compatible 2019-12-19 11:46:30 +00:00
Dustin Brody 350a6c071a mark a couple compatible dozen 0.9.2 functions/data structures as 0.9.3 2019-12-18 16:13:39 +00:00
Dustin Brody 0328cd49bf mark several more functions and data structures as 0.9.3 2019-12-18 10:00:17 +00:00
Dustin Brody 1610981be6 mark several more spec-referencing code/data parts as 0.9.3 2019-12-17 16:51:46 +00:00
Dustin Brody 27bf8908a9 mark a couple dozen functions and data structures as 0.9.3; fix TRACE-level compiling; stop downloading 0.9.2 test vectors; update cosmetic/display spec versions in test suite 2019-12-17 16:51:46 +00:00
Dustin Brody c824416f56 initial 0.9.3 spec update 2019-12-17 00:06:03 +02:00
Jacek Sieka a3145f0f5c cleanups
* move stuff out of beaconnode
* fix use-after-free in query
2019-12-10 14:47:52 +00:00
Jacek Sieka eea523a89e logging fixes
* don't log out-of-sync when there's no work to do
* small logging fixes
2019-12-10 10:18:30 +00:00
Dustin Brody c762c0232d render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators (#617)
* render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators

* follow addBlock(...) after refactoring

* [skip ci] fix typo
2019-12-04 11:49:59 +01:00
Dustin Brody e88e559caf render visible gradual validator emptying on catchup 2019-12-02 18:24:30 +02:00
Dustin Brody bde20436e8 ~50% speedup by fast-pathing Eth2Digest merkle hashing 2019-11-26 15:23:27 +00:00
Dustin Brody 1494bcc262 replace links to Apache and MIT licenses with HTTPS versions (#592) 2019-11-25 16:30:02 +01:00
Dustin Brody b82328b148 convert hash_tree_roots_prefix(...) to an iterator 2019-11-25 14:48:59 +00:00
Dustin Brody f6de87b80e remove O(n^2) eth1 deposit processing and scaffolding comparing it with O(n) deposit processing; re-enable now-viable-to-run genesis state initialization tests with around 16,384 validators 2019-11-25 14:48:59 +00:00
Dustin Brody fa2a703a17 switch more funcs and procs from global to module scope; create scaffolding for non-O(n^2) eth1 deposit processing with assertions for equivalent functionality; fix a few more shellcheck warnings 2019-11-25 14:48:59 +00:00
Dustin Brody a8de94ad05 add validator_leaving logging at debug chronicles level for all call paths to initiate_validator_exit(...) 2019-11-25 09:08:15 +00:00
Dustin Brody 8d9fdb0afd miscellanous cleanups and removing a few funcs/procs from unnecessary global scopes 2019-11-25 09:08:15 +00:00
Dustin Brody edfd65fd5d stop download 0.9.1 test vectors; finish 0.9.2 update, primarily via fixing preset constants; remove a couple 0.19.6 shims 2019-11-22 16:41:43 +00:00
Dustin Brody 41fa377638 mark all non-changed 0.9.1 items as 0.9.2 (#576) 2019-11-22 09:29:04 +01:00
Jacek Sieka 7c6d87051a
simplify validator api (#573)
* avoid passing whole state to validator api (easier when it goes
out-of-process)
* clean up some cruft
2019-11-21 10:57:59 +01:00
Dustin Brody c0f009908c
re-enable test_interop based on zcli with 0.9.1 specs and update initialize_beacon_state_from_eth1(...) to 0.9.1 (#569)
* re-enable test_interop based on zcli with 0.9.1 specs and update initialize_beacon_state_from_eth1(...) to 0.9.1

* switch many procs to funcs

* fix import os.nim instead; ospaths is deprecated [Deprecated] warnings
2019-11-21 09:15:10 +00:00
Dustin Brody 2749cb14a6 convert mostly-uncompiled verify_merkle_branch(...) calls to newer is_valid_merkle_branch(...) name; re-enable several lines of deposit testing code and outline roadmap for completion 2019-11-18 17:35:58 +00:00
Jacek Sieka 14712bbbdc
random warning fixes 2019-11-15 12:05:18 +01:00
Dustin Brody c08698bffe
scaffolding to remove last vestiges of get_committee_count(...), SHARD_COUNT, and rest of that 0.8.x infrastructure; this demonstrates equivalence between two, before discarding old code 2019-11-14 18:37:51 +01:00
Dustin Brody df1d00f13a remove UnusedImport spam (#549)
* remove UnusedImport spam

* re-add random import to beacon_node
2019-11-14 11:47:55 +01:00
Dustin Brody 5a54c823d8 update to 0.9.1: don't try to run removed tests; re-add 1.5 tests (one only in minimal); remove bls_verify_multiple(...) and AttestationDataAndCustodyBit; and update process_attester_slashing(...), get_indexed_attestation(...), and is_valid_indexed_attestation(...) 2019-11-13 13:49:38 +00:00
Dustin Brody d5ce142511 fix network sim finalization; remove get_attestation_data_slot(...); remove 2 more get_crosslink_committee(...) calls 2019-11-12 14:49:26 +00:00
Dustin Brody 568fa1dcb9 mark functions, constants, and data structures unchanged by 0.9.1 as such 2019-11-11 10:25:57 +00:00
Dustin Brody 949d735155 fix remaining test 0.9.0 test fixtures and switch back to non-transitiontest setup; remove get_committee_count(...); keep SHARD_COUNT consistent with MAX_COMMITTEES_PER_SLOT 2019-11-11 10:25:57 +00:00
Dustin Brody e984b2c932 mark is_valid_merkle_branch(...), process_justification_and_finalization(...), and process_rewards_and_penalties(...) as 0.9.0; update check_attestation(...) to 0.9.0; add 11 working test fixtures, having updated SSZ fixture directory to 0.9.0 2019-11-08 13:23:08 +00:00
Dustin Brody 3bb95f571f remove remaining Crosslink and CompactCommittee vestiges 2019-11-08 13:23:08 +00:00
Dustin Brody 61c2cf9415 rm get_compact_committees_root(...), Transfer, and references to 0.9-removed BeaconState.{active_index_roots,compact_committees_roots; temporarily disable genesis interop test pending 0.9 BeaconState 2019-11-08 13:23:08 +00:00
Dustin Brody 68654848cb update get_attesting_indices(...) to 0.9.0; remove Crosslink from AttestationData to update AttestationData to 0.9.0; rm get_winning_crosslink_and_attesting_indices(...) and get_crosslink_deltas(...) 2019-11-08 13:23:08 +00:00
Dustin Brody cb5454c4eb double-check that one can back-infer shard/epoch from slot/committee-index, in preparation for removing crosslink from AttestationData 2019-11-08 13:23:08 +00:00
Dustin Brody c2ebe482a5 temporarily disable SSZ tests, which fail when trying to read AttestationData or structures which embed it; implemenet get_committee_count_at_slot(...) and get_beacon_committee(...); express get_crosslink_committee(...) in terms of get_beacon_committee(...) and ensure that results are identical 2019-11-08 13:23:08 +00:00
Dustin Brody 0073555752
transition deposit operation fixtures to 0.9.0 (#515)
* transition deposit operations fixture to 0.9.0

* mark slash_validator(...) as 0.9.0

* switch remaining non-ref objects to ref objects to maybe avoid crashes in CI

* remove unused helpers/debug_state imports
2019-11-06 15:02:06 +00:00
Dustin Brody 63e621c27d
initial 0.9.0 spec sync (#509)
* rename compute_epoch_of_slot(...) to compute_epoch_at_slot(...)

* remove some unnecessary imports; remove some crosslink-related code and tests; complete renaming of compute_epoch_of_slot(...) to compute_epoch_at_slot(...)

* rm more transfer-related code and tests; rm more unnecessary strutils imports

* rm remaining unused imports

* remove useless get_empty_per_epoch_cache(...)/compute_start_slot_of_epoch(...) calls

* rename compute_start_slot_of_epoch(...) to compute_start_slot_at_epoch(...)

* rename ACTIVATION_EXIT_DELAY to MAX_SEED_LOOKAHEAD

* update domain types to 0.9.0

* mark AttesterSlashing, IndexedAttestation, AttestationDataAndCustodyBit, DepositData, BeaconBlockHeader, Fork, integer_squareroot(...), and process_voluntary_exit(...) as 0.9.0

* mark increase_balance(...), decrease_balance(...), get_block_root(...), CheckPoint, Deposit, PendingAttestation, HistoricalBatch, is_active_validator(...), and is_slashable_attestation_data(...) as 0.9.0

* mark compute_activation_exit_epoch(...), bls_verify(...), Validator, get_active_validator_indices(...), get_current_epoch(...), get_total_active_balance(...), and get_previous_epoch(...) as 0.9.0

* mark get_block_root_at_slot(...), ProposerSlashing, get_domain(...), VoluntaryExit, mainnet preset Gwei values, minimal preset max operations, process_block_header(...), and is_slashable_validator(...) as 0.9.0

* mark makeWithdrawalCredentials(...), get_validator_churn_limit(...), get_total_balance(...), is_valid_indexed_attestation(...), bls_aggregate_pubkeys(...), initial genesis value/constants, Attestation, get_randao_mix(...), mainnet preset max operations per block constants, minimal preset Gwei values and time parameters, process_eth1_data(...), get_shuffled_seq(...), compute_committee(...), and process_slots(...) as 0.9.0; partially update get_indexed_attestation(...) to 0.9.0 by removing crosslink refs and associated tests

* mark initiate_validator_exit(...), process_registry_updates(...), BeaconBlock, Eth1Data, compute_domain(...), process_randao(...), process_attester_slashing(...), get_base_reward(...), and process_slot(...) as 0.9.0
2019-10-30 19:41:19 +00:00
Dustin Brody 700e7d921b
update to beacon chain phase 0 spec version 0.8.4; no substantive changes (#494) 2019-10-24 13:36:36 +00:00
Jacek Sieka b7f9d9e4be fix BlsValue inits 2019-10-08 23:52:50 +09:00
Dustin Brody 255980c9f6 fix issue #367 and remove too-small range type usage for ValidatorIndex 2019-10-03 16:41:25 +03:00
Zahary Karadjov 806836714a Add an accidentally forgotten fix 2019-10-01 17:50:13 +03:00
Mamy Ratsimbazafy 6be2e24a8f
Attester slashing - split process_attester_slashing/processAttesterSlashings + tests (#424)
* attester slashing tests - pending #415

* split process_attester_slashing/processAttesterSlashings

* Add logs to attester_slashing

* deactivate bls tests for now (https://github.com/status-im/nim-beacon-chain/issues/429) and cherry-pick from 60f2437dd2
2019-09-10 18:03:06 -04:00
Yuriy Glukhov d4c480ff5f
Eth1 validators (#393) 2019-09-09 18:59:02 +03:00
Mamy Ratsimbazafy 3f446e6383
Attestations EF tests + Proper Real/Opaque BlsValue deserialization (#410)
* Add attestation unit test

* process_attestation doesn't throw exceptions

* Allow SSZ deserialization of both real and invalid signatures

* Add new process_attestation checks - pass all process_attestation tests

* Add sanity check for #361

* Fix SSZ testing after fromBytes/fromSSZBytes changes
2019-09-08 23:33:24 -04:00
Jacek Sieka 5f58af2112
fix ssz list, protocol names (#405) 2019-09-08 16:54:31 +02:00
Mamy Ratsimbazafy 82b9e008d6
Fix genesis blockheader zero signature (supercedes #395) (#400) 2019-09-07 20:56:24 -04:00
Dustin Brody 0c174036a5 update get_attesting_indices(...) to 0.8.3; mark IndexedAttestation, Attestation, PendingAttestation, and get_randao_mix(...) as 0.8.3; rm duplicate/dead code get_unslashed_attesting_indices(...) (#391) 2019-09-06 21:58:38 +02:00
Yuriy Glukhov e3bd4410d8
Revert "Eth1 validators" (#392)
This reverts commit bb8955bdbc.
2019-09-06 13:00:56 +03:00
Yuriy Glukhov bb8955bdbc Eth1 validators 2019-09-06 02:38:38 +02:00
Mamy Ratsimbazafy c11f37e550 V0.8.3 tests part 3 - sanity checks for slots and blocks (#375)
* Add sanity check for slot processing (also impacted by https://github.com/status-im/nim-beacon-chain/issues/373)

* use reportDiff also for all state tests vs EF

* initial sanity checks for blocks - workaround zero signature in block headers: https://github.com/status-im/nim-beacon-chain/issues/374

* Remove generic object variant compare commented code

* Add the one block state transition sanity checks

* generalize blocks test to multiple blocks

* simplify slots test runner

* Add official epoch transitions, sanity blocks, sanity slots to the test suite

* Fix index out-of-bounds in initiate_validator_exit - enable proposer slashings unittest
2019-09-03 18:02:21 +00:00
Mamy Ratsimbazafy 297e9079d4 V0.8.3 tests part2 - BLS + Shuffling + State transition epoch (#372)
* Update BLS fixtures to 0.8.3

* Bump fixtures with shuffling with minimal preset

* Update shuffling tests

* parseTest generic over file format (Json or SSZ)

* Initial crosslink parsing commit for debugging Nim crash

* Workaround https://github.com/status-im/nim-beacon-chain/issues/369

* Crosslink test works for minimal - https://github.com/status-im/nim-beacon-chain/issues/369 is back on mainnet

* Use ref objects to workaround https://github.com/status-im/nim-beacon-chain/issues/369

* Generalize state transition epoch test to all epoch tests

* Fix slashing (potential uint64 overflow in previous spec)

* Add a state debugging macro to deeply inspect the wrong fields

* make reportDiff visible

* Improve the debug state macro for containers
2019-09-03 03:12:09 +00:00
Jacek Sieka 6d798c821f log json errors when loading deposits 2019-09-02 22:14:18 +03:00
Jacek Sieka 9cfbdb5e16 more interop fixes
* fix eth1 interop block hash
* update initialize_beacon_state_from_eth1 to 0.8.3
2019-09-02 22:14:18 +03:00
Jacek Sieka 7b73b40bab interop updates
* add interop launcher scripts
* stick validator_keygen into beacon_node
* fix lmd ghost slot number on missing block
* use mocked eth1data when producing blocks
* use bls public key method for withdrawal credentials
* fix deposit domain
* prefer lowercase for a bunch of toHex
* build simulation binary in data folder to avoid data types confusion
2019-09-02 22:14:18 +03:00
Mamy Ratsimbazafy ca4f29caca v0.8.1 tests refactor (#326)
* Introduce new mocking proc to replace:
- makeFakeValidatorPrivKey
- hackPrivKey
- getNextBeaconProposerIndex
- addBlock
- makeBlock

* Add comments on datastructure unsynced with the spec

* Add merkle tree constructor and initial mocking for deposits (missing merkle proofs)

* [Mock] Implement sparse merkle tree and merkle proof builder

* [Mocking] Genesis deposits

* Add compact_committees_roots init + mock genesis state

* [Tests] Add first deposit test using the new mocking procedures

* [Tests -deposits] add at and over 32 ETH deposit tests

* [Tests - deposits] Add test for validator top-up

* [Tests -deposits] Mention the TODO to test for invalid conditions

* [Tests] Add stub to test "is_valid_genesis_state"

* [Merkle proofs] Implement round-trip checks

* Deactivate roundtrips test

* SSZ - use EF convention for hash_tree_root / hashTreeRoot

* [Tests - Attestation] Attestation mocking + initial test

* Add mocking + 3 new tests for valid attestations + mention future invalid attestation tests

* Add crosslinks test (1 failing to attestations in block being duplicated in state transition)

* Single attestation crosslink test - workaround https://github.com/status-im/nim-beacon-chain/issues/361

* Add test for failed crosslink penalty

* Rebase fixes + add refactored tests to test suite

* justif-finalization helpers first batch

* Add 234 finalization tests

* Fix justif test, Rule I 234 finalization does not happen with sufficient support.
(Also unittest check template does not fail properly in some cases)

* Add tests for all finalization rules

* Properly delete nim-byteutils following c91727e7e5 (diff-7c3613dba5171cb6027c67835dd3b9d4)

* use digest helper for deposit root
2019-08-28 12:07:00 +00:00
Dustin Brody 8665efbbd1 bump beacon chain versions to 0.8.3, which has no relevant changes in this regard (#358) 2019-08-26 11:09:47 +02:00
Dustin Brody cc0429a643
slightly update/rename verify_merkle_branch(...) to 0.8.2 is_valid_merkle_branch(...); mark check_attestation(...), get_active_validator_indices(...), get_seed(...), and BeaconBlockBody as 0.8.2; update compute_domain(...) 0.8.2 by reversing field order (#356) 2019-08-26 07:57:09 +00:00
Jacek Sieka 78f7a4a505 restore network sim finalization
* relax attestation validation when attestation is incoming but make it
more strict when adding to block
* share attestation validation logic between attestation pool and state
transition
* remove a bunch of redundant logging
* fix potential underflow in attestation delay checking
* fix committee used for attestation shard selection when attesting
* fix attestation data construction
2019-08-19 21:05:54 +03:00
Jacek Sieka e639cd95c7
humaneXNum -> shortLog 2019-08-15 18:01:55 +02:00
Zahary Karadjov 4daeebbf7e
Fix the build when json logging is enabled 2019-08-14 17:38:39 +02:00
Dustin Brody 3e587d4667
Justification/finalization amelioration using testnet1 shard count and epoch length (#331)
* working justification, 1 node, 64 validators

* closer to tn1 params: 256 validators

* more debugging output; switch to minimum test case

* working justification and finalization in local network simulation

* fix currently incorrect state transition/attestation test assumption
2019-08-14 08:56:32 +00:00
Dustin Brody 6704749eb0 mark unchanged 0.8.1 as 0.8.2 spec aspects (#336) 2019-08-09 14:08:00 +02:00
Zahary Karadjov 398ea55801
Implement the latest SSZ specification and integrate the official SSZ test suite 2019-08-05 03:18:48 +03:00
Yuriy Glukhov 958e4cd3b4 Getting genesis through web3 2019-07-31 16:39:26 +03:00
Dustin Brody 712038fe21
mark applicable 0.8.0-compliant functions and data structures as 0.8.1 and update a couple of other comments (#321) 2019-07-15 12:57:18 +00:00
Dustin Brody dfa062db7d 0.8.0, continued (#315)
* rename get_genesis_beacon_state(...) to initialize_beacon_state_from_eth1(...); remove unused vestiges get_temporary_block_header(...) and get_empty_block(...); partly align initialize_beacon_state_from_eth1(...) with 0.8.0 version; implement CompactCommittee object type; update process_final_updates(...) to 0.8.0

* mark get_shard_delta(...) as 0.8.0

* mark get_total_active_balance(...) and epoch transition helper functions as 0.8.0

* move FORK_CHOICE_BALANCE_INCREMENT, not in spec anymore, to sole user in fork_choice
2019-07-10 14:23:02 +02:00
Dustin Brody 68be98cafa
More 0.8.0 updates (#311)
* replace BeaconState.finalized_{epoch,root} with BeaconState.finalized_checkpoint; rename get_delayed_activation_exit_epoch(...) to compute_activation_exit_epoch(...) and mark as 0.8.0; update get_churn_limit(...)/get_validator_churn_limit(...) to 0.8.0; update process_registry_updates(...) to 0.8.0

* update process_crosslinks(...) to 0.8.0; mark compute_start_slot_of_epoch(...) and get_committee_count(...) as 0.8.0

* mark Fork, is_slashable_validator(...), and get_beacon_proposer_index(...) as 0.8.0

* rename LATEST_SLASHED_EXIT_LENGTH to EPOCHS_PER_SLASHINGS_VECTOR; update process_slashings(...) to 0.8.0; remove pointless type conversion warning in get_previous_epoch(...)

* convert remaining references to finalized_epoch to finalized_checkpoint.epoch

* update slash_validator(...) to 0.8.0; mark inital value, Gwei, and time constants as 0.8.0; mark hash(...) and processBlockHeader(...) as 0.8.0

* rename WHISTLEBLOWING_REWARD_QUOTIENT to WHISTLEBLOWER_REWARD_QUOTIENT; rename LATEST_ACTIVE_INDEX_ROOTS_LENGTH to EPOCHS_PER_HISTORICAL_VECTOR (randao will also get merged into this); remove get_active_index_root(...); mark time parameter, signature domain types, and max operations per block constants as 0.8.0; update rewards and penalties constants to 0.8.0

* update is_valid_indexed_attestation(...) to 0.8.0; mark process_slot(...) as 0.8.0

* replace BeaconState.{current,previous}_justified_{epoch,root} with BeaconState.{current,previous}_justified_checkpoint
2019-07-05 08:30:05 +00:00