Commit Graph

133 Commits

Author SHA1 Message Date
tersec ea59f2ecf2
comments-only changes: update a bunch of v0.10.1 spec references to v0.11.0 and explain rationale for extended validation design in code (#847) 2020-04-01 11:41:39 +00:00
tersec cd388bc9bb
extended validation (#812)
* initial extended validation setup

* flesh out all TODO items for attestation and beaconblock verification

* fix finalization and add chronicles debugging messages

* directly use blockPool.headState rather than pointlessly updating it and document this constraint

* fix logic relating to first-attestation checking; support validating blocks across multiple forks
2020-03-31 18:39:02 +00:00
Zahary Karadjov 1d78a43724 Don't return Option[BlockRef] because BlockRef is a ref type 2020-03-22 19:00:09 +02:00
Zahary Karadjov adcec61081 Initial implementation of a JSON-RPC service 2020-03-22 19:00:09 +02:00
tersec a327c82820 Revert "finalizing state_transition (#792)"
This reverts commit 91d75bdfdd.
2020-03-10 16:44:42 +00:00
Joao Gabriel Carvalho 91d75bdfdd
finalizing state_transition (#792) 2020-03-10 12:19:00 +01:00
Jacek Sieka 9a3db7a81f
[WIP] Fake bls at runtime (#735)
* Initial implementation of runtime bls skipping.

Add libnfuzz skipBLSValidation handling, check that it propagates.

* Rename skipBLSValidation -> skipBlsValidation, start skipStateRootValidation
* Replace skipValidation flags with more granular flags.

Also added skipBlockParentRootValidation flag
Mainly replaced with skipBlsValidation but also StateRoot or
BlockParentRootValidation flags where appropriate.

* Adjust interop test to pass when skipping merkle validation.
* Stop skipping validation for mainchain_monitor.
* Remove comment.
* Also skipMerkleValidation for test_beacon_chain_db.
2020-03-05 13:52:10 +01:00
Joao Gabriel Carvalho 91f87b55b4
SignedBeaconBlock in state_transition (#773)
* using SignedBeaconBlock in state_transition
2020-02-29 16:15:44 +01:00
Dustin Brody 6c87af9aab split attestation beacon topic per honest validator spec 2020-02-21 19:44:03 +00:00
Zahary Karadjov ee7c2c9dff Unify the bootstrap nodes handling code
We no longer discriminate between ENR, MultiAddress or ENode
bootstrap records (all of them are remapped to ENodes).

The discovery loop will stochastically try to reconnect to
accidentally disconnected nodes.
2020-02-18 12:53:49 +02:00
Jacek Sieka 521b0ed6ba
Clean 20200205 (#729)
* beacon node code cleanup
* rudimentary error checking on mainnet monitor
* start client even when sending deposit
* work around missing block number exception
* connect to testnet with web3 url
* pretty-print digests in json
2020-02-07 08:13:38 +01:00
Jacek Sieka 7efd113a75 store empty slot states also 2020-02-05 19:24:07 +00:00
Jacek Sieka fb9c4fabf4 fix state rewind
* rewind fast path comparison was not taking skipped slots into account
properly
* less messy blockref creation
2020-02-05 19:24:07 +00:00
Mamy Ratsimbazafy 1ba2cebd85 v0.10.1 (test vectors + unchanged parts) (#696)
* update to 0.10.1

* SSZ Generic and nbench uses the v0.10.1 fixtures

* Tests + spec links: v0.10.0 -> v0.10.1

* Add v0.10.1 TODO in get_latest_attesting_balance (forkchoice)

* SSZ Bytes are now ByteList

* Remove nim-result submodules that was leftover/added by mistake in the branch
2020-01-27 10:56:32 +00:00
Jacek Sieka d9c98b15c8 fix block pool finalization point init 2020-01-23 14:34:58 +02:00
Jacek Sieka 95437e103a disable state pruning
* fix crash when state root is present but state is missing
* fix state root removal when state is removed
* fix block pool initialization which needs tail state
* remove tail block pruning
  * incomplete - fork states are not pruned
  * incomplete - fork blocks are not pruned
  * incomplete - empty slot states are not pruned
  * unknown - tail/finalized block on empty slot might be incorrect
2020-01-22 19:45:38 +02:00
Jacek Sieka 6cfa28e1f5 fix finalization cleanup
* add block pool finalization test
2020-01-21 15:53:53 +02:00
Jacek Sieka 7a8054d36d clean up block pool
* remove BlockPool.blocksBySlot (unused)
* simplify head pruning condition
* add head list smoke tests
* additional logging
2020-01-21 00:54:23 +02:00
Jacek Sieka 3439763a9e fix beacon node init
* update forkversion correctly
* ensure supplied state snapshot starts at genesis - we don't support
anything else
* more database sanity checks
2020-01-17 21:26:28 +00:00
Jacek Sieka 860be026e1 fix block pool init head selection
the head state is not necessarily saved in the database, so we need to
make sure we update things to the correct place
2020-01-15 12:17:49 +00:00
Dustin Brody 8eaebf7163 update to spec version 0.10.0 2020-01-14 18:17:16 +00:00
Jacek Sieka 148527c716 slot fixes
* fix slot time navigation, add tests
* skip block proposal if head is more recent already - shouldn't happen
* use correct head when attesting to previous blocks
* log slot start/end processing
2019-12-23 17:56:56 +00:00
Jacek Sieka b994da78a7 Move some attestation/block logic out of beacon node
* state data cache in block pool
* keep head state around
* more attestation logic in attestation pool
* first fork choice tests (!)
* fix fork choice (it's still likely broken / out of date)
2019-12-19 16:17:45 +00:00
Dustin Brody 27bf8908a9 mark a couple dozen functions and data structures as 0.9.3; fix TRACE-level compiling; stop downloading 0.9.2 test vectors; update cosmetic/display spec versions in test suite 2019-12-17 16:51:46 +00:00
Dustin Brody c824416f56 initial 0.9.3 spec update 2019-12-17 00:06:03 +02:00
Jacek Sieka c66ca39cee update heads when adding resolved blocks 2019-12-16 12:18:08 +02:00
Jacek Sieka 4e85b50cb6 update head while syncing 2019-12-16 12:18:08 +02:00
Zahary Karadjov a641331448
Reduce the sync-related tracing in the testnets [skip ci] 2019-12-03 13:07:43 +02:00
Jacek Sieka 2a728d9468 fix stack overflow in isAncestor recursion 2019-11-28 17:36:07 +02:00
Jacek Sieka c98a0d28ae add workaround for stack overflow on block add 2019-11-28 17:36:07 +02:00
Zahary Karadjov 8c3ea1cabf
Handle missing blocks better in block_pool.getBlockRange 2019-11-27 13:45:34 +02:00
Zahary Karadjov 897176761d
even more tracing for testnet1 [skip ci] 2019-11-27 11:21:45 +02:00
Zahary Karadjov 1099548775
Restore the number of simulation nodes to 4; More block pool tracing [skip ci] 2019-11-26 19:02:56 +02:00
Zahary Karadjov 843c461af6
Remove some debugging output 2019-11-26 02:47:41 +02:00
Zahary Karadjov 8b89bbe391 Fix off-by-one error in syncing 2019-11-26 02:45:25 +02:00
Zahary Karadjov 8de4d38e4e More tracing 2019-11-26 02:45:25 +02:00
Zahary Karadjov 258f6f6334 detailed tracing in block_pool.getBlockRange 2019-11-26 02:45:25 +02:00
Yuriy Glukhov 777b3f4e29 State/block pruning 2019-11-26 02:45:25 +02:00
Jacek Sieka 76d3e74b02
fix head update message 2019-11-25 18:50:45 +01:00
Dustin Brody f5b8931fad switch assert(...) to doAssert(...) and minor cleanup 2019-11-22 16:39:05 +00:00
Dustin Brody c0f009908c
re-enable test_interop based on zcli with 0.9.1 specs and update initialize_beacon_state_from_eth1(...) to 0.9.1 (#569)
* re-enable test_interop based on zcli with 0.9.1 specs and update initialize_beacon_state_from_eth1(...) to 0.9.1

* switch many procs to funcs

* fix import os.nim instead; ospaths is deprecated [Deprecated] warnings
2019-11-21 09:15:10 +00:00
Zahary Karadjov 5a93e50b5e Sync-related fixes:
* off-by-one error in the returned range of blocks
* larger request time-outs to deal with non-responsive servers
* fix an unhandled exception when we fail to deliver a response chunk
2019-11-12 23:43:38 +00:00
Dustin Brody 63e621c27d
initial 0.9.0 spec sync (#509)
* rename compute_epoch_of_slot(...) to compute_epoch_at_slot(...)

* remove some unnecessary imports; remove some crosslink-related code and tests; complete renaming of compute_epoch_of_slot(...) to compute_epoch_at_slot(...)

* rm more transfer-related code and tests; rm more unnecessary strutils imports

* rm remaining unused imports

* remove useless get_empty_per_epoch_cache(...)/compute_start_slot_of_epoch(...) calls

* rename compute_start_slot_of_epoch(...) to compute_start_slot_at_epoch(...)

* rename ACTIVATION_EXIT_DELAY to MAX_SEED_LOOKAHEAD

* update domain types to 0.9.0

* mark AttesterSlashing, IndexedAttestation, AttestationDataAndCustodyBit, DepositData, BeaconBlockHeader, Fork, integer_squareroot(...), and process_voluntary_exit(...) as 0.9.0

* mark increase_balance(...), decrease_balance(...), get_block_root(...), CheckPoint, Deposit, PendingAttestation, HistoricalBatch, is_active_validator(...), and is_slashable_attestation_data(...) as 0.9.0

* mark compute_activation_exit_epoch(...), bls_verify(...), Validator, get_active_validator_indices(...), get_current_epoch(...), get_total_active_balance(...), and get_previous_epoch(...) as 0.9.0

* mark get_block_root_at_slot(...), ProposerSlashing, get_domain(...), VoluntaryExit, mainnet preset Gwei values, minimal preset max operations, process_block_header(...), and is_slashable_validator(...) as 0.9.0

* mark makeWithdrawalCredentials(...), get_validator_churn_limit(...), get_total_balance(...), is_valid_indexed_attestation(...), bls_aggregate_pubkeys(...), initial genesis value/constants, Attestation, get_randao_mix(...), mainnet preset max operations per block constants, minimal preset Gwei values and time parameters, process_eth1_data(...), get_shuffled_seq(...), compute_committee(...), and process_slots(...) as 0.9.0; partially update get_indexed_attestation(...) to 0.9.0 by removing crosslink refs and associated tests

* mark initiate_validator_exit(...), process_registry_updates(...), BeaconBlock, Eth1Data, compute_domain(...), process_randao(...), process_attester_slashing(...), get_base_reward(...), and process_slot(...) as 0.9.0
2019-10-30 19:41:19 +00:00
Jacek Sieka ee5d1c1467
minimize imports, specially for spec - cuts 2/3 of ncli build time (#500)
* minimize imports, specially for spec - cuts 2/3 of ncli build time
* ptr_arith->ptrops
2019-10-25 12:59:56 +02:00
Dustin Brody 43f353e665 implement beacon_reorgs_total metric (#491) 2019-10-22 13:57:34 +02:00
Mamy Ratsimbazafy 0b68f3dc63
[Interop branch] Log improvements - part 1 (#439)
* use service/category/process for blockpool logs
Only track fork choice logs in block pool (vs beacon_node)
Reduce verbosity on usual event in block pool

* rework beacon node logs

* log for attestations in blockpool

* log - att pool improvement

* use logScope and topics cf review and discussion

* use 7 letters for beacon_node

[log] report peers at slot start + fix bracket prefix [Block pool] Attestation sent
2019-09-23 11:35:41 +02:00
Zahary Karadjov cdff79ec6d More sync fixes
* Fix IncompleteData issues brought by the new spec-compliant stream closing
* Fix logic errors in the sync algorithms
2019-09-10 09:49:54 -04:00
Jacek Sieka 7b73b40bab interop updates
* add interop launcher scripts
* stick validator_keygen into beacon_node
* fix lmd ghost slot number on missing block
* use mocked eth1data when producing blocks
* use bls public key method for withdrawal credentials
* fix deposit domain
* prefer lowercase for a bunch of toHex
* build simulation binary in data folder to avoid data types confusion
2019-09-02 22:14:18 +03:00
Jacek Sieka 225465578c fix startup time casts, log fixes 2019-08-19 21:05:54 +03:00
Jacek Sieka e639cd95c7
humaneXNum -> shortLog 2019-08-15 18:01:55 +02:00
Dustin Brody 3e587d4667
Justification/finalization amelioration using testnet1 shard count and epoch length (#331)
* working justification, 1 node, 64 validators

* closer to tn1 params: 256 validators

* more debugging output; switch to minimum test case

* working justification and finalization in local network simulation

* fix currently incorrect state transition/attestation test assumption
2019-08-14 08:56:32 +00:00
Mamy Ratsimbazafy d1d5497233
v0.7.x-v0.8.1 state transition refactor part2 (#322)
* State sim compiles and run

* Works on libp2p testnet1

* Successfully update the state transition tests

* naive update of attestation pool (failing attstations can arrive in any order)

* Remove an official assert to allow attestation reordering
2019-07-15 23:10:40 +02:00
Dustin Brody 68be98cafa
More 0.8.0 updates (#311)
* replace BeaconState.finalized_{epoch,root} with BeaconState.finalized_checkpoint; rename get_delayed_activation_exit_epoch(...) to compute_activation_exit_epoch(...) and mark as 0.8.0; update get_churn_limit(...)/get_validator_churn_limit(...) to 0.8.0; update process_registry_updates(...) to 0.8.0

* update process_crosslinks(...) to 0.8.0; mark compute_start_slot_of_epoch(...) and get_committee_count(...) as 0.8.0

* mark Fork, is_slashable_validator(...), and get_beacon_proposer_index(...) as 0.8.0

* rename LATEST_SLASHED_EXIT_LENGTH to EPOCHS_PER_SLASHINGS_VECTOR; update process_slashings(...) to 0.8.0; remove pointless type conversion warning in get_previous_epoch(...)

* convert remaining references to finalized_epoch to finalized_checkpoint.epoch

* update slash_validator(...) to 0.8.0; mark inital value, Gwei, and time constants as 0.8.0; mark hash(...) and processBlockHeader(...) as 0.8.0

* rename WHISTLEBLOWING_REWARD_QUOTIENT to WHISTLEBLOWER_REWARD_QUOTIENT; rename LATEST_ACTIVE_INDEX_ROOTS_LENGTH to EPOCHS_PER_HISTORICAL_VECTOR (randao will also get merged into this); remove get_active_index_root(...); mark time parameter, signature domain types, and max operations per block constants as 0.8.0; update rewards and penalties constants to 0.8.0

* update is_valid_indexed_attestation(...) to 0.8.0; mark process_slot(...) as 0.8.0

* replace BeaconState.{current,previous}_justified_{epoch,root} with BeaconState.{current,previous}_justified_checkpoint
2019-07-05 08:30:05 +00:00
Dustin Brody fb601bd5b8 rename BeaconState.validator_registry to BeaconState.validators 2019-07-01 14:44:57 +03:00
Dustin Brody e4321dc4ed Initial frozen phase 0/v0.8.0 spec (#305)
* rename slot_to_epoch(...) to compute_epoch_of_slot(...)

* rename aggregation_bitfield/custody_bitfield fields to aggregation_bits/custody_bits; rename convert_to_indexed(...) to get_indexed_attestation(...); mark BeaconBlockHeader as 0.8.0; update minimal preset MIN_ATTESTATION_INCLUSION_DELAY/time parameters in general to 0.8

* fix beacon node compilation; it used slightly different capitalizations of slot_to_epoch/slotToEpoch

* mark integer_squareroot(...), Deposit, VoluntaryExit, and Transfer as 0.8.0; rename get_epoch_start_slot(...) to compute_start_slot_of_epoch(...)

* add new Domain alias for uint64; rename bls_domain(...) to compute_domain(...), MAX_INDICES_PER_ATTESTATION to MAX_VALIDATORS_PER_COMMITTEE, and SignatureDomain to DomainType; update get_domain(...) to 0.8.0

* update/mark initiate_validator_exit(...) and Crosslink as 0.8.0; rename BeaconState.latest_block_roots -> BeaconState.block_roots; mark HistoricalBatch as 0.8.0
2019-07-01 09:53:42 +02:00
Dustin Brody 2a6e64d3f9
0.7.0 updates including semantic changes (#285)
* update get_previous_epoch(...)

* update BeaconBlock to 0.7.0 by renaming previous_block_root field to parent_root
2019-06-14 13:50:47 +00:00
Yuriy Glukhov 10c7920b27 Proto changes to facilitate backward sync (#271)
* Proto changes to facilitate backward sync

* Update to latest spec types in sync proto

* Use blockpool for more straightforward block headers collection

* Added BlockPool.getRef

* Update beacon_chain/sync_protocol.nim

Co-Authored-By: Jacek Sieka <arnetheduck@gmail.com>
2019-06-10 07:13:53 -04:00
Dustin Brody d977c96ae1 0.6.1 - signing_root, deposits, validator indices and committee count (#261)
* signed_root -> signing_root

* implement new process_deposit; update timing parameters to 0.6.1; update Deposit to 0.6.1 and remove DepositInput

* update get_active_validator_indices and get_epoch_committee_count to 0.6.1; rm get_current_epoch_committee_count, get_previous_epoch_committee_count, and get_next_epoch_committee_count; bump state_sim default validator count a bit more

* re-introduce 0.5.1-ish get_active_validator_indices, get_epoch_committee_count as scaffolding for still-0.5.1ish shuffling
2019-05-09 14:27:37 +02:00
Jacek Sieka 17fa499755 cache state root a bit more aggressively (#260)
needed to be able to run state sim, too slow otherwise
2019-05-04 14:10:45 +00:00
Jacek Sieka bd4893d2e8 head block selection fixes (#259)
* track justified and head blocks correctly in presence of blank slots
* fix new block state root
* keep list of viable heads instead of walking children
* disable json bitfield deserializer (needs a corresponding serializer)
* fix handshake best & finalized root
2019-05-01 12:19:29 +03:00
Jacek Sieka c7e6d39279 fix statedata fields not being updated correctly (#255) 2019-04-29 11:14:22 +03:00
Yuriy Glukhov 61ec6fa167 Move incoming block from unresolved to pending if it can't be applied (#253)
* Move resolved block from unresolved to pending if it can't be applied

* Renamed unresolved to missing
2019-04-26 10:38:56 -06:00
Yuriy Glukhov ff7adcefbd Persist head block 2019-04-02 15:35:09 +03:00
Zahary Karadjov e0d8bf9e99 Logging in preInit works just fine locally; pushing to test on the CI 2019-03-28 16:54:50 +02:00
Jacek Sieka f9e0418b5b fetch multiple blocks at a time
* avoid crash on invalid block production (fixes #209)
* fetch blocks every second, roughly
* fix start.sh param order
* run beacon node sim at slower pace
2019-03-28 10:42:16 +02:00
Jacek Sieka 3ad6053298 Use block, slot tuple to idenfiy fork
this is the beginning of tracking block-slots more precisely, so we can
the justification epoch slot bug.

* avoid asyncDiscard (swallows assertions)
* fix attestation delay
* fix several state root cache bugs
* introduce workaround for genesis epoch spec issue
2019-03-28 10:32:09 +02:00
Jacek Sieka 81c66fbce0
use signed_root as canonical block root (#211)
* no need to pass prev_block_root when updating state
* some Slot fixes
* fix `hash_tree_root` for Slot, Epoch
* detect missing hash_tree_root type support
* remove some <0.5 state checks
2019-03-26 19:32:35 -06:00
Jacek Sieka e7b36c4389 add beacon clock and rework timing
* have regular slot tick, decide on work to do lazily
* use beacon clock instance that will eventually sync time
* skip work when falling behind
* add one more state cache for justified state (since that's used in
fork choice)
* remove old sync cruft
* single timer per slot, serializing block production and attestation
2019-03-26 13:21:48 +02:00
Yuriy Glukhov e4c10a31c9 Use latest state from the DB in sync. Fixed off by one in block roots (#203) 2019-03-25 14:52:23 -06:00
Jacek Sieka 1b0e67c88c
ssz: update to 0.5.1:ish (#202)
* ssz: update to 0.5.1:ish
* slightly fewer seq allocations
* still a lot of potential for optimization
* fixes #174
* ssz: avoid reallocating leaves (logN merkle impl)
2019-03-25 10:46:31 -06:00
Mamy André-Ratsimbazafy ca49ae9b5a
Report connected peers before fork choice 2019-03-22 12:33:10 +01:00
Yuriy Glukhov 894816fa0c Fixed crash caused by findAncestorBySlot (#196)
closes #193
2019-03-22 11:57:19 +01:00
Dustin Brody f36c2d86dc begin 0.5.0 spec update (#179)
* begin 0.5.0 spec update: parent_root -> previous_block_root, BeaconState.justified_epoch -> BeaconState.current_justified_epoch, DOMAIN_PROPOSAL -> DOMAIN_BEACON_BLOCK, temporarily rename BeaconBlockHeader to BeaconBlockHeaderRLP to allow for gradual re-merging without disrupting RLP; mark a few unchanged functions and data types, implement get_temporary_block_header/get_empty_block/should_update_validator_registry/processBlockHeader/cacheState; update a few others

* a dozen or so more trivial mostly comment changes, finding more unchanged parts of 0.5.0

* several more trivial changes; goal is to reduce noise around the upcoming substantial changes (epoch processing, etc)
2019-03-16 13:52:37 -06:00
Jacek Sieka 1cb8ae9004
state processing fixes (#177)
* remove some redundant state updates
* when attesting late, use correct state / head
* don't send out obsolete attestations
* don't propose obsolete blocks
* remove some more resundant state updates :)
* simplify block logging (experimental)
* document fork choice division
* fix some Slot / Epoch conversion warnings
2019-03-14 07:33:56 -06:00
Ștefan Talpalaru 9ff1eb4ac8
Merge branch 'master' of github.com:status-im/nim-beacon-chain 2019-03-14 00:05:12 +01:00
Ștefan Talpalaru de295619be
assert() -> doAssert() 2019-03-14 00:04:43 +01:00
Jacek Sieka b0f4034060
implement fork choice (#175)
* keep track of a finalized block
* keep track of all justified blocks
* use naive spec version of LMD ghost
* cache slot number and a few more things in BlockRef
* keep track of the latest vote of each validator
* depend less on the state of node.state (it's a cache, effectively)
2019-03-13 16:59:20 -06:00
Dustin Brody 50a0948bc0 Mostly convert to using distinct Slot type (#172) 2019-03-12 16:21:32 -06:00
Mamy André-Ratsimbazafy 6dea18c140
Squashed commit of the following:
commit bed6909f33
Author: Mamy André-Ratsimbazafy <mamy_github@numforge.co>
Date:   Mon Mar 11 12:22:35 2019 +0100

    Cleanup imports

commit 241f62ed8d
Author: Mamy André-Ratsimbazafy <mamy_github@numforge.co>
Date:   Mon Mar 11 12:06:00 2019 +0100

    Group beacon types to prevent recursive imports
2019-03-12 16:03:14 +01:00
Yuriy Glukhov b587455e7b Use BlockPool for slot->blocks mapping (#166) 2019-03-11 09:38:36 -06:00
Jacek Sieka aec31b1126 Updates for finalizing chain! (#162)
* set epoch_boundary_root - chain finalizes!
* fix slotStart to offset GENESIS_SLOT
* work around bug that will be fixed by
https://github.com/ethereum/eth2.0-specs/pull/732
* compile with debug info (there was a GC-related crash in C land)
2019-03-09 04:23:14 +00:00
Jacek Sieka 6bcefc0e42
verify blocks before storing in block pool / database (#158)
* fix state db lookup typo
* fix randao reveal slot when proposing blocks
* only store blocks that can be applied to a state
* store state at every epoch boundary (yes, needs pruning!)
* split out state advancement function when there's no block
* default state sim to 0.9 attestation ratio
2019-03-08 10:40:17 -06:00
Jacek Sieka 125231d321
add initial block pool (#139)
* implement in-memory block graph
* store tail block in database
* resolve unknown parents by syncing them from peers
* introduce concept of resolved blocks and attestations - those that
follow minimal protocol rules
* update state head lazily
* log more stuff
* shortHash -> shortLog
* start 9/10 beacon nodes by default, last can be started manually
* see also #134
* fix start.sh epoch length
2019-02-28 15:21:29 -06:00