* don't write blocks that get added to database
* don't write states
* write to folders
* add state dumping feature to `ncli_db` to get any known state from the
database
* random fixes
* create dump dir on startup
* don't crash on failure to write dump
* fix a few `uint64` instances being used when indexing arrays - this
should be a compile error but isn't due to compiler bugs
* fix standalone test_block_pool compilation
* add signed block processing in ncli
* reuse cache entry instead of allocating a new one
* allow for small clock disparities when validating blocks
- each BN gets half of its previous validators as inProcess and the other half goes to the respective VC for that BN - using separate data dirs where the keys are copied
- also removed a few command line options which are no longer necessary
- block proposals originating from a VC are propagated from one BN to the rest properly
- other cleanup & moving code back to since it is no longer used elsewhere
These are the 3 most important things in the life of a beacon-node -
finalization checkpoint for security, head + time for seeing that we're
synced and working.
The order given is the expected increasing order that things should be
moving in - finalization lags head, and head lags clock - the structure
helps the user remember, but also see the pattern where time is getting
out of sync with head.
A validator would also want to know when the next action is going to be
(ie next attestation / block production) but that's for another day.
* reorder ssz
* split into hash_trees and ssz_serialization, roughly, for hashing and
IO
* move bitseqs into ssz (from stew)
* clean up imports
* docs, imports
Add SeenTable to avoid continuous attempts to dead peers.
Refactor onSecond.
Block backward sync while forward sync is working.
SyncManager now checks responses according corresponding requests + tests.
SyncManager now watching for not progressing local_head_slot and resets SyncQueue.
* be stricter about SSZ length prefix
* compute zeroHash list at compile time
* remove SSZ schema stuff
* move SSZ navigation to ncli
* cleanup a few leftover openArray uses
* Add ability to reset state of sync manager.
Fix bug when sync got stuck on `zero-point` reset.
Fix bug when sync got stuck when some of the workers waiting for failing one.
* Remove debugging comments and imports.
* Remove not used pendingLock.
- we have a new binary which connects via RPC to the respective BN and has an internal clock - waking it up on every slot
- the BN has a new option called --external-validators and currently in order to have the VC binaries to run we need to pass EXTERNAL_VALIDATORS=yes to make
- factored some code out of beacon_node.nim for easier reuse in validator_api.nim and validator_client.nim
- the VC loads its associated private keys from the datadir for its BN
- most of the validator API calls have been implemented as a stub.
- the VC polls its BN at the start of each epoch - getting a list of all active validators for the current epoch - and then continues to request blocks and sign them with its appropriate validators when necessary
* sync fixes
* fix Status message finalized info
* work around sync starting before initial status exchange
* don't fail block on deposit signature check failure (fixes#989)
* print ForkDigest and Version nicely
* dump incoming blocks
* fix crash when libp2p peer connection is closed
* update chunk size to 16 to work around missing blocks when syncing
* bump libp2p
* bump libp2p
* better deposit skip message
* fix some warnings related to beacon_node splitting; reimplement finalization verification more robustly; improve attestation pool block selection logic
* re-add missing import
* whitelist allowed state transition flags and make rollback/restore naming more consistent
* restore usage of update flags passed into skipAndUpdateState(...) in addition to the potential verifyFinalization flag
* switch rest of rollback -> restore
When replaying state transitions, for the slots that have a block, the
state root is taken from the block. For slots that lack a block, it's
currently calculated using hash_tree_root which is expensive.
Caching the empty slot state roots helps us avoid recalculating this
hash, meaning that for replay, hashes are never calculated. This turns
blocks into fairly lightweight "state-diffs"!
* avoid re-saving state when replaying blocks
* advance empty slots slot-by-slot and save root
* fix sim randomness
* fix sim genesis filename
* introduce `isEpoch` to check if a slot is an epoch slot
* ssz: move ref support outside
Instead of allocating ref's inside SSZ, move it to separate helper:
* makes `ref` allocations explicit
* less magic inside SSZ
* `ref` in nim generally means reference whereas SSZ was loading as
value - if a type indeed used references it would get copies instead of
references to a single value on roundtrip which is unexpected
TODO: EF tests would benefit from some refactoring since they all do the
same thing practically..
Co-authored-by: Zahary Karadjov <zahary@gmail.com>
Please see the newly added 'schlesi-dev' Makefile target.
It demonstrates how the log level can be specified for individual topics.
Additionally, when connecting to testnets like 'schlesi' there will be
two additional log files produced in the working directory:
* json-log.txt
* text-log.txt (in the textblocks format)
* Add "drop by score" ability to PeerPool.
Add tests.
Fix syncmanager queue to start from most fresh data.
* Fix endless cycle at the end of syncing process.
* if we don't have validators, don't consider aggregation work
* if we do have validators, don't aggregate when we're out of sync
* when we do aggregate, use a fresh state, and not one from before
sleeping
* fix warnings by switching from deprecated chronos API addTimer(...) to setTimer(...) and removing especially some unnecessary chronicles and extras imports from test suite modules
* update a couple v0.10.1 spec references to v0.11.1
* crypto: cleanup
* fix several Defect-on-user-input
* make crypto interface more similar to secp
* use `crypto.nim` in all of nbc
* digest: raises
* fix
* vendor
* initial attestation aggregation
* fix usage of committee index, vs index in committee; uniformly set trailing/following distance; document how the only-broadcast-if mechanism works better and what aggregation already happens, not otherwise sufficiently clear; use correct BlockSlot across epoch boundaries
* address inconsistent notion of which slot in past to target for aggregate broadcast; follow 0.11.x aggregate broadcast p2p interface topic
* Fix get_slot_signature(...) call after get_domain(...) change required genesis_validators_root
* mark all spec references which aren't dealt with in other PRs as v0.11.1
* update two more spec refs to v0.11.1
* initial extended validation setup
* flesh out all TODO items for attestation and beaconblock verification
* fix finalization and add chronicles debugging messages
* directly use blockPool.headState rather than pointlessly updating it and document this constraint
* fix logic relating to first-attestation checking; support validating blocks across multiple forks
* initial 0.11.1 spec commit; no test regressions and finalizes in eth2_network_simulation
* with BLS 0.10/0.11 available, stop skipping attester slashing, proposer slashing, and voluntary exist operations fixture tests
* switch param orders to group state.{fork, genesis_validators_root}; bump spec/datatypes spec version for network purposes
* mark attestation construction and broadcast and some minimal/mainnet constants as 0.11.1-compatible; remove phase 1 sharding constants from minimal which don't exist in that preset
* Initial implementation of runtime bls skipping.
Add libnfuzz skipBLSValidation handling, check that it propagates.
* Rename skipBLSValidation -> skipBlsValidation, start skipStateRootValidation
* Replace skipValidation flags with more granular flags.
Also added skipBlockParentRootValidation flag
Mainly replaced with skipBlsValidation but also StateRoot or
BlockParentRootValidation flags where appropriate.
* Adjust interop test to pass when skipping merkle validation.
* Stop skipping validation for mainchain_monitor.
* Remove comment.
* Also skipMerkleValidation for test_beacon_chain_db.
Turns out the DiscV5 code relies heavily on the presence of ENR
records at the moment, so we cannot drive it with ENodes. @kdeme
is working on refactoring that will relax these requirements.
* The bootstrap_nodes.txt file in the node's data dir is now optional
* Log more data on start-up
* Use the latest ENR APIs
* Fix simulation build errors
We no longer discriminate between ENR, MultiAddress or ENode
bootstrap records (all of them are remapped to ENodes).
The discovery loop will stochastically try to reconnect to
accidentally disconnected nodes.
* beacon node code cleanup
* rudimentary error checking on mainnet monitor
* start client even when sending deposit
* work around missing block number exception
* connect to testnet with web3 url
* pretty-print digests in json
* simplify data storage to key-value, tries are not relevant for NBC
* locked-down version of lmdb dependency
* easier to build / maintain on various platforms
* fix slot time navigation, add tests
* skip block proposal if head is more recent already - shouldn't happen
* use correct head when attesting to previous blocks
* log slot start/end processing
* state data cache in block pool
* keep head state around
* more attestation logic in attestation pool
* first fork choice tests (!)
* fix fork choice (it's still likely broken / out of date)
* per honest validator and naïve/simple aggregator attestation specs, move attesting up from halfway to one third of the way through slots
* Update beacon_chain/beacon_node.nim
Co-Authored-By: Jacek Sieka <jacek@status.im>
On your very first connection to each testnet, you'll be asked to
become a validator. Please consult our private repo for a Goerli
Eth1 private key that you can use for deposits.
Other changes:
* Added a simple wrapper ./connect-to-testnet script calling the
nims file in the correct environment. No extension was used to
make the command the same on Unix and Windows.
* Bumped a number of modules with fixes from this week
* `make testnet0` and `make testnet1` will no longer delete your
existing database. This is considered a more appropriate behavior
for testing forward sync.
this is a temporary measure until we figure something better out - as it
stands, we'll advance with empty slots and crash because all validators
are out.
* Move BeaconNode type to its own file (fewer imports)
* disentangle sync protocol/request manager
* fix some old nimisms
* de-fear some logs
* simplify eth1 data production
* add stack tracing to release builds
* drop release compile flag for testnet
* re-enable test_interop based on zcli with 0.9.1 specs and update initialize_beacon_state_from_eth1(...) to 0.9.1
* switch many procs to funcs
* fix import os.nim instead; ospaths is deprecated [Deprecated] warnings
* off-by-one error in the returned range of blocks
* larger request time-outs to deal with non-responsive servers
* fix an unhandled exception when we fail to deliver a response chunk
When the connect_to_testnet script is invoked it will first verify that
the genesis file of the testnet hasn't changed. If it has changed, any
previously created database associated with the testnet will be erased.
To facilitate this, the genesis file of each network is written to the
data folder of the beacon node. The beacon node will refuse to start if
it detects a discrepancy between the data folder and any state snapshot
specified on the command-line.
Since the testnet sharing spec requires us to use SSZ snapshots, the Json
support is now phased out. To help with the transition and to preserve the
functionality of the multinet scripts, the beacon node now supports a CLI
query command that can extract any data from the genesis state. This is
based on new developments in the SSZ navigators.
* rename compute_epoch_of_slot(...) to compute_epoch_at_slot(...)
* remove some unnecessary imports; remove some crosslink-related code and tests; complete renaming of compute_epoch_of_slot(...) to compute_epoch_at_slot(...)
* rm more transfer-related code and tests; rm more unnecessary strutils imports
* rm remaining unused imports
* remove useless get_empty_per_epoch_cache(...)/compute_start_slot_of_epoch(...) calls
* rename compute_start_slot_of_epoch(...) to compute_start_slot_at_epoch(...)
* rename ACTIVATION_EXIT_DELAY to MAX_SEED_LOOKAHEAD
* update domain types to 0.9.0
* mark AttesterSlashing, IndexedAttestation, AttestationDataAndCustodyBit, DepositData, BeaconBlockHeader, Fork, integer_squareroot(...), and process_voluntary_exit(...) as 0.9.0
* mark increase_balance(...), decrease_balance(...), get_block_root(...), CheckPoint, Deposit, PendingAttestation, HistoricalBatch, is_active_validator(...), and is_slashable_attestation_data(...) as 0.9.0
* mark compute_activation_exit_epoch(...), bls_verify(...), Validator, get_active_validator_indices(...), get_current_epoch(...), get_total_active_balance(...), and get_previous_epoch(...) as 0.9.0
* mark get_block_root_at_slot(...), ProposerSlashing, get_domain(...), VoluntaryExit, mainnet preset Gwei values, minimal preset max operations, process_block_header(...), and is_slashable_validator(...) as 0.9.0
* mark makeWithdrawalCredentials(...), get_validator_churn_limit(...), get_total_balance(...), is_valid_indexed_attestation(...), bls_aggregate_pubkeys(...), initial genesis value/constants, Attestation, get_randao_mix(...), mainnet preset max operations per block constants, minimal preset Gwei values and time parameters, process_eth1_data(...), get_shuffled_seq(...), compute_committee(...), and process_slots(...) as 0.9.0; partially update get_indexed_attestation(...) to 0.9.0 by removing crosslink refs and associated tests
* mark initiate_validator_exit(...), process_registry_updates(...), BeaconBlock, Eth1Data, compute_domain(...), process_randao(...), process_attester_slashing(...), get_base_reward(...), and process_slot(...) as 0.9.0
Multi-client testing requires more portable formats, and SSZ is
much better specified than our flavour of Json.
Tools like ncli and zcli can be now used to inspect the contents
of the SSZ files.
* use service/category/process for blockpool logs
Only track fork choice logs in block pool (vs beacon_node)
Reduce verbosity on usual event in block pool
* rework beacon node logs
* log for attestations in blockpool
* log - att pool improvement
* use logScope and topics cf review and discussion
* use 7 letters for beacon_node
[log] report peers at slot start + fix bracket prefix [Block pool] Attestation sent
tersec
Zahary Karadjov
Eugene Kabanov
Jacek Sieka
Mamy Ratsimbazafy
Viktor Kirilov
Ștefan Talpalaru
Diederik Loerakker
kdeme
Joao Gabriel Carvalho
Yuriy Glukhov
Yuriy Glukhov
Mamy Ratsimbazafy