The loader has been tested with the presets published by Lighthouse.
You can try connecting to one of their testnets by running:
cd nim-beacon-chain
./connect-to-testnet lighthouse/testnet0
* simplify data storage to key-value, tries are not relevant for NBC
* locked-down version of lmdb dependency
* easier to build / maintain on various platforms
* fix slot time navigation, add tests
* skip block proposal if head is more recent already - shouldn't happen
* use correct head when attesting to previous blocks
* log slot start/end processing
* nbench PoC
* Remove the yaml files from the example scenarios
* update README with current status
* Add an alternative implementation that uses defer
* Forgot to add the old proc body
* slots-processing
* allow benching state_transition failures
* Add Attestations processing (workaround confutils bug:
- https://github.com/status-im/nim-confutils/issues/10
- https://github.com/status-im/nim-confutils/issues/11
- https://github.com/status-im/nim-confutils/issues/12
* Add CLI command in the readme
* Filter report and add notes about CPU cycles
* Report averages
* Add debugecho style time/cycle print
* Report when we skip BLS and state root verification
* Update to 0.9.3
* Generalize scenario parsing
* Support all block processing scenarios
* parallel bench runner PoC
* gitBetter load issues reporting (the load issues were invalid signature and expected to fail)
* state data cache in block pool
* keep head state around
* more attestation logic in attestation pool
* first fork choice tests (!)
* fix fork choice (it's still likely broken / out of date)
* per honest validator and naïve/simple aggregator attestation specs, move attesting up from halfway to one third of the way through slots
* Update beacon_chain/beacon_node.nim
Co-Authored-By: Jacek Sieka <jacek@status.im>
On your very first connection to each testnet, you'll be asked to
become a validator. Please consult our private repo for a Goerli
Eth1 private key that you can use for deposits.
Other changes:
* Added a simple wrapper ./connect-to-testnet script calling the
nims file in the correct environment. No extension was used to
make the command the same on Unix and Windows.
* Bumped a number of modules with fixes from this week
* `make testnet0` and `make testnet1` will no longer delete your
existing database. This is considered a more appropriate behavior
for testing forward sync.
this is a temporary measure until we figure something better out - as it
stands, we'll advance with empty slots and crash because all validators
are out.
* Move BeaconNode type to its own file (fewer imports)
* disentangle sync protocol/request manager
* fix some old nimisms
* de-fear some logs
* simplify eth1 data production
* add stack tracing to release builds
* drop release compile flag for testnet
* re-enable test_interop based on zcli with 0.9.1 specs and update initialize_beacon_state_from_eth1(...) to 0.9.1
* switch many procs to funcs
* fix import os.nim instead; ospaths is deprecated [Deprecated] warnings
* off-by-one error in the returned range of blocks
* larger request time-outs to deal with non-responsive servers
* fix an unhandled exception when we fail to deliver a response chunk
The number of user nodes is now specified with `USER_NODES`.
To make the instructions more stable, the "numeric id" of the user
nodes will be starting from 0 (so you can always use `run_node.sh 0`
to start a user node).
If you specify a node index above the total number of nodes, you'll
launch a node without any validators attached (this is useful for
testing the sync for example).
When the connect_to_testnet script is invoked it will first verify that
the genesis file of the testnet hasn't changed. If it has changed, any
previously created database associated with the testnet will be erased.
To facilitate this, the genesis file of each network is written to the
data folder of the beacon node. The beacon node will refuse to start if
it detects a discrepancy between the data folder and any state snapshot
specified on the command-line.
Since the testnet sharing spec requires us to use SSZ snapshots, the Json
support is now phased out. To help with the transition and to preserve the
functionality of the multinet scripts, the beacon node now supports a CLI
query command that can extract any data from the genesis state. This is
based on new developments in the SSZ navigators.
* update get_seed(...) and get_beacon_proposer_index(...) to 0.9.0, implement compute_proposer_index(...), and render 3 more test fixtures working
* rm stray Crosslink reference which prevented static SSZ tests from building
* remove references to removed tests in attestations test fixture; add minimal-preset block sanity test, plus all but one of mainnet tests for block sanity to transition fixtures
* transition deposit operations fixture to 0.9.0
* mark slash_validator(...) as 0.9.0
* switch remaining non-ref objects to ref objects to maybe avoid crashes in CI
* remove unused helpers/debug_state imports
* rename compute_epoch_of_slot(...) to compute_epoch_at_slot(...)
* remove some unnecessary imports; remove some crosslink-related code and tests; complete renaming of compute_epoch_of_slot(...) to compute_epoch_at_slot(...)
* rm more transfer-related code and tests; rm more unnecessary strutils imports
* rm remaining unused imports
* remove useless get_empty_per_epoch_cache(...)/compute_start_slot_of_epoch(...) calls
* rename compute_start_slot_of_epoch(...) to compute_start_slot_at_epoch(...)
* rename ACTIVATION_EXIT_DELAY to MAX_SEED_LOOKAHEAD
* update domain types to 0.9.0
* mark AttesterSlashing, IndexedAttestation, AttestationDataAndCustodyBit, DepositData, BeaconBlockHeader, Fork, integer_squareroot(...), and process_voluntary_exit(...) as 0.9.0
* mark increase_balance(...), decrease_balance(...), get_block_root(...), CheckPoint, Deposit, PendingAttestation, HistoricalBatch, is_active_validator(...), and is_slashable_attestation_data(...) as 0.9.0
* mark compute_activation_exit_epoch(...), bls_verify(...), Validator, get_active_validator_indices(...), get_current_epoch(...), get_total_active_balance(...), and get_previous_epoch(...) as 0.9.0
* mark get_block_root_at_slot(...), ProposerSlashing, get_domain(...), VoluntaryExit, mainnet preset Gwei values, minimal preset max operations, process_block_header(...), and is_slashable_validator(...) as 0.9.0
* mark makeWithdrawalCredentials(...), get_validator_churn_limit(...), get_total_balance(...), is_valid_indexed_attestation(...), bls_aggregate_pubkeys(...), initial genesis value/constants, Attestation, get_randao_mix(...), mainnet preset max operations per block constants, minimal preset Gwei values and time parameters, process_eth1_data(...), get_shuffled_seq(...), compute_committee(...), and process_slots(...) as 0.9.0; partially update get_indexed_attestation(...) to 0.9.0 by removing crosslink refs and associated tests
* mark initiate_validator_exit(...), process_registry_updates(...), BeaconBlock, Eth1Data, compute_domain(...), process_randao(...), process_attester_slashing(...), get_base_reward(...), and process_slot(...) as 0.9.0
Multi-client testing requires more portable formats, and SSZ is
much better specified than our flavour of Json.
Tools like ncli and zcli can be now used to inspect the contents
of the SSZ files.
* use service/category/process for blockpool logs
Only track fork choice logs in block pool (vs beacon_node)
Reduce verbosity on usual event in block pool
* rework beacon node logs
* log for attestations in blockpool
* log - att pool improvement
* use logScope and topics cf review and discussion
* use 7 letters for beacon_node
[log] report peers at slot start + fix bracket prefix [Block pool] Attestation sent
* Prepare test suite for transfers
* split API process_transfer / processTransfers
* Add range checks on transfer
* Fix invalid transfer conditions
* don't test on windows 64-bit #435
Changes:
* Do not send separate network packets for response codes and msg
len prefixes
* Close streams according to the spec
* Implement more timeouts according to the spec
* Make hello requests during syncing to update our knowledge of
the head block of the other peer.
* Hello is no longer a handshake message
(all handshakes related code was deleted for clarity)
* Deal with the single-parameter inlining defined in the new spec
* add test suite for voluntary exit
* update API to process_voluntary_exit
* Add range check of validator_index for voluntary exits
* Revert to dual single + multiple voluntary exits API + enable in test suite
* no cache or mocking needed
* Add attestation unit test
* process_attestation doesn't throw exceptions
* Allow SSZ deserialization of both real and invalid signatures
* Add new process_attestation checks - pass all process_attestation tests
* Add sanity check for #361
* Fix SSZ testing after fromBytes/fromSSZBytes changes
* Fixed getBeaconBlocks() and getRecentBeaconBlocks() to use BlockPool, not db.
* Got the sync_protocol to compiling state; Removed all obsolete RPC calls
* fix network sim
* mark BeaconState, state list/vector lengths, misc values, get_base_reward(...), verifyStateRoot(...), and process_slot(...) as 0.8.3; update minimal/mainnet config initial values to 0.8.3 by removing GENESIS_FORK_VERSION
* Add sanity check for slot processing (also impacted by https://github.com/status-im/nim-beacon-chain/issues/373)
* use reportDiff also for all state tests vs EF
* initial sanity checks for blocks - workaround zero signature in block headers: https://github.com/status-im/nim-beacon-chain/issues/374
* Remove generic object variant compare commented code
* Add the one block state transition sanity checks
* generalize blocks test to multiple blocks
* simplify slots test runner
* Add official epoch transitions, sanity blocks, sanity slots to the test suite
* Fix index out-of-bounds in initiate_validator_exit - enable proposer slashings unittest
* Update BLS fixtures to 0.8.3
* Bump fixtures with shuffling with minimal preset
* Update shuffling tests
* parseTest generic over file format (Json or SSZ)
* Initial crosslink parsing commit for debugging Nim crash
* Workaround https://github.com/status-im/nim-beacon-chain/issues/369
* Crosslink test works for minimal - https://github.com/status-im/nim-beacon-chain/issues/369 is back on mainnet
* Use ref objects to workaround https://github.com/status-im/nim-beacon-chain/issues/369
* Generalize state transition epoch test to all epoch tests
* Fix slashing (potential uint64 overflow in previous spec)
* Add a state debugging macro to deeply inspect the wrong fields
* make reportDiff visible
* Improve the debug state macro for containers
* add interop launcher scripts
* stick validator_keygen into beacon_node
* fix lmd ghost slot number on missing block
* use mocked eth1data when producing blocks
* use bls public key method for withdrawal credentials
* fix deposit domain
* prefer lowercase for a bunch of toHex
* build simulation binary in data folder to avoid data types confusion
* Introduce new mocking proc to replace:
- makeFakeValidatorPrivKey
- hackPrivKey
- getNextBeaconProposerIndex
- addBlock
- makeBlock
* Add comments on datastructure unsynced with the spec
* Add merkle tree constructor and initial mocking for deposits (missing merkle proofs)
* [Mock] Implement sparse merkle tree and merkle proof builder
* [Mocking] Genesis deposits
* Add compact_committees_roots init + mock genesis state
* [Tests] Add first deposit test using the new mocking procedures
* [Tests -deposits] add at and over 32 ETH deposit tests
* [Tests - deposits] Add test for validator top-up
* [Tests -deposits] Mention the TODO to test for invalid conditions
* [Tests] Add stub to test "is_valid_genesis_state"
* [Merkle proofs] Implement round-trip checks
* Deactivate roundtrips test
* SSZ - use EF convention for hash_tree_root / hashTreeRoot
* [Tests - Attestation] Attestation mocking + initial test
* Add mocking + 3 new tests for valid attestations + mention future invalid attestation tests
* Add crosslinks test (1 failing to attestations in block being duplicated in state transition)
* Single attestation crosslink test - workaround https://github.com/status-im/nim-beacon-chain/issues/361
* Add test for failed crosslink penalty
* Rebase fixes + add refactored tests to test suite
* justif-finalization helpers first batch
* Add 234 finalization tests
* Fix justif test, Rule I 234 finalization does not happen with sufficient support.
(Also unittest check template does not fail properly in some cases)
* Add tests for all finalization rules
* Properly delete nim-byteutils following c91727e7e5 (diff-7c3613dba5171cb6027c67835dd3b9d4)
* use digest helper for deposit root
* relax attestation validation when attestation is incoming but make it
more strict when adding to block
* share attestation validation logic between attestation pool and state
transition
* remove a bunch of redundant logging
* fix potential underflow in attestation delay checking
* fix committee used for attestation shard selection when attesting
* fix attestation data construction
* working justification, 1 node, 64 validators
* closer to tn1 params: 256 validators
* more debugging output; switch to minimum test case
* working justification and finalization in local network simulation
* fix currently incorrect state transition/attestation test assumption
* State sim compiles and run
* Works on libp2p testnet1
* Successfully update the state transition tests
* naive update of attestation pool (failing attstations can arrive in any order)
* Remove an official assert to allow attestation reordering
* include both static and dynamic check for invariants to ensure that get_crosslink_committee(...) can access all committees by ensure that SHARD_COUNT >= SLOTS_PER_EPOCH
* add state_sim to CI, to ensure that it also doesn't break SHARD_COUNT/SLOTS_PER_EPOCH assumptions, along with just in general not breaking
- added a "quitProc" for shutting down the p2pd process, after seeing it
survive an unhandled exception (this required a global var for the
DaemonApi instance)
- Makefile testnet0/1 targets now depend on the "p2pd" binary
- prefixed the testnet command line in the build script message with
"../../env.sh" so it can find the superproject's "p2pd"
* rename get_genesis_beacon_state(...) to initialize_beacon_state_from_eth1(...); remove unused vestiges get_temporary_block_header(...) and get_empty_block(...); partly align initialize_beacon_state_from_eth1(...) with 0.8.0 version; implement CompactCommittee object type; update process_final_updates(...) to 0.8.0
* mark get_shard_delta(...) as 0.8.0
* mark get_total_active_balance(...) and epoch transition helper functions as 0.8.0
* move FORK_CHOICE_BALANCE_INCREMENT, not in spec anymore, to sole user in fork_choice
* remove some low-hanging perf silliness from get_winning_crosslink_and_attesting_indices(...) and switch inner/outer loop nesting order to get 10-15x function speedup for 128 and 512 validator cases by avoiding accidentally quadratic behavior, while keeping function signature unchanged and allowing easy ongoing verification of correctness of optimization
* avoid memory allocation with an openarray rather than seq
* update get_crosslink_deltas(...) to 0.8.0
* move non-spec ValidatorSetDeltaFlags from datatypes.nim to sync_protocol; fix mainnet MIN_ATTESTATION_INCLUSION_DELAY preset; update get_attestation_deltas(...) to 0.8.0; for 512 validators, get 90x speedup for get_attestation_deltas(...) from 179s to 2s
* explain purpose of when false portions, so they can be removed once accomplished
* replace BeaconState.finalized_{epoch,root} with BeaconState.finalized_checkpoint; rename get_delayed_activation_exit_epoch(...) to compute_activation_exit_epoch(...) and mark as 0.8.0; update get_churn_limit(...)/get_validator_churn_limit(...) to 0.8.0; update process_registry_updates(...) to 0.8.0
* update process_crosslinks(...) to 0.8.0; mark compute_start_slot_of_epoch(...) and get_committee_count(...) as 0.8.0
* mark Fork, is_slashable_validator(...), and get_beacon_proposer_index(...) as 0.8.0
* rename LATEST_SLASHED_EXIT_LENGTH to EPOCHS_PER_SLASHINGS_VECTOR; update process_slashings(...) to 0.8.0; remove pointless type conversion warning in get_previous_epoch(...)
* convert remaining references to finalized_epoch to finalized_checkpoint.epoch
* update slash_validator(...) to 0.8.0; mark inital value, Gwei, and time constants as 0.8.0; mark hash(...) and processBlockHeader(...) as 0.8.0
* rename WHISTLEBLOWING_REWARD_QUOTIENT to WHISTLEBLOWER_REWARD_QUOTIENT; rename LATEST_ACTIVE_INDEX_ROOTS_LENGTH to EPOCHS_PER_HISTORICAL_VECTOR (randao will also get merged into this); remove get_active_index_root(...); mark time parameter, signature domain types, and max operations per block constants as 0.8.0; update rewards and penalties constants to 0.8.0
* update is_valid_indexed_attestation(...) to 0.8.0; mark process_slot(...) as 0.8.0
* replace BeaconState.{current,previous}_justified_{epoch,root} with BeaconState.{current,previous}_justified_checkpoint
* rename slot_to_epoch(...) to compute_epoch_of_slot(...)
* rename aggregation_bitfield/custody_bitfield fields to aggregation_bits/custody_bits; rename convert_to_indexed(...) to get_indexed_attestation(...); mark BeaconBlockHeader as 0.8.0; update minimal preset MIN_ATTESTATION_INCLUSION_DELAY/time parameters in general to 0.8
* fix beacon node compilation; it used slightly different capitalizations of slot_to_epoch/slotToEpoch
* mark integer_squareroot(...), Deposit, VoluntaryExit, and Transfer as 0.8.0; rename get_epoch_start_slot(...) to compute_start_slot_of_epoch(...)
* add new Domain alias for uint64; rename bls_domain(...) to compute_domain(...), MAX_INDICES_PER_ATTESTATION to MAX_VALIDATORS_PER_COMMITTEE, and SignatureDomain to DomainType; update get_domain(...) to 0.8.0
* update/mark initiate_validator_exit(...) and Crosslink as 0.8.0; rename BeaconState.latest_block_roots -> BeaconState.block_roots; mark HistoricalBatch as 0.8.0
* migrate attestation pool tests from get_crosslink_committees_at_slot(...) to get_crosslink_committee(...)
* rm obsolete, unused get_crosslink_committees_at_slot_cached(...) and migrate tests/test_state_transition from get_crosslink_committees_at_slot(...) to get_crosslink_committee(...)
* migrate tests/testutil from get_crosslink_committees_at_slot(...) to get_crosslink_committee(...)
* use more pervasive caching infrastructure, initially of compute_committee; remove buggy (and per-index) shuffling to fix research/state_sim, which was noticing validators on multiple shard committees; rm now-unused specific get_attesting_balance_cached
* add some get_active_validator_index caching
* rm obsolete/unused get_attesting_indices_cached
* rm get_crosslink_committees_at_slot(...)
* some more 0.7 changes -- some of which can't be completed pending some data structure updates -- and shard handling changes to offset with epoch start shards
* update Deposit and BeaconBlockHeader to 0.7.0
* update get_total_balance(...) and get_base_reward(...) to 0.7.0
* rename initial value constant BLS_WITHDRAWAL_PREFIX_BYTE to BLS_WITHDRAWAL_PREFIX
* mark minimal presets as entirely migrated to 0.7.0; update process_rewards_and_penalties(...) to 0.7.0
* specify that targeting 0.7.0; adopt 0.7.0 refactoring of bls_domain/get_domain to update get_domain(...) to 0.7.0; mark that mainnet constant presets and block header processing are 0.7.0-consistent; update process_justification_and_finalization to 0.7.0 to fix potential underflow errors
* address mratsim's comment to copy whole spec comment
* 0.7.0 updates which don't change semantics; mostly comment changes
* mark get_attesting_indices, bls_verify, ProposerSlashing, BeaconBlockBody, deposit contract constants, state list length constants, compute_committee, get_crosslink_committee, is_slashable_attestation_data, processAttesterSlashings as 0.7.0; rename BASE_REWARD_QUOTIENT to BASE_REWARD_FACTOR
* complete marking unchanged-in-0.7.0 datatypes as such; a few more have trivial field renamings, etc
* update IndexedAttestation, verify_slashable_attestation/verify_indexed_attestation, and attester slashing processing to 0.6.3
* rm debug scaffolding
* rename Attestation.aggregate_signature -> Attestation.signature; convert various references to AttestationData.slot to get_attestation_slot; implement convert_to_indexed; update checkAttestation and processAttestations to 0.6.3; remove spurious assertion in beacon node related to invalid attestations
* replace 0.5 get_winning_root_and_participants with 0.6 get_winning_crosslink_and_attesting_indices; update process_crosslinks to 0.6.3
* mark both remaining 0.6.0 spec implementations as 0.6.3
* clear out all remaining spec version 0.6.1 refs
* GENESIS_SLOT and GENESIS_EPOCH are 0
* rm 0.5 get_attestation_participants in favor of 0.6 get_attesting_indices
* address mratsim's comment
* time can be equal to genesis
* fix invalid block assertions; those were essentially spurious
* allow toBeaconTime to handle time before genesis (in accordance with now(...) which states time can exist before GENESIS)
* Proto changes to facilitate backward sync
* Update to latest spec types in sync proto
* Use blockpool for more straightforward block headers collection
* Added BlockPool.getRef
* Update beacon_chain/sync_protocol.nim
Co-Authored-By: Jacek Sieka <arnetheduck@gmail.com>
* update process_justification_and_finalization to 0.6.2; mark AttesterSlashing as 0.6.2
* replace get_effective_balance(...) with state.validator_registry[idx].effective_balance; rm get_effective_balance, process_ejections, should_update_validator_registry, update_validator_registry, and update_registry_and_shuffling_data; update get_total_balance to 0.6.2; implement process_registry_updates
* rm exit_validator; implement is_slashable_attestation_data; partly update processAttesterSlashings
* mark HistoricalBatch and Eth1Data as 0.6.2; implement get_shard_delta(...); replace 0.5 finish_epoch_update with 0.6 process_final_updates
* mark increase_balance, decrease_balance, get_delayed_activation_exit_epoch, bls_aggregate_pubkeys, bls_verify_multiple, Attestation, Transfer, slot_to_epoch, Crosslink, get_current_epoch, int_to_bytes*, various constants, processEth1Data, processTransfers, and verifyStateRoot as 0.6.2; rm is_double_vote and is_surround_vote
* mark get_bitfield_bit, verify_bitfield, ProposerSlashing, DepositData, VoluntaryExit, PendingAttestation, Fork, integer_squareroot, get_epoch_start_slot, is_active_validator, generate_seed, some constants to 0.6.2; rename MIN_PENALTY_QUOTIENT to MIN_SLASHING_PENALTY_QUOTIENT
* rm get_previous_total_balance, get_current_epoch_boundary_attestations, get_previous_epoch_boundary_attestations, and get_previous_epoch_matching_head_attestations
* update BeaconState to 0.6.2; simplify legacy get_crosslink_committees_at_slot infrastructure a bit by noting that registry_change is always false; reimplment 0.5 get_crosslink_committees_at_slot in terms of 0.6 get_crosslink_committee
* mark process_deposit(...), get_block_root_at_slot(...), get_block_root(...), Deposit, BeaconBlockHeader, BeaconBlockBody, hash(...), get_active_index_root(...), various constants, get_shard_delta(...), get_epoch_start_shard(...), get_crosslink_committee(...), processRandao(...), processVoluntaryExits(...), cacheState(...) as 0.6.2
* rm removed-since-0.5 split(...), is_power_of_2(...), get_shuffling(...); rm 0.5 versions of get_active_validator_indices and get_epoch_committee_count; add a few tests for integer_squareroot
* mark bytes_to_int(...) and advanceState(...) as 0.6.2
* rm 0.5 get_attesting_indices; update get_attesting_balance to 0.6.2
* another tiny commit to poke AppVeyor to maybe not timeout at connecting to GitHub partway through CI: mark get_churn_limit(...), initiate_validator_exit(...), and Validator as 0.6.2
* mark get_attestation_slot(...), AttestationDataAndCustodyBit, and BeaconBlock as 0.6.2
* implement get_churn_limit from 0.6.1; update initiate_validator_exit and get_previous_epoch to 0.6.1; remove obsoleted/non-spec reduce_balance, replaced by decrease_balance; mark BeaconBlock as 0.6.1
* rename get_block_root to get_block_root_at_slot and introduce epoch-oriented get_block_root; implement get_attestation_slot, get_matching_source_attestations, get_matching_target_attestations, get_matching_head_attestations, 0.6.1 get_attesting_indices, get_unslashed_attesting_indices, get_attestation_deltas, process_rewards_and_penalties; rm get_inactivity_penalty
* update Validator and processVoluntaryExits to 0.6.1; rm obsolete inclusion_slots/inclusion_distances
* rm removed activate_validator; mark DepositData, misc values, Gwei values, Randao processing, state caching as 0.6.1; update GENESIS_SLOT to 64, since it doesn't quite yet work at 0
* mark BeaconBlockHeader as 0.6.1; update BeaconBlockBody to 0.6.1; rename WHISTLEBLOWER_REWARD_QUOTIENT to WHISTLEBLOWING_REWARD_QUOTIENT; update slash_validator to 0.6.1
* implement 0.6.2 is_slashable_validator; update processProposerSlashings to 0.6.2; mark state caching as 0.6.2
* mark get_total_active_balance and process_slashings as 0.6.2; update get_base_reward to 0.6.2
* rm prepare_validator_for_withdrawal, process_exit_queue; mark mainnet misc constants as 0.6.2; update process block header to 0.6.2
* address mratsim's code review comment
* Initial move constants to mainnet preset
* Bump fixtures to include shuffling with minimal preset
* Add minimal preset with shuffling test
* Add minimal preset to the full test suite
* use preset() everywhere
* initial new get_beacon_proposer_index and dependencies (get_shuffled_index, get_shard_delta, get_epoch_start_shard, get_crosslink_committee)
* more 0.6.1 updates
* clean up debugEchos and fix validator keygen to create correct indices
The shell expansion syntax used in the run_node script was not handled properly
by the bash shell shipped with macOS. In particular, it was not able to handle
range expressions involving interpolated vars:
cp foo/{$FIRST_INDEX..$LAST_INDEX} ...
* Change digests to SHA2-256 (from Keccak256)
* Fix sha256 digest initialization - pass shuffling test
* Add comments + remove old shuffling tests
* Remove stale tree hashing tests
* Small toOpenArray optim + notes on in-place hashing
* Revert "Revert "Update test repo""
This reverts commit f385467cd3.
* Revert "Revert "Fix SIGFPE on shuffling for 0""
This reverts commit 226d380f87.
* Revert "Revert "Implement shuffling test (pending typedesc and shuffling algo fix)""
This reverts commit 813cb6fbb8.
* withEth2hash templates now needs ctx.init()
* Use init-update-finish to avoid burnMem
* signed_root -> signing_root
* implement new process_deposit; update timing parameters to 0.6.1; update Deposit to 0.6.1 and remove DepositInput
* update get_active_validator_indices and get_epoch_committee_count to 0.6.1; rm get_current_epoch_committee_count, get_previous_epoch_committee_count, and get_next_epoch_committee_count; bump state_sim default validator count a bit more
* re-introduce 0.5.1-ish get_active_validator_indices, get_epoch_committee_count as scaffolding for still-0.5.1ish shuffling
* track justified and head blocks correctly in presence of blank slots
* fix new block state root
* keep list of viable heads instead of walking children
* disable json bitfield deserializer (needs a corresponding serializer)
* fix handshake best & finalized root
* begin 0.6.0: new get_domain/increase_balance/reduce_balance, BeaconState.validator_balances -> BeaconState.balances, some renamed constants, transaction processing changes, SlashableAttestation field name changes, 0.6.0 get_beacon_proposer_index always uses given state's slot, update tests subrepo
* mark get_bitfield_bit/bls_verify_multiple/stat-list-lengths/is_active_validator/is_surround_vote/slot_to_epoch/int_to_bytes/etc as unchanged in 0.6.0; rm Eth1DataVote/maybe_reset_eth1_period and thus adjust expected tree hash test results
* mark verify_bitfield/bls_verify/deposit-contract/VoluntaryExit/PendingAttestation/Historicalbatch/Fork as 0.6.0; update DOMAIN_BEACON_BLOCK to DOMAIN_BEACON_PROPOSER
* update Crosslink to 0.6.0 (also requires tree hashing test result change, so isolate in individual commit)
* mark verify_merkle_branch/get_delayed_activation_exit_epoch/ProposerSlashing/Attestation/AttestationDataAndCustodyBit/hash/integer_squareroot/get_epoch_start_slot/is_double_vote/get_randao_mix/generate_seed as 0.6.0; update reward and penalty quotients; SlashableAttestation -> IndexedAttestation; rm get_fork_version; ATTESTATION_INCLUSION_REWARD_QUOTIENT -> PROPOSER_REWARD_QUOTIENT
* Load full state test + Add json-serialization for Bitfield
* Implement empty_block_transition test (but test is missing the end state)
* Use the provided empty block instead of mocking one
* Add failing block signing test
* Tests that can't be passed now are now "for information" + indent the "information"/hash given
* fix node.state being used unsafely across async callbacks (fixes#230)
* node.state -> node.stateCache
* fix attestation validation not to use asserts
* re-enable attestation pool tests
* prepare for resizing validator registry
* rm now-superceded shuffling cache (shuffling is only called from get_crosslinks), which was badly architected due to trying to exist in state; rm one more vestige of previous light-client regime (one more to go, from datatypes)
* fix wrong shuffling list size (active validator size, not validator size) to make consistent with 0.5.1 (will be inconsistent with testnet0); fix typo and change defaults in state_sim
* doAssert a couple of constant relationships necessary to avoid underflow; rm non-spec, unused helper function get_new_recent_block_roots
* refactor separate crosslink_committee_cache and winning_root_participants_cache(s) into StateData object; remove last vestige of previous shuffling cache
* separate out caching parts of StateData to new StateCache object
* ssz: avoid memory allocations
* a bit fishy with the 32-item stack.. this should be a smallvector
* digest: avoid another burnmem
* avoid a few allocations by using iterator
* avoid crash on invalid block production (fixes#209)
* fetch blocks every second, roughly
* fix start.sh param order
* run beacon node sim at slower pace
this is the beginning of tracking block-slots more precisely, so we can
the justification epoch slot bug.
* avoid asyncDiscard (swallows assertions)
* fix attestation delay
* fix several state root cache bugs
* introduce workaround for genesis epoch spec issue
* On the testnet site, the first N instead of the last N validators
will be considered reserved for users
* The number of validators is consistently named `totalValidators`
in all commands
* Proper keys are generated by default in validator_keygen
* no need to pass prev_block_root when updating state
* some Slot fixes
* fix `hash_tree_root` for Slot, Epoch
* detect missing hash_tree_root type support
* remove some <0.5 state checks
* have regular slot tick, decide on work to do lazily
* use beacon clock instance that will eventually sync time
* skip work when falling behind
* add one more state cache for justified state (since that's used in
fork choice)
* remove old sync cruft
* single timer per slot, serializing block production and attestation
* ssz: update to 0.5.1:ish
* slightly fewer seq allocations
* still a lot of potential for optimization
* fixes#174
* ssz: avoid reallocating leaves (logN merkle impl)
* update processExits to 0.5.1
* mark process_deposit, get_empty_block as 0.5.1; update get_genesis_beacon_state to 0.5.1; implement processDeposits
* hotfix via https://github.com/ethereum/eth2.0-specs/pull/821
* rm gone-in-0.5.0 Proposal, verifyBlockSignature, and slot check which moved to spec function processBlockHeader
* mark get_attestation_participants and get_epoch_committee_count as 0.5.1; finish updating processAttestations to 0.5.1; add kludgy workaround for bug relating to get_winning_roots_etc using crosslink_data_root as index when we have that as ZERO_HASH for all leading to it confusing attesters on different shards; rm BeaconState.latest_attestations, which splits into previous_epoch_attestations and current_epoch_attestations
* Fix CI due to removed latest_attestations field
* add simple bitfield type (fixes#19)
* fix bit endianess to match spec
* consolidate attestation and block logging
* cruft cleanup
* run optimizer on tests, speeds up build
* begin refactoring 0.5.0 epoch transition at justification and finalization
* update processTransfers to 0.5.0 spec
* update get_crosslink_committees_at_slot to 0.5.0 except for noted workaround for pre-0.5.0 spec bug related to epoch boundary conditions, to be flipped in more coordinated way with other epoch processing changes
* mark ejection processing as 0.5.0
* 0.5.0 epoch update finalization
* rm BEACON_CHAIN_SHARD_NUMBER to complete updating miscellaneous constants to 0.5.0
* mark verify_slashable_attestation as 0.5.0
* update process_attester_slashing to 0.5.0
* update process_slashings(...) to 0.5.0 and mark process_exit_queue as 0.5.0
* refactor epoch processing by implementing 0.5.0's update_registry_and_shuffling_data(...)
With these changes, running a simulation is very close to running
an actual testnet. Some checks have been added in the client to
make sure you are not connecting to an incompatible network (e.g.
a network running with a different number of shards).
* begin 0.5.0 spec update: parent_root -> previous_block_root, BeaconState.justified_epoch -> BeaconState.current_justified_epoch, DOMAIN_PROPOSAL -> DOMAIN_BEACON_BLOCK, temporarily rename BeaconBlockHeader to BeaconBlockHeaderRLP to allow for gradual re-merging without disrupting RLP; mark a few unchanged functions and data types, implement get_temporary_block_header/get_empty_block/should_update_validator_registry/processBlockHeader/cacheState; update a few others
* a dozen or so more trivial mostly comment changes, finding more unchanged parts of 0.5.0
* several more trivial changes; goal is to reduce noise around the upcoming substantial changes (epoch processing, etc)
* remove some redundant state updates
* when attesting late, use correct state / head
* don't send out obsolete attestations
* don't propose obsolete blocks
* remove some more resundant state updates :)
* simplify block logging (experimental)
* document fork choice division
* fix some Slot / Epoch conversion warnings
* keep track of a finalized block
* keep track of all justified blocks
* use naive spec version of LMD ghost
* cache slot number and a few more things in BlockRef
* keep track of the latest vote of each validator
* depend less on the state of node.state (it's a cache, effectively)
To enable it, comment out the 'withLibp2p' line in nim.cfg
The history was squashed in order to remove an accidentally
commited binary file.
Other changes:
* SSZ was adapted to use the common serialization framework
* gossibsup.subscribe is not using async handlers at the moment
and this allowed me to simplify it
* set epoch_boundary_root - chain finalizes!
* fix slotStart to offset GENESIS_SLOT
* work around bug that will be fixed by
https://github.com/ethereum/eth2.0-specs/pull/732
* compile with debug info (there was a GC-related crash in C land)
* fix state db lookup typo
* fix randao reveal slot when proposing blocks
* only store blocks that can be applied to a state
* store state at every epoch boundary (yes, needs pruning!)
* split out state advancement function when there's no block
* default state sim to 0.9 attestation ratio
* processEpoch 5x faster, total 2x faster, with 16k validators
* don't recompute active_validator_indices as much
* remove duplicate process_ejections function
* allow running more or fewer validators
* use deterministic key generation for tests to avoid exhausting system
RNG
* update README with simulator docs
* write the data of each validator to separate file, instead of a big
chainstart.json (makes it easier to run different validator counts)
* prevent balance underflow, per spec implementation note
* verify that permutation shuffing works: add YAML dependency to be used for ingesting test cases; switch from blake2 to keccak256 to match EF test cases; remove inefficient naive-spec-version of shuffling algorithm now that protolambda's can be tested directly
* Complete scan through v0.3.0 spec; remaining known discreptancy is in bls_verify_multiple
* switch remaining slot-misuses of get_active_validator_indices to epochs
* spec workaround
* finish first 0.3.0 pass by getting slow, naive version of bls_verify_multiple running
* implement in-memory block graph
* store tail block in database
* resolve unknown parents by syncing them from peers
* introduce concept of resolved blocks and attestations - those that
follow minimal protocol rules
* update state head lazily
* log more stuff
* shortHash -> shortLog
* start 9/10 beacon nodes by default, last can be started manually
* see also #134
* fix start.sh epoch length
* convert some asserts to doAsserts to keep them in release mode builds; rename get_initial_beacon_state to get_genesis_beacon_state to track spec; switch target spec version to 0.3.0; switch references to penalize_validator to slash_validator/slashValidator to track spec; make some function returns safer by omitting 'return'
* 2x shuffling speedup by hoisting pivot calculations per https://github.com/protolambda/eth2-shuffle
* fix initial attestation pool on reordered attestations
* simplify db layer api
* load head block from database on startup, then load state
* significantly changes database format
* move subscriptions to separate proc's
* implement block replay from historical state
* avoid rescheduling epoch actions on block receipt (why?)
* make sure genesis block is created and used
* relax initial state sim parameters a bit
* move attestation pool to separate file
* combine attestations lazily when needed
* advance state when there's a gap while attesting
* compile beacon node with optimizations - it's tooo slow right now
* log when unable to keep up
* switch to v0.2.0 shuffling scheme
* fix some >80 char lines; add get_total_balance helper function (used in epoch processing, etc); note intended removal of repeat_hash and users thereof; add some int_to_bytes and bytes_to_int helpers; more underflow comments; add Gwei type; adjust generate_seed to include additional entropy source; note egregious inefficiencies
* minor code cleanup
* move some general helper functions to helpers.nim and clarify little-endian comments
* s/slot_included/inclusion_slot/; s/participation_bitfield/aggregation_bitfield/; rearrange some type definitions to match spec order
* a couple more Eth1Data-related spec syncs
* s/CrosslinkRecord/Crosslink/
* {MIN,MAX}_DEPOSIT to {MIN,MAX}_DEPOSIT_AMOUNT; rm SHARD_PERSISTENT_COMMITTEE_CHANGE_PERIOD; per https://github.com/ethereum/eth2.0-specs/issues/378 and 35adc9c61a rm persistent_committees, persistent_committee_reassignments, and ShardReassignmentRecord
* complete switchover from status/ValidatorStatusCodes to status_flags/INITIATED_EXIT/WITHDRAWABLE validator representation
You'll need the latest versions of nim-eth-p2p, nim-serialization
and nim-json-serialization.
Before starting the simulation script, make sure to delete any previous
json files from the simulation folder:
```
rm tests/simulation/*.json
tests/simulation/start.sh
```
This should survive the creation of few blocks before diying with a
block validation error.
* spec updates
* balances move out to separate seq
* bunch of placeholders for proof-of-custody / phase1
* fix inclusion distance adjustment
* modify state in-place in `updateState` (tests spent over 80% time
copying state! now it's down to 25-50)
* document several conditions and conversations
* some renames here and there to follow spec
* initial rough commit of a work/attestation pool
* add kludgy participation_bitfield handling, some error handling, and document future improvments from spec update #47
* rename addAttestation to add for Nim consistency
* spec updates
* random small updates
* ssz no longer sorts by field, fix enum serialization
* rewire block processing a little to avoid a few state copies
* add a state simulation tool that writes out jsons
* ssz: finish implementation
* add object support, simplify implementation
* fix extra round of hashing in tree_hash_root
* ssz: cleanups
* work around Nim range bug for Uint24, cleanups
* some minor renames
* fix some leftover block slot references (replaced by state slot)
* fix some more uint underflows
* update epoch calculations (reward et al)
* now works past 100 epochs at least with empty blocks (very slow)
* use repeat_hash from spec in randao
* add some logging to state processing
* export crypto string converters in spec insulation layer
* enable block signature verification
* ssz: add support for arrays, remove custom data type code
* correctly handle previous and new block
* add trivial block processing tests
* prefer iterative repeat_hash
* state transition mostly done
* handle most specials and slashings
* spec updates
* make several constants uint64 to help minimize casting
* document data type woes - will have to revisit these
* change comment style on fields and constants to make room for better
comments
* add BLSVerify and BLSAddPbkeys facades to insulate spec code from
milagro
* fix proof of possession type
* drop explicitly ordered container fields from ssz - there's an issue
open to sort this out before committing it to the spec
* more work on block processing
* fix some unsigned issues
* fix ssz issue when last chunk is not full
* work around empty const seq compiler bug
* XXX -> TODO
* rename from tree_ssz
* add special cases for some types based on beacon chain code
* add smoke test
* avoids some trivial allocations, but the big one remains - the
temporary
chunk buffer is still heap allocated
* update to handle zero-length buffer case same as latest spec
* first part of spec updating; some constants, structure changes, data type changes from int to uint
* Update beacon_chain/datatypes.nim
Co-Authored-By: tersec <tersec@users.noreply.github.com>
* signature-aggreg-in-block-processing
* Message is a Blake2 hash not simple concat
* Address comments:
- Range change in parent_hashes computation
- Slot not beind module Cycles anymore
- justified_slot part of the hash
Not addressed: using parent.slot_number instead of block.slot_number
Zahary Karadjov
Jacek Sieka
Ștefan Talpalaru
Dustin Brody
Mamy André-Ratsimbazafy
kdeme
cheatfate
Yuriy Glukhov
Yuriy Glukhov
Mamy Ratsimbazafy
Bruno Škvorc
andri lim
Jacek Sieka
cheatfate
Danny Ryan