PBKDF2 based keystore files are required to have `dklen >= 32`.
This patch ensures that keystores not fulfilling that requirement
are properly rejected.
In the ChainDAG, 3 block pointers are kept: genesis, tail and head. This
PR adds one more block pointer: the backfill block which represents the
block that has been backfilled so far.
When doing a checkpoint sync, a random block is given as starting point
- this is the tail block, and we require that the tail block has a
corresponding state.
When backfilling, we end up with blocks without corresponding states,
hence we cannot use `tail` as a backfill pointer - there is no state.
Nonetheless, we need to keep track of where we are in the backfill
process between restarts, such that we can answer GetBeaconBlocksByRange
requests.
This PR adds the basic support for backfill handling - it needs to be
integrated with backfill sync, and the REST API needs to be adjusted to
take advantage of the new backfilled blocks when responding to certain
requests.
Future work will also enable moving the tail in either direction:
* pruning means moving the tail forward in time and removing states
* backwards means recreating past states from genesis, such that
intermediate states are recreated step by step all the way to the tail -
at that point, tail, genesis and backfill will match up.
* backfilling is done when backfill != genesis - later, this will be the
WSS checkpoint instead
There are multiple copies of `bitseqs` (`nim-stew`, `nim-eth`, and
`nim-ssz-serialization`). To avoid confusion, this patch removes the
final remaining reference of a non-`nim-ssz-serialization` copy.
Makes `readSszForkedTrustedSignedBeaconBlock` consistent with
`readSszForkedHashedBeaconState`, adjusting the length check
to verify the same type that is used for length computation,
and using the same formatting.
* BlockId reform
Introduce `BlockId` that helps track a root/slot pair - this prepares
the codebase for backfilling and handling out-of-dag blocks
* move block dag code to separate module
* fix finalised state root in REST event stream
* fix finalised head computation on head update, when starting from
checkpoint
* clean up chaindag init
* revert `epochAncestor` change in introduced in #3144 that would return
an epoch ancestor from the canoncial history instead of the given
history, causing `EpochRef` keys to point to the wrong block
* batch-verify sync messages for a small perf boost
Generally reuses the same structure as attestation and aggregate
verification
* normalize `signatures` and `signature_batch` to use the same pattern
of verification
* normalize parameter names, order etc for signature stuff in general
* avoid calling `blsSign` directly - instead, go through `signatures`
consistently
This removes some dead code from `getSubcommitteePositionsAux` which is
no longer needed since the introduction of `SyncCommitteeCache`.
This also cleans up some formatting, uses `let` instead of `var` where
possible, and uses implicit `pairs` in one case for consistency.
* Introduce slot->BlockRef mapping for finalized chain
The finalized chain is linear, thus we can use a seq to lookup blocks by
slot number.
Here, we introduce such a seq, even though in the future, it should
likely be backed by a database structure instead, or, more likely, a
flat era file with a flat lookup index.
This dramatically speeds up requests by slot, such as those coming from
the REST interface or GetBlocksByRange, as these are currently served by
a linear iteration from head.
* fix REST block requests to not return blocks from an earlier slot when
the given slot is empty
* fix StateId interpretation such that it doesn't treat state roots as
block roots
* don't load full block from database just to return its root
This PR fixes two issues with block publishing:
* Gossip-valid blocks are published before integrating them into the
chain, giving broadcasting a head start, both for rest block and
* Outright invalid blocks from the API that could lead to the descoring
of the node are no longer broadcast
Bonus:
* remove undocumented and duplicated `post_v1_validator_block` JSON-RPC
call
* move quarantine outside of chaindag
The quarantine has been part of the ChainDAG for the longest time, but
this design has a few issues:
* the function in which blocks are verified and added to the dag becomes
reentrant and therefore difficult to reason about - we're currently
using a stateful flag to work around it
* quarantined blocks bypass the processing queue leading to a processing
stampede
* the quarantine flow is unsuitable for orphaned attestations - these
should also should be quarantined eventually
Instead of processing the quarantine inside ChainDAG, this PR moves
re-queueing to `block_processor` which already is responsible for
dealing with follow-up work when a block is added to the dag
This sets the stage for keeping attestations in the quarantine as well.
Also:
* make `BlockError` `{.pure.}`
* avoid use of `ValidationResult` in block clearance (that's for gossip)
* Add some indicators to help fixing issue.
* Bump presto to help debugging.
* Fix compilation problems in presto.
* Fix SIGSEGV.
* Bump latest changes in chronos and presto.
Fix rare cases in validator_client.
* Use proper commits for chronos and presto.
Used during initial development of the spec, `nbench` has fallen behind
and by and large been superceded by `block_sim`, `state_sim` and
`ncli_db bench`.
Some `LightClientUpdate` documentation of the spec is meant to span
multiple fields but was attached to a single, incorrect field in Nimbus.
This patch moves affected documentation to correct locations.
This allows `blockchain_dag`'s `withState` template to be called more
than once in a single function. This led to a compilation error before
because the injected variables and functions shared the same scope.
Validator clients such as Vouch can be configured to work with multiple
beacon nodes simultaneously. In this configuration, the validator client
will try to broadcast the gossip messages through each of the connected
beacon nodes which may lead to a situation where some of the nodes see a
message arriving from the network before it arrives through the REST API.
This should not be considered an error and the beacon node should still
broadcast the message as the intented purpose of the Vouch strategy is
to ensure that the message will reach as many peers as possible.
Turning uncompressed pubkeys into cooked ones is fast, but unnecessary -
this should avoid a little work for every signature validation we do by
pre-loading them at startup.
* add EF fork choice tests to CI
* checkpoints
* compilation fixes and add test to preset dependent suite
* support longpaths on Windows CI
* skip minimal tests (long paths issue + impl detals tested)
* fix stackoverflow on some platforms
* rebase on top of https://github.com/status-im/nimbus-eth2/pull/3054
* fix stack usage
Renames and cleanups split out from the validator monitoring branch, so
as to reduce conflict area vs other PR:s
* add constants for expected message timing
* name validators after the messages they validate, mostly, to make
grepping easier
* unify field naming of EpochInfo across forks to make cross-fork code
easier
Currently, we don't have a good answer to the question "are we synced
yet" - the sync manager syncs based on the peers it's connected to, but
just because some peer looks like it should be synced from doesn't mean
we're out of sync.
Instead, we use a very silly time-based heuristic - the problem with
that is that the network can go into a rut where nobody produces blocks
- better heuristics would be needed here, but in the meantime, a command
line option can get us out of a tight spot - this PR places such an
option in the client, in the unlikely event it should be needed (most
likely in a testnet).
* ncli_db: add putState, putBlock
These tools allow modifying an existing nimbus database for the purpose
of recovery or reorg, moving the head, tail and genesis to arbitrary
points.
* remove potentially expensive `putState` in `BeaconStateDB`
* introduce `latest_block_root` which computes the root of the latest
applied block from the `latest_block_header` field (instead of passing
it in separately)
* avoid some unnecessary BeaconState copies during init
* discover https://github.com/nim-lang/Nim/issues/19094
* prefer `HashedBeaconState` in a few places to avoid recomputing state
root
* fetch latest block root from state when creating blocks
* harden `get_beacon_proposer_index` against invalid slots and document
* move random spec function tests to `test_spec.nim`
* avoid unnecessary state root computation before block proposal
Etan Kissling
Jacek Sieka
tersec
Eugene Kabanov
Zahary Karadjov
Mamy Ratsimbazafy