* add column support to beacon chain db
* slight fix
* fix db util
* fix import issues
* gate fulu columns if fulu fork epoch is far future epoch
* fixed all tests file
* fix comment
* add: get_extended_sample_count with test
* drop return
* reviews
* review fix
* fixed
* fix doc
* hooked to all_tests
* rm bin
* add updated test file
* early return, maybe need results?
* refactor function intricacies
* drop columnsCount
* attestation pool support and tests
* REST endpoints changes
* initial ncli and validator client support
* updated tests file
* fixed typos
* review improvements
* remove V1 endpoint
* revert v1 removal
* V2 endpoint version available to pre electra
---------
Co-authored-by: Pedro Miranda <pedro.miranda@nimbus.team>
On `ELECTRA_FORK_EPOCH`, PeerDAS is not yet activated, hence the current
mechanism based on `BlobSidecar` is still in use. With EIP-7688, the
generalized indices of `BeaconBlockBody` get reindexed, changing the
length of the inclusion proof within the `BlobSidecar`. Because network
Req/Resp operations allow responses across fork boundaries, this creates
the need for a `ForkedBlobSidecar` in that layer, same as already done
for `ForkedSignedBeaconBock` for similar reasons.
Note: This PR is only needed if PeerDAS is adopted _after_ EIP-7688.
If PeerDAS is adopted _before_ EIP-7688, a similar PR may be needed for
forked columns. Coincidental `Forked` jank can only be fully avoided if
both features activate at the same epoch, actual changes to blobs aside.
Delaying EIP-7688 for sole purpose of epoch alignemnt is not worth it.
* electra attestation updates
In Electra, we have two attestation formats: on-chain and on-network -
the former combines all committees of a slot in a single committee bit
list.
This PR makes a number of cleanups to move towards fixing this -
attestation packing however still needs to be fixed as it currently
creates attestations with a single committee only which is very
inefficient.
* more attestations in the blocks
* signing and aggregation fixes
* tool fix
* test, import
During sync, sometimes the same block gets encountered and added to
quarantine multiple times. If its parent is already known, quarantine
incorrectly registers it as missing, leading to re-download. This can
be fixed by registering the parent's deepest missing parent recursively.
Also increase the stickiness of `missing`. We only perform 4 attempts
within ~16 seconds before giving up. Very frequently, this is not enough
and there is no progress until sync manager kicks in even on holesky.
Fix the `/eth/v1/beacon/deposit_snapshot` API to produce proper EIP-4881
compatible `DepositTreeSnapshot` responses. The endpoint used to expose
a Nimbus-specific database internal format.
Also fix trusted node sync to consume properly formatted EIP-4881 data
with `--with-deposit-snapshot`, and `--finalized-deposit-tree-snapshot`
beacon node launch option to use the EIP-4881 data. Further ensure that
`ncli_testnet` produces EIP-4881 formatted data for interoperability.
EIP-4881 was never correctly implemented, the `DepositTreeSnapshot`
structure has nothing to do with its actual definition. Reflect that
by renaming the type to a Nimbus-specific `DepositContractSnapshot`,
so that an actual EIP-4881 implementation can use the correct names.
- https://eips.ethereum.org/EIPS/eip-4881#specification
Notably, `DepositTreeSnapshot` contains a compressed sequence in
`finalized`, only containing the minimally required intermediate roots.
That also explains the incorrect REST response reported in #5508.
The non-canonical representation was introduced in #4303 and is also
persisted in the database. We'll have to maintain it for a while.
This PR allows sharing the pubkey data between validators by using a
thread-local cache for pubkey data, netting about a 400mb mem usage
reduction on holesky due to us keeping 3 permanent + several ephemeral
state copies in memory at all times and each state copy holding a full
validator.
The PR also introduces a hash cache for the key which gives ~14% speedup
for a full state `hash_tree_root` - the key makes up for a large part of
the `Validator` htr time.
Finally, the time it takes to copy a state goes down as well from ~80m
ms to ~60, for reasons similar to htr.
We use a `ptr` even if a `ref` could in theory have been used - there is
not much practical benefit to a `ref` (given it's mutable) while a `ptr`
is cheaper and easier to copy (when copying temporary states).
We could go further and cache a cooked pubkey but it turns out this is
quite intrusive - in all the relevant places, we're already using a
cooked key from the immutable validator data so there are no immediate
performance gains of doing so while managing the compressed -> cooked
key mapping would become more difficult - something for a future PR
perhaps.
Co-authored-by: Etan Kissling <etan@status.im>
Etan Kissling
tersec
Agnish Ghosh
Eugene Kabanov
Pedro Miranda
Pedro Miranda
Etan Kissling
Jacek Sieka