Commit Graph

3958 Commits

Author SHA1 Message Date
Ștefan Talpalaru 609c86f056
Grafana: don't interpolate over null values (#3173) 2021-12-08 14:21:48 +01:00
Etan Kissling 38e64b3441 cleanup sync subcommittee accessors
This removes some dead code from `getSubcommitteePositionsAux` which is
no longer needed since the introduction of `SyncCommitteeCache`.
This also cleans up some formatting, uses `let` instead of `var` where
possible, and uses implicit `pairs` in one case for consistency.
2021-12-07 18:17:03 +02:00
tersec aeb90a1585
bump nim-web3 to adjust engine API field names for alpha.5 (#3166) 2021-12-07 14:30:16 +00:00
Etan Kissling 3f6b7ec8f1
fix test results totals (#3168)
Updates `AllTests-mainnet.md` output for `make test -j16` execution.
2021-12-07 13:43:06 +00:00
Jacek Sieka e2a2157370
RPM/DEB for systemd-based systems (#3034)
* RPM/DEB for systemd-based systems

Similar to https://github.com/status-im/nimbus-eth2/pull/2691, this PR
introduces packaging for nimbus beacon node.

Instead of relying on interactive prompts, a default configuration is
included that connects to mainnet and expects there to be an execution
client running with websockets enabled on the same host, on the standard
websocket port.

Should the user need to configure their nimbus after installation (for
example with a different web3 url), the "standard" way of doing so via
`systemctl` edit is recommended.
2021-12-07 15:23:57 +02:00
Jacek Sieka 89d6a1b403
Introduce slot->BlockRef mapping for finalized chain (#3144)
* Introduce slot->BlockRef mapping for finalized chain

The finalized chain is linear, thus we can use a seq to lookup blocks by
slot number.

Here, we introduce such a seq, even though in the future, it should
likely be backed by a database structure instead, or, more likely, a
flat era file with a flat lookup index.

This dramatically speeds up requests by slot, such as those coming from
the REST interface or GetBlocksByRange, as these are currently served by
a linear iteration from head.

* fix REST block requests to not return blocks from an earlier slot when
the given slot is empty
* fix StateId interpretation such that it doesn't treat state roots as
block roots
* don't load full block from database just to return its root
2021-12-06 20:52:35 +02:00
Jacek Sieka 850eece949 Broadcast blocks before integrating in ChainDAG
This PR fixes two issues with block publishing:

* Gossip-valid blocks are published before integrating them into the
chain, giving broadcasting a head start, both for rest block and
* Outright invalid blocks from the API that could lead to the descoring
of the node are no longer broadcast

Bonus:

* remove undocumented and duplicated `post_v1_validator_block` JSON-RPC
call
2021-12-06 17:15:47 +02:00
Jacek Sieka 1a8b7469e3
move quarantine outside of chaindag (#3124)
* move quarantine outside of chaindag

The quarantine has been part of the ChainDAG for the longest time, but
this design has a few issues:

* the function in which blocks are verified and added to the dag becomes
reentrant and therefore difficult to reason about - we're currently
using a stateful flag to work around it
* quarantined blocks bypass the processing queue leading to a processing
stampede
* the quarantine flow is unsuitable for orphaned attestations - these
should also should be quarantined eventually

Instead of processing the quarantine inside ChainDAG, this PR moves
re-queueing to `block_processor` which already is responsible for
dealing with follow-up work when a block is added to the dag

This sets the stage for keeping attestations in the quarantine as well.

Also:

* make `BlockError` `{.pure.}`
* avoid use of `ValidationResult` in block clearance (that's for gossip)
2021-12-06 10:49:01 +01:00
tersec a8c801eddd
fix Altair fork tests in minimal preset (#3163) 2021-12-06 05:56:46 +00:00
tersec 07e15a26fb
non-march=native builds should use SSSE3 instead of SSE3 (#3162) 2021-12-05 18:02:58 +00:00
tersec e6921f808f
cleanups, partly from kintsugi branch (#3161)
* cleanups, partly from kintsugi branch

* re-export shortLog(EthBlock) and preserve exception messages in batchVerify and processBatch
2021-12-05 17:32:41 +00:00
tersec 8d7df05f2e
Revert "Bump chronos and presto. (#3159)" (#3160)
This reverts commit 4c90b82d9f.
2021-12-04 15:52:28 +00:00
Eugene Kabanov 4c90b82d9f
Bump chronos and presto. (#3159)
* Add some indicators to help fixing issue.

* Bump presto to help debugging.

* Fix compilation problems in presto.

* Fix SIGSEGV.

* Bump latest changes in chronos and presto.
Fix rare cases in validator_client.

* Use proper commits for chronos and presto.
2021-12-04 14:26:16 +00:00
tersec 4378f3f096
almost all remaining ethereum/{eth2.0-specs -> consensus-specs} (#3158) 2021-12-03 20:01:13 +00:00
Zahary Karadjov f0f9735955
Version 1.5.5 2021-12-03 20:04:38 +02:00
tersec cc51f3fd12
v1.1.{5 -> 6} phase 0 and altair spec URL updates (#3157) 2021-12-03 17:40:23 +00:00
zah 74c63ed740
More efficient implementation of the 'POST beacon_committee_subscriptions' API (#3153) 2021-12-03 17:04:58 +02:00
Zahary Karadjov 6fddff524c Switch back to WebSocket URLs in the Eth1Monitor
The HTTP support is not stable enough yet.
2021-12-03 17:04:29 +02:00
Jacek Sieka cb3f1fd4fc
remove nbench (#3152)
Used during initial development of the spec, `nbench` has fallen behind
and by and large been superceded by `block_sim`, `state_sim` and
`ncli_db bench`.
2021-12-03 15:46:56 +01:00
nbc-bump-bot[bot] fa2cf028e2
auto-bump nim-libp2p (#3114)
Co-authored-by: = <Menduist@users.noreply.github.com>
2021-12-03 15:03:34 +01:00
Jacek Sieka 065d72fb15 move head update to storeBlock
when blocks are supplied via rest, this ensures the newly posted head is
chosen
2021-12-03 11:18:37 +02:00
Etan Kissling 5e9625c1be extend `makeTestBlocks` for sync aggregates
This extends the `makeTestBlocks` function used in tests with a new
parameter `syncCommitteeRatio` to control whether the produced blocks
should be signed by the validators assigned to the sync committee.
A similar parameter already exists to configure whether attestations
for the test blocks should be produced.
2021-12-01 20:10:33 +02:00
cheatfate 40a9da385d Fix for /eth/v1/node/identity call missing `syncnets` parameter. 2021-12-01 18:39:43 +02:00
sacha a1db321f0a
Update eth1.md
remover warning
2021-12-01 15:23:51 +01:00
tersec 8da2f5307d
remove obsolete amphora interop documentation (#3145) 2021-12-01 14:01:24 +01:00
tersec 61fb458f89
use v1.1.6 test vectors (#3146) 2021-12-01 12:55:42 +00:00
Jacek Sieka aa1dea03cd
speed up gossip and sync block validation (#3143)
* avoid recomputing hash for block signature check
* check block slot match before hitting the database
2021-12-01 10:52:40 +01:00
Ștefan Talpalaru 5e50297f95
bump nimbus-build-system (#3140) 2021-12-01 05:31:22 +01:00
sacha 335f7c1072
Update infura-guide.md
revert endpoint advice
2021-11-30 18:37:54 +01:00
cheatfate b3ee5d67bd Remove nimbus_signing_process. 2021-11-30 16:48:36 +02:00
rasberrylasagna 13280f230c
[FEAT] Updating some infura doc (#3137)
* [FEAT] updating some infura doc

* [FIX] Suggested changes

Co-authored-by: sacha <sacha.saint-leger@ethereum.org>

Co-authored-by: sacha <sacha.saint-leger@ethereum.org>
2021-11-30 12:54:14 +01:00
Eugene Kabanov e62c7c7c37
Remote signing client/server. (#3077) 2021-11-30 03:20:21 +02:00
zah 3aa804035f
Allow /api/eth/v1/validator/duties/sync/{epoch} to be called for epochs in the next sync committee period (#3133) 2021-11-30 03:14:31 +02:00
Zahary Karadjov 872b88db50
Hotfix: non-crashing version of nim-json-rpc/news 2021-11-30 02:51:35 +02:00
zah bc6a6a8b8f
Switch back to the news package in nim-json-rpc (#3136) 2021-11-29 21:47:31 +01:00
Etan Kissling bdf2d3cc27 fix `LightClientUpdate` documentation
Some `LightClientUpdate` documentation of the spec is meant to span
multiple fields but was attached to a single, incorrect field in Nimbus.
This patch moves affected documentation to correct locations.
2021-11-29 15:27:53 +02:00
Etan Kissling eb777a6c8b allow `withState` to be called multiple times
This allows `blockchain_dag`'s `withState` template to be called more
than once in a single function. This led to a compilation error before
because the injected variables and functions shared the same scope.
2021-11-29 15:24:12 +02:00
Zahary Karadjov 4570f6e4ec
Allow broadcasting "ignored" gossip messages
Validator clients such as Vouch can be configured to work with multiple
beacon nodes simultaneously. In this configuration, the validator client
will try to broadcast the gossip messages through each of the connected
beacon nodes which may lead to a situation where some of the nodes see a
message arriving from the network before it arrives through the REST API.

This should not be considered an error and the beacon node should still
broadcast the message as the intented purpose of the Vouch strategy is
to ensure that the message will reach as many peers as possible.
2021-11-28 01:54:36 +02:00
Zahary Karadjov 7902e7684c Sync with Eth1 even when there are no validators attached 2021-11-27 18:43:01 +02:00
Zahary Karadjov ef1de66316 Add polling support in the Eth1Monitor (extracted from the merge branch) 2021-11-27 18:43:01 +02:00
Jakub ff50670147
ci: specify macos host architecture in labels (#3132)
Since now we have a 5th Gen Mac Mini with `arm64` M1 CPU.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2021-11-26 14:55:33 +01:00
Jakub 0dd497640a
ci: throttle builds to avoid tests clashing (#3127)
Uses:
https://plugins.jenkins.io/throttle-concurrents/#example-1-throttling-of-node-runs

The `nimbus-eth` category limits builds to one per node in global settings.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2021-11-25 22:22:08 +01:00
Jacek Sieka f69b272850
Keep cooked pubkeys in cache (#3122)
Turning uncompressed pubkeys into cooked ones is fast, but unnecessary -
this should avoid a little work for every signature validation we do by
pre-loading them at startup.
2021-11-25 19:41:54 +01:00
Mamy Ratsimbazafy 97da6e1365
Fork choice EF consensus tests (#3041)
* add EF fork choice tests to CI

* checkpoints

* compilation fixes and add test to preset dependent suite

* support longpaths on Windows CI

* skip minimal tests (long paths issue + impl detals tested)

* fix stackoverflow on some platforms

* rebase on top of https://github.com/status-im/nimbus-eth2/pull/3054

* fix stack usage
2021-11-25 19:41:39 +01:00
Jacek Sieka a223d62b07
Cleanups (#3123)
Renames and cleanups split out from the validator monitoring branch, so
as to reduce conflict area vs other PR:s

* add constants for expected message timing
* name validators after the messages they validate, mostly, to make
grepping easier
* unify field naming of EpochInfo across forks to make cross-fork code
easier
2021-11-25 13:20:36 +01:00
tersec cc0dbd5bc0
implement terminal-total-difficulty-override; keep kintsugi TTDs consistent (#3118)
* implement terminal-total-difficulty-override; keep TTD consistent for m2 scripts/docs

* use Option[uint64] instead of uint64
2021-11-25 10:53:31 +00:00
Jacek Sieka 9c2f43ed0e
Speed up altair block processing 2x (#3115)
* Speed up altair block processing >2x

Like #3089, this PR drastially speeds up historical REST queries and
other long state replays.

* cache sync committee validator indices
* use ~80mb less memory for validator pubkey mappings
* batch-verify sync aggregate signature (fixes #2985)
* document sync committee hack with head block vs sync message block
* add batch signature verification failure tests

Before:

```
../env.sh nim c -d:release -r ncli_db --db:mainnet_0/db bench --start-slot:-1000
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
    5830.675,        0.000,     5830.675,     5830.675,            1, Initialize DB
       0.481,        1.878,        0.215,       59.167,          981, Load block from database
    8422.566,        0.000,     8422.566,     8422.566,            1, Load state from database
       6.996,        1.678,        0.042,       14.385,          969, Advance slot, non-epoch
      93.217,        8.318,       84.192,      122.209,           32, Advance slot, epoch
      20.513,       23.665,       11.510,      201.561,          981, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database load
       0.000,        0.000,        0.000,        0.000,            0, Database store
```

After:

```
    7081.422,        0.000,     7081.422,     7081.422,            1, Initialize DB
       0.553,        2.122,        0.175,       66.692,          981, Load block from database
    5439.446,        0.000,     5439.446,     5439.446,            1, Load state from database
       6.829,        1.575,        0.043,       12.156,          969, Advance slot, non-epoch
      94.716,        2.749,       88.395,      100.026,           32, Advance slot, epoch
      11.636,       23.766,        4.889,      205.250,          981, Apply block, no slot processing
       0.000,        0.000,        0.000,        0.000,            0, Database load
       0.000,        0.000,        0.000,        0.000,            0, Database store
```

* add comment
2021-11-24 13:43:50 +01:00
Zahary Karadjov 88c623e250 Add support for HTTPS Web3 providers 2021-11-23 15:56:18 +02:00
Ștefan Talpalaru 2d986c5346
bump nimbus-build-system (#3117) 2021-11-23 02:43:04 +01:00
Ștefan Talpalaru 4639b04b2a
bump nimbus-build-system (#3116) 2021-11-22 21:21:13 +01:00