Commit Graph

354 Commits

Author SHA1 Message Date
zah 6d11ad6ce1
Support for distributed keystores with multiple remotes based on threshold signatures (#3616)
Other fixes:

* Fix bit rot in the `make prater-dev-deposit` target.
* Correct content-type in the responses of the Nimbus signing node
* Invalid JSON payload was being sent in the web3signer requests
2022-05-10 03:32:12 +03:00
Etan Kissling 89ac586bd4
serve light client data in CI / dev builds (#3487)
Adjust config for CI / dev builds to serve light client data by default:
`--serve-light-client-data=1 --import-light-client-data=only-new`
2022-03-12 22:12:18 +01:00
tersec 21b71bd29c
update URL and document Nim bug blocking further genericizing cleanups (#3483) 2022-03-11 15:03:47 +00:00
Ștefan Talpalaru 857a71be6c
launch_local_testnet.sh: Lighthouse VC nodes (#3477)
* launch_local_testnet.sh: Lighthouse VC nodes
2022-03-11 13:44:56 +01:00
tersec 35f712e378
kiln v2 documentation and testing (#3446) 2022-03-05 16:40:58 +02:00
zah cdeae90806
Add support for TOML config files (--config-file) (#3442) 2022-03-05 04:33:15 +02:00
Ștefan Talpalaru ebba093362
Nim-1.6 compatibility (#3434) 2022-02-25 10:19:12 +02:00
tersec 05bc61b712
add mev-boost RPC test, with docs (#3430)
* bump nim-web3 and add mev-boost RPC test, with docs

* remove trailing space

* use specific commithash
2022-02-24 14:38:31 +01:00
Ștefan Talpalaru 555ce310cc
Jenkins: get rid of macOS warnings (#3412)
* Jenkins: get rid of macOS warnings

We're linking object files with debug info with ones without it. Perfectly justified, from our point of view, but dsymutil complains verbosely about it, with no switch to silence it.
2022-02-18 15:30:43 +01:00
tersec 8df4290dde
simple script to time module compilations (#3414) 2022-02-18 10:52:01 +00:00
zah 178431102d
[skip ci] scripts/run-geth-el.sh Allow connecting to any merge devnet (#3409) 2022-02-17 19:14:15 +02:00
tersec f64fc6dc3a
include Bellatrix for in CI finalization tests (#3407) 2022-02-17 14:02:56 +01:00
Jacek Sieka 87e98b9e54
Revert "bump submodules (#3366)" (#3406)
This reverts commit 6e1ad080e8.
2022-02-17 12:50:37 +01:00
Ștefan Talpalaru 6e1ad080e8
bump submodules (#3366)
and add Nim-1.6 compatibility
2022-02-16 13:41:50 +02:00
tersec 2160d02977
use REST for make pyrmont/prater; use --rest options for local testnet (#3380) 2022-02-12 12:27:43 +01:00
Dustin Brody 3daa52ab87
update docs for geth/kiln 2022-02-11 20:06:06 +00:00
tersec fc0ce57b68
kiln merge test vectors for EL (#3377) 2022-02-11 18:17:37 +00:00
tersec bf3ef987e4
deactivate doppelganger protection during genesis (#3362)
* deactivate Doppelganger Protection during genesis

* also don't actually flag supposed-doppelgangers (because they're before broadcastStartEpoch) on GENESIS_SLOT start
2022-02-07 07:12:36 +02:00
tersec c9aa1bee01
spec URL updates (#3342) 2022-01-31 09:56:59 +00:00
tersec 2b4a960270
rename On{Merge,Bellatrix}BlockAdded and Rollback{Merge,Bellatrix}HashedProc (#3321) 2022-01-26 13:21:29 +01:00
tersec ae61512ee9
rename upgrade_to_{merge,bellatrix}; detect unchanging spec YAMLs (#3265) 2022-01-10 09:39:43 +00:00
tersec a4667d1840
script to automate aspects of spec URL updating (#3252) 2022-01-06 12:59:36 +00:00
Emil 2a12d1c49f Support more content-types when specifying the Graffiti string 2022-01-06 10:56:59 +02:00
tersec d4680df8d2
convert between engine and consensus ExecutionPayloads (#3228)
* convert between engine and consensus ExecutionPayloads
2022-01-03 13:22:56 +01:00
Zahary Karadjov 6b4f32ae23
Replicate a recent fix from the launch_local_testnet script due to a wide-spread code duplication 2021-12-22 17:59:45 +02:00
Zahary Karadjov 5eabeef75d Allow a docker image to be used instead of building nimbus_beacon_node
This would be useful when testing nimbus-eth1 in merge-mode simulations
2021-12-16 14:10:18 +02:00
Jacek Sieka 1a87e60f7a
remove unnecessary binaries from builds (#3179)
* `fork_choice` and `proto_array` are already covered by better tests
* `nimbus_signing_node` is not used by local testnet
2021-12-10 11:56:51 +01:00
Jacek Sieka e2a2157370
RPM/DEB for systemd-based systems (#3034)
* RPM/DEB for systemd-based systems

Similar to https://github.com/status-im/nimbus-eth2/pull/2691, this PR
introduces packaging for nimbus beacon node.

Instead of relying on interactive prompts, a default configuration is
included that connects to mainnet and expects there to be an execution
client running with websockets enabled on the same host, on the standard
websocket port.

Should the user need to configure their nimbus after installation (for
example with a different web3 url), the "standard" way of doing so via
`systemctl` edit is recommended.
2021-12-07 15:23:57 +02:00
tersec e6921f808f
cleanups, partly from kintsugi branch (#3161)
* cleanups, partly from kintsugi branch

* re-export shortLog(EthBlock) and preserve exception messages in batchVerify and processBatch
2021-12-05 17:32:41 +00:00
Eugene Kabanov e62c7c7c37
Remote signing client/server. (#3077) 2021-11-30 03:20:21 +02:00
tersec 39f6a6534e
document how to run merge local testnet with Nethermind (#3110) 2021-11-17 20:45:39 +01:00
tersec c5fa7387f9
update kintsugi docs (#3104) 2021-11-15 17:01:27 +00:00
Zahary Karadjov 5e62df5ad5 /bin/bash -> /usr/bin/env bash (needed on NixOS) 2021-11-15 15:45:01 +02:00
tersec 97ad5d49b0
kintsugi merge vector tests (#3072) 2021-11-10 13:41:02 +02:00
zah 2c0be052a5
Reduce the logging-related breaking changes (#3070)
* Reduce the logging-related breaking changes

* Don't disable the stdout log when the `--log-file` option is used
* Rename `--log-stdout` to `--log-format` and hide it
* Deprecate the `--log-file` option
2021-11-10 11:02:18 +02:00
Ștefan Talpalaru b09cd71348 build: show stderr by default
Also added: `-d:cwarnings` and `-d:limitStackUsage`
2021-11-05 16:06:48 +02:00
Jacek Sieka 3bfb5ad367
logtrace needs json logs (#3047) 2021-11-03 07:24:03 +01:00
Jacek Sieka 233d756518
Logging and startup improvements (#3038)
* Logging and startup improvements

Color support for released binaries!

* startup scripts no longer log to file by default - this only affects
source builds - released binaries don't support file logging
* add --log-stdout option to control logging to stdout (colors, json)
* detect tty:s vs redirected logs and log accordingly
* add option to disable log colors at runtime
* simplify several "common" logs, showing the most important information
earlier and more clearly
* remove line numbers / file information / tid - these take up space and
are of little use to end users
  * still enabled in debug builds and tools
* remove `testnet_servers_image` compile-time option
* server images, released binaries and compile-from-source now offer
the same behaviour and features
* fixes https://github.com/status-im/nimbus-eth2/issues/2326
* fixes https://github.com/status-im/nimbus-eth2/issues/1794
* remove instanteneous block speed from sync message, keeping only
average

before:

```
INF 2021-10-28 16:45:59.000+02:00 Slot start                                 topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:884 lastSlot=2384027 wallSlot=2384028 delay=461us84ns peers=0 head=75a10ee5:3348 headEpoch=104 finalized=cd6804ba:3264 finalizedEpoch=102 sync="wwwwwwwwww:0:0.0000:0.0000:00h00m (3348)"
INF 2021-10-28 16:45:59.046+02:00 Slot end                                   topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:821 slot=2384028 nextSlot=2384029 head=75a10ee5:3348 headEpoch=104 finalizedHead=cd6804ba:3264 finalizedEpoch=102 nextAttestationSlot=-1 nextProposalSlot=-1 nextActionWait=n/a
```

after:

```
INF 2021-10-28 22:43:23.033+02:00 Slot start                                 topics="beacnde" slot=2385815 epoch=74556 sync="DDPDDPUDDD:10:5.2258:01h19m (2361088)" peers=37 head=eacd2dae:2361096 finalized=73782:a4751487 delay=33ms687us715ns
INF 2021-10-28 22:43:23.291+02:00 Slot end                                   topics="beacnde" slot=2385815 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 head=eacd2dae:2361096
```

* fix comment

* documentation updates

* mention `--log-file` may be deprecated in the future
* update various docs
2021-11-02 18:06:36 +01:00
Zahary Karadjov 6b3b82d549
Enable REST and metrics in the ./run-mainnet-node.sh script 2021-10-14 14:36:08 +03:00
tersec 2eb9a608a4
add payloadId; add merge vector test script; remove consensusValidated (#2982) 2021-10-13 16:08:50 +02:00
tersec 1f51331dd3
use specified execution_payload when constructing blocks; update Geth launch script (#2975)
* use specified execution_payload when constructing blocks; update Geth launch script

* revert DepositContractState change
2021-10-12 13:37:40 +02:00
Jacek Sieka 0c4bfb1911
test fixture cleanups (#2976)
* "official" -> "scenarios", like the submodule
* fewer test binaries - various compile hacks have been improved over
time, test suite should follow
* remove obsolete bls tests - there are better test vectors in
nim-blscurve
* remove obsolete mentions of `ssz_testing`
* remove obsolete comments about proc vs globals, unittest2 already uses
proc's
2021-10-12 13:36:52 +02:00
Dustin Brody 4a26e48226
add initial merge test vector scripts 2021-09-30 03:00:25 +00:00
zah 3689c68cbf
Carry out the sync committee gossip duties
Other changes:

* Add server getBlockV2(), and produceBlockV2().
* Add getBlockV2() to REST test suite.
* Add client getBlockV2(), and produceBlockV2().
* Fix URLs in comments.
* Add some primitives and fix some issues in forks.nim.
* Switch `validator_client` to V2 calls usage.
* Bump `chronos` with imports fixes.
* Bump `nim-json-serialization` for `requireAllFields`.
2021-08-30 03:58:30 +03:00
Zachinquarantine 80f2d4d9ee
run-beacon-node.sh: update URL (#2803) 2021-08-21 22:39:01 +02:00
Jacek Sieka a67970e9a4
subscribe to all subnets in minimal local testnet (#2799) 2021-08-20 16:41:16 +02:00
tersec df48409969
enable trace-level networking logging in local simulation (#2780)
* enable trace-level networking logging in local simulation

* remove Jenkinsfile overrides
2021-08-13 14:12:40 +00:00
Jacek Sieka 9697b73e71
forkedbeaconstate_helpers -> forks (#2772)
Simpler module name for stuff that covers forks

* check that runtime config matches database state
* also include some assorted altair cleanups
* use "standard" genesis fork in local testnet to work around missing
runtime config support
2021-08-10 22:46:35 +02:00
tersec 61825f4979
add aggregated attestation tracing to logtrace and enable it in Jenkins (#2766)
* add aggregated attestation tracing to logtrace and enable it in Jenkins CI

* use a slightly less cryptic acronym than aasr

* mostly, nimbus and the eth2 spec use aggregate attestation, not aggregated attestation
2021-08-06 21:25:30 +00:00
Ștefan Talpalaru 59b1a4772c
don't try to use lsof on Windows (#2762) 2021-08-05 10:49:34 +02:00
Ștefan Talpalaru ca96a98131
launch_local_testnet.sh: start the timeout earlier (#2753) 2021-08-03 12:12:21 +02:00
Ștefan Talpalaru 17c94c4d8e
restapi.sh: prevent port clashes (#2746) 2021-07-29 18:34:01 +02:00
Ștefan Talpalaru 7fe4569586
launch_local_testnet.sh: check if lsof is installed (#2742) 2021-07-27 21:45:36 +02:00
Ștefan Talpalaru 6299e14ffb
launch_local_testnet.sh: kill old procs before start (#2739) 2021-07-27 17:57:16 +02:00
tersec 941cc125a3
update to v1.1.0-beta.2 tests, with exception for altair sync committees (#2736)
* update to v1.1.0-beta.2 tests, with exception for altair sync committees

* bootstrap node can't drop any attestations
2021-07-26 17:34:13 +00:00
zah d9f2a91374
Remove the obsolete testnet0/1 scripts (#2727)
Also fixes `make eth2_network_simulation`
2021-07-16 13:02:27 +03:00
Jacek Sieka 3f9c1fdf4e
More RuntimeConfig cleanup (#2716)
* remove from BeaconChainDB (doesn't depend on runtime config)
* eth2-testnets -> eth2-networks
* use `cfg` name throughout
2021-07-13 16:27:10 +02:00
Eugene Kabanov 3b6f4fab4a
New validator client using REST API. (#2651)
* Initial commit.

* Exporting getConfig().

* Add beacon node checking procedures.

* Post rebase fixes.

* Use runSlotLoop() from nimbus_beacon_node.
Fallback implementation.
Fixes for ETH2 REST serialization.

* Add beacon_clock.durationToNextSlot().
Move type declarations from beacon_rest_api to json_rest_serialization.
Fix seq[ValidatorIndex] serialization.
Refactor ValidatorPool and add some utility procedures.
Create separate version of validator_client.

* Post-rebase fixes.
Remove CookedPubKey from validator_pool.nim.

* Now we should be able to produce attestations and aggregate and proofs.
But its not working yet.

* Debugging attestation sending.

* Add durationToNextAttestation.
Optimize some debug logs.
Fix aggregation_bits encoding.
Bump chronos/presto.

* Its alive.

* Fixes for launch_local_testnet script.
Bump chronos.

* Switch client API to not use `/api` prefix.

* Post-rebase adjustments.

* Fix endpoint for publishBlock().

* Add CONFIG_NAME.
Add more checks to ensure that beacon_node is compatible.

* Add beacon committee subscription support to validator_client.

* Fix stacktrace should be an array of strings.
Fix committee subscriptions should not be `data` keyed.

* Log duration to next block proposal.

* Fix beacon_node_status import.

* Use jsonMsgResponse() instead of jsonError().

* Fix graffityBytes usage.
Remove unnecessary `await`.
Adjust creation of SignedBlock instance.
Remove legacy files.

* Rework durationToNextSlot() and durationToNextEpoch() to use `fromNow`.

* Fix race condition for block proposal and attestations for same slot.
Fix local_testnet script to properly kill tasks on Windows.
Bump chronos and nim-http-tools, to allow connections to infura.io (basic auth).

* Catch services errors.
Improve performance of local_testnet.sh script on Windows.
Fix race condition when attestation producing.

* Post-rebase fixes.

* Bump chronos and presto.

* Calculate block publishing delay.
Fix pkill in one more place.

* Add error handling and timeouts to firstSuccess() template.
Add onceToAll() template.
Add checkNodes() procedure.
Refactor firstSuccess() template.
Add error checking to api.nim calls.

* Deprecated usage onceToAll() for better stability.
Address comment and send attestations asap.

* Avoid unnecessary loop when calculating minimal duration.
2021-07-13 13:15:07 +02:00
Jacek Sieka 23eea197f6
Implement split preset/config support (#2710)
* Implement split preset/config support

This is the initial bulk refactor to introduce runtime config values in
a number of places, somewhat replacing the existing mechanism of loading
network metadata.

It still needs more work, this is the initial refactor that introduces
runtime configuration in some of the places that need it.

The PR changes the way presets and constants work, to match the spec. In
particular, a "preset" now refers to the compile-time configuration
while a "cfg" or "RuntimeConfig" is the dynamic part.

A single binary can support either mainnet or minimal, but not both.
Support for other presets has been removed completely (can be readded,
in case there's need).

There's a number of outstanding tasks:

* `SECONDS_PER_SLOT` still needs fixing
* loading custom runtime configs needs redoing
* checking constants against YAML file

* yeerongpilly support

`build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG`

* load fork epoch from config

* fix fork digest sent in status
* nicer error string for request failures
* fix tools

* one more

* fixup

* fixup

* fixup

* use "standard" network definition folder in local testnet

Files are loaded from their standard locations, including genesis etc,
to conform to the format used in the `eth2-networks` repo.

* fix launch scripts, allow unknown config values

* fix base config of rest test

* cleanups

* bundle mainnet config using common loader
* fix spec links and names
* only include supported preset in binary

* drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 15:01:38 +02:00
Zahary Karadjov ed2f6f753d Allow the custom testnet metadata to specify a path to the genesis file 2021-06-01 15:50:50 +03:00
tersec ea9ceb693a
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622)
* update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot()

* update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
Zahary Karadjov b7aa30adfd
Merge stable into unstable 2021-05-20 13:50:40 +03:00
tersec d8bb91d9a9
partially integrate eth1 merge changes (#2548)
* partially integrate eth1 merge changes

* use hexToSeqByte() and validate execution engine opaque transaction length

* remove incorrect REST serialization code
2021-05-20 10:44:13 +00:00
tersec 290b889ce6
non-intrusive, novel portions of merge (#2535) 2021-05-04 11:54:19 +00:00
Zahary Karadjov 6806ffe1c8
Remove some unused parameters (part 2) 2021-04-14 21:49:42 +03:00
Ștefan Talpalaru 37b3f8d577 run-beacon-node.sh: handle "--web3-url=..." properly
Now the script looks at those command line arguments meant for
nimbus_beacon_node to decide whether to prompt the user for a WEB3 URL
or not.
2021-04-14 16:08:54 +03:00
Zahary Karadjov 61669f269d Clarify that the deposit cretion tools are intended for testnets 2021-04-08 19:58:22 +03:00
Jacek Sieka f821bc878e
Remove `-d:insecure` compile option (#2468)
With metrics running on top of chronos, the metrics server no longer
needs to be compiled in conditionally - it remains disabled by default.
2021-04-01 14:44:11 +02:00
Jacek Sieka 714620b13f
script for finding slow logs 2021-03-30 09:39:16 +02:00
Ștefan Talpalaru eafe694536 dist: faster builds for ARM and ARM64
We're moving from very slow QEMU-virtualised builds to very fast ones
based on cross toolchains.

- refactor jobs; add checksums to release note
- build and publish Docker images with ARM/ARM64 binaries
2021-03-15 14:38:39 +02:00
tersec dfc3322fe2
last few v1.0.0 spec refs to v1.0.1 (#2404) 2021-03-13 20:51:39 +00:00
Jacek Sieka ff94e1a212
Document log rotation (#2327)
* Document log rotation

* phrasing updates

Co-authored-by: Dustin Brody <tersec@users.noreply.github.com>
2021-02-18 07:57:41 +01:00
Ștefan Talpalaru f398720fc4
compile_nim_program.sh: bring back Nim warnings with V=0 (#2325) 2021-02-15 18:08:18 +01:00
Ștefan Talpalaru e4f4cb1513
remove "beacon_chain.nimble" (#2316)
Tests have been moved to the Makefile and the test binaries are built in
parallel.

The Nim compilation scheme has been moved to a script and adapted to
work with different binaries being built in parallel from the same main
source file (think minimal/mainnet tests).
2021-02-14 20:04:54 +00:00
Ștefan Talpalaru f84a52e83e launch_local_testnet.sh: NPROC for macOS 2021-02-11 22:08:30 +02:00
Mamy Ratsimbazafy 03f47c8f2f
Slashing protection refactor - EIP 3076 (#2094)
* Create CLI tool for slashing export

* Use SQLite as a DB instead of a KV-store

* Keeps v1 and v2 DBs around

* Uses the same schema as Lighthouse v1.1.0

* Passes all interchange tests + skeleton of finalization pruning

* Removes tests that would violate v5 / minimal slashing DB and MinSlot rules

* Migration tool added using low-watermark scheme for faster migration of large number of validators
2021-02-09 17:23:06 +02:00
Ștefan Talpalaru 979bb39b41 docs: binary distribution internals 2021-02-08 14:53:12 +02:00
Ștefan Talpalaru 80c11546ff Windows binary release
CI: use both cores on GitHub Actions and set timeouts for the local testnet tests
2021-02-04 10:25:44 +02:00
Dustin Brody 67e4a045a3 simplify doppelganger detection to boolean 2021-02-03 20:55:33 +02:00
Ștefan Talpalaru cadb1aae15
bump submodules (#2279)
* bump submodules

* undo a couple of bumps (32-bit failure)

* launch_local_testnet.sh: timeout and various fixes

* undo nim-http-utils bump (breaks RPC for VC)
2021-02-02 19:02:52 +01:00
tersec 6141286547
rename doppelganger protection to doppelganger detection; switch default from warn to stop (#2281) 2021-02-01 12:18:16 +01:00
Dustin Brody 281853eee8 rename options and internal structures to doppelgangerFoo and remove probing 2021-01-30 00:17:54 +02:00
Kim De Mey 40a5d44887
Fix selection of bootstrap nodes from metadata (#2273)
Also removes again the doubling of max peers
2021-01-29 08:56:02 +01:00
Dmitriy Ryajov 1117993a24
Libp2p connection limits (#2272)
* max peers gets set as connection limits

* pass max connections
2021-01-28 07:31:29 +01:00
Ștefan Talpalaru 20435880db CI: build and publish a Docker image for end-users
Also disable the log file and log colours for distribution binaries, to
avoid duplicate logs.
2021-01-26 18:52:59 +02:00
Dustin Brody a16f5afcd5 pre-emptive duplicate validator detection heuristic 2021-01-21 16:03:02 +02:00
Kim De Mey 6fabefa76f
Fix discv5 loop in case of no nodes (#2243)
* Fix discovery loop in case of no peers in routing table

* local testnet: Stop searching when amount of peers in testnet is reached
2021-01-18 14:13:26 +01:00
asymmetric 51f967a548
scripts: make shebang more portable (#2216)
Some Linux distros (like NixOS) don't place bash in /bin/bash.
2021-01-08 11:29:43 +01:00
Ștefan Talpalaru e58a355a1c
CI release: ARM and ARM64 builds (#2213) 2021-01-07 10:19:29 +01:00
Ștefan Talpalaru 92750d5313 reduce peak memory usage during build 2020-12-10 17:17:12 +02:00
Ștefan Talpalaru ad54e9d3bf
Prometheus/Grafana changes (#2092)
- grafana/prometheus.yml.example
- drop the custom label
- raise the minimum supported Grafana version to 7.2
- increase scrape_interval
- add validator reward rate panel (Zahary's)
2020-12-02 21:47:00 +01:00
Ștefan Talpalaru 9c5cef346b set file and dir permissions 2020-11-17 23:28:26 +02:00
Zahary Karadjov 500301c6c7
Restore the NODE_ID functionality and use the old data directories 2020-11-10 10:11:13 +02:00
tersec a010ef2061
update tests from v1.0.0-rc.0 to v1.0.0 (#1976) 2020-11-09 17:44:18 +00:00
Zahary Karadjov 1774da15f1 Implement help message for the run-*-beacon-node scripts 2020-11-09 17:09:49 +02:00
Zahary Karadjov e9b9cd75ee Rename binaries; Mimic the original repo layout in the distribution 2020-11-09 11:38:52 +02:00
yslcrypto 6f9d0fa29e update beth1 client guide link 2020-11-07 15:16:53 +01:00
Zahary Karadjov 0ea68d61ed
Add another missing file 2020-11-07 13:51:14 +02:00
Ștefan Talpalaru f3254fd754
dist: add version and book (#1965) 2020-11-06 17:26:03 +01:00
Ștefan Talpalaru c7d8b81c86 dist: add Medalla binary and starting script 2020-11-06 01:29:03 +02:00