Commit Graph

298 Commits

Author SHA1 Message Date
Etan Kissling 41634f6385
avoid port 8301 on Jenkins (#5271)
Port 8301 is used by Consul: https://github.com/status-im/nimbus-eth2/issues/4575#issuecomment-1667490331

Avoid using it in the local testnet simulation when EXECUTOR_NUMBER == 2
2023-08-07 19:58:24 +00:00
Etan Kissling 6a2bac5cee
need even more log lines for debugging keymanager (#5260)
Keymanager test logs are still cut off, further increase log lines.
2023-08-07 18:02:08 +02:00
Etan Kissling c26e8132fc
fix ubsan for `test_libnimbus_lc` (#5258)
Pass `-fsanitize=undefined` flag when compiling `test_libnimbus_lc`
when building in ubsan mode.
2023-08-07 07:57:56 +02:00
Etan Kissling 6d24e73f8d
adjust keymanager test port base (#5263)
Port 9952 sometimes seems blocked, maybe by another Status project
being tested on the same host, or by UPS process, or maybe it needs
SO_REUSEPORT as well?

Switch to 9962 for now to see if that improves things.

```
{"lvl":"DBG","ts":"2023-08-06 13:44:56.834+03:00","msg":"Initializing ELManager","topics":"beacnde","tid":231629305,"file":"el_manager.nim:1823","depositContractBlockNumber":1,"depositContractBlockHash":"00000000"}
{"lvl":"ERR","ts":"2023-08-06 13:44:56.834+03:00","msg":"Could not create HTTP server instance","topics":"beacnde","tid":231629305,"file":"server.nim:50","address":{"family":"IPv4","address_v4":[127,0,0,1],"port":9952},"serverIdent":"nim-presto/0.0.3 (arm64/macosx)","serverFlags":["NotifyDisconnect","QueryCommaSeparatedArray"],"socketFlags":["ReuseAddr"],"serverUri":{"scheme":"","username":"","password":"","hostname":"","port":"","path":"","query":"","anchor":"","opaque":false,"isIpv6":false},"maxConnections":-1,"backlogSize":100,"bufferSize":4096,"httpHeadersTimeout":"15250w1d23h47m16s854ms775us807ns","maxHeadersSize":131072,"maxRequestBodySize":16777216,"err":"(48) Address already in use"}
{"lvl":"NOT","ts":"2023-08-06 13:44:56.835+03:00","msg":"Rest server could not be started","topics":"beacnde","tid":231629305,"file":"nimbus_binary_common.nim:370","address":"127.0.0.1:9952","reason":"Could not create HTTP server instance"}
```
2023-08-07 04:58:50 +02:00
Etan Kissling 5b70a686e3
further adjust `test_keymanager_api` logs (#5259)
It is still unclear how `test_keymanager_api` sometimes fails in CI;
further adjust logging parameters.
2023-08-05 13:12:44 +00:00
Etan Kissling 40e9bdc220
update port assignments to support more Jenkins executors (#5243)
Some Linux hosts have 3 concurrent executors. To avoid job failure
due to network port conflicts, adjust the port assignments.
2023-08-03 20:43:45 +00:00
Etan Kissling ed213538c7
more log lines when jobs fail (#5226)
Increase log lines emitted on test failure to 1000.
2023-07-31 15:03:32 +02:00
Etan Kissling 971b4483c5
add `libnimbus_lc.a` C library (#5122)
Add a new C library for processing light client data based on the Nimbus
implementation. This can be used from other, non-Nimbus components.
2023-07-19 09:48:39 +02:00
Miran 283f716593
update required mdbook versions (#5125)
I couldn't install those packages with the versions previously stated.
The new versions install successfully,
and `make publish-book` works without errors.
2023-06-26 16:04:01 +00:00
tersec 38a6706509
initial version of mock builder relay (#5034) 2023-06-06 15:21:15 +00:00
Etan Kissling d7890ac013
avoid port number conflict in `test_signing_node` (#5018)
Allow multiple copies of `test_signing_node` to run concurrently by
ensuring that they use unique network port numbers.
2023-06-01 09:33:25 +00:00
zah 8833acbe23
Add support for using custom remote signers in local sim (#4989)
* Add support for using custom remote signers in local sim

Other changes:

* Enable the Nimbus remote signer in the minimal simulation
* Move all log files into the `logs` folder of the simulation
* Create PID files for all processes and use them during the clean-up
  phase instead of the previous more fragile methods for killing the
  remaining processes.
2023-05-25 15:05:38 +00:00
Zahary Karadjov 01a5a6c14a
Remove some obsolete Makefile targets 2023-05-15 17:41:12 +03:00
henridf 573228ffa0
Rename eth1/ -> el/ and eth1_monitor.nim -> el_monitor.nim (#4944) 2023-05-15 05:05:12 +00:00
tersec 8686991890
add optimized Linux amd64 flavor (#4914)
* add optimized Linux amd64 flavor

* copyright year
2023-05-11 13:05:52 +03:00
tersec 630ab3567e
actually reach capella fork epoch in mainnet local sims (#4845) 2023-04-20 18:18:35 +03:00
Zahary Karadjov 69d737e61d
Remove the Makefile targets related to the deprecated zhejiang testnet 2023-04-18 15:07:11 +03:00
Etan Kissling 450f06566b
accelerate execution layer sync using light client (#4805)
Turn on `--sync-light-client` option by default, now that it has shown
stability in local testnets.
2023-04-10 14:28:46 +00:00
Eugene Kabanov 0ff86e9538
web3signer refactoring and test suite. (#4775)
* Refactor nimbus_signing_node to support Unix signals.

* Fix SN unable to close REST server properly.

* Fix `keys`, `deposit` and `validator_registration` endpoints issues.
Add getValidatorExitSignature() and getDepositMessageSignature() to validator_pool.

* Add /reload endpoint and implementation.
Fix signData to not cancel `timer`.
Fix validator_pool should clear attachedValidators table.

* Diva protocol enhancement implementation.
2023-04-06 16:16:21 +03:00
zah 8771e91d53
Support for driving multiple EL nodes from a single Nimbus BN (#4465)
* Support for driving multiple EL nodes from a single Nimbus BN

Full list of changes:

* Eth1Monitor has been renamed to ELManager to match its current
  responsibilities better.

* The ELManager is no longer optional in the code (it won't have
  a nil value under any circumstances).

* The support for subscribing for headers was removed as it only
  worked with WebSockets and contributed significant complexity
  while bringing only a very minor advantage.

* The `--web3-url` parameter has been deprecated in favor of a
  new `--el` parameter. The new parameter has a reasonable default
  value and supports specifying a different JWT for each connection.
  Each connection can also be configured with a different set of
  responsibilities (e.g. download deposits, validate blocks and/or
  produce blocks). On the command-line, these properties can be
  configured through URL properties stored in the #anchor part of
  the URL. In TOML files, they come with a very natural syntax
  (althrough the URL scheme is also supported).

* The previously scattered EL-related state and logic is now moved
  to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim`
  in a follow-up commit). State is assigned properly either to the
  `ELManager` or the to individual `ELConnection` objects where
  appropriate.

  The ELManager executes all Engine API requests against all attached
  EL nodes, in parallel. It compares their results and if there is a
  disagreement regarding the validity of a certain payload, this is
  detected and the beacon node is protected from publishing a block
  with a potential execution layer consensus bug in it.

  The BN provides metrics per EL node for the number of successful or
  failed requests for each type Engine API requests. If an EL node
  goes offline and connectivity is resoted later, we report the
  problem and the remedy in edge-triggered fashion.

* More progress towards implementing Deneb block production in the VC
  and comparing the value of blocks produced by the EL and the builder
  API.

* Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
tersec adeaa9e6c4
build make all targets in debug mode on GitHub Actions CI (#4655) 2023-02-27 11:30:13 +00:00
tersec cdca07908b
no remote signer or threshold in default CI (#4667)
* no remote signer or threshold in default CI

* also no remote validators
2023-02-25 10:52:10 +00:00
tersec 73797fb911
revert that version of no-remote-signer 2023-02-25 07:17:01 +00:00
tersec 3c542c3ae0
no remote signer in default CI 2023-02-25 00:18:34 +00:00
zah 6036f2e7d7
Local sim impovements (#4551)
* Local sim impovements

* Added support for running Capella and EIP-4844 simulations
  by downloading the correct version of Geth.

* Added support for using Nimbus remote signer and Web3Signer.
  Use 2 out of 3 threshold signing configuration in the mainnet
  configuration and regular remote signing in the minimal one.

* The local testnet simulation can now use a payload builder.
  This is currently not activated in CI due to lack of automated
  procedures for installing third-party relays or builders.

  You are adviced to use mergemock for now, but for most realistic
  results, we can create a simple builder based on the nimbus-eth1
  codebase that will be able to propose transactions from the regular
  network mempool.

* Start the simulation from a merged state. This would allow us
  to start removing pre-merge functionality such as the gossip
  subsciption logic. The commit also removes the merge-forcing
  hack installed after the TTD removal.

* Consolidate all the tools used in the local simulation into a
  single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
Etan Kissling ea6a6b1acd
track slot as part of fork choice debug API (#4565)
Extends fork choice state to also track slot numbers to improve accuracy
of `/eth/v1/debug/fork_choice` endpoint. Autoenable this API on devnet,
and disable some extra checks on devnet to aid focused testing efforts.
Align fork choice pruning logic with API based on checkpoints vs root.
2023-01-31 12:35:01 +00:00
tersec b975455254
avoid gnosis-build-triggered OOMs (#4531) 2023-01-20 14:14:08 +00:00
zah a2bc10e51c
Makefile targets for capella-devnet-3 (#4521) 2023-01-18 17:19:02 +02:00
tersec 87a34bff6c
build largest two RAM consumers, nimbus_beacon_node and all_tests, by default separately (#4513) 2023-01-16 11:34:40 +01:00
zah 0f758c5f02
Working Makefile targets for Capella devnet2 (#4494)
* Working Makefile targets for Capella devnet2

make capella-devnet-2
make clean-capella-devnet-2

You'll need to have https://github.com/tmuxinator/tmuxinator installed.
It's available as a regular package in most Linux distributions or through
Nix or Brew on macOS.

This commit also fixes the initial hang in the Eth1 monitor in the "find
TTD block" procedure through a fix to the network metadata files which
hasn't been upstreamed yet.

Other changes:

* Disabled Geth snap sync in the simulation

When all Geth nodes are configured to run with snap sync enabled, they all
start snap sync after the first forkchoiceUpdated which causes the BNs to
skip validator duties because the EL is syncing. The snap sync never completes
due to poor connectivity between the Geth nodes in the simulation.
2023-01-13 12:21:58 +02:00
zah 9861f0ab7d
Enable the gnosis build in CI on non-Windows targets (#4501) 2023-01-13 12:17:53 +02:00
Zahary Karadjov 3b03ef8ffb
Remove the genesis detection code 2023-01-13 04:28:30 +02:00
Zahary Karadjov 2ac28b1346
Disable the CI build of Gnosis 2023-01-13 04:28:30 +02:00
Zahary Karadjov b06502bf65
Gnosis const preset 2023-01-13 04:28:29 +02:00
Zahary Karadjov c01e53e35e
Fix bitrot in the gnosis build; Up-to-date bootstrap nodes
To prevent similar bitrot from occuring in the future, the gnosis-build
is now part of the default `make` targets.
2023-01-13 04:28:29 +02:00
Etan Kissling 44aa1f5152
avoid VC keymanager port conflict in CI (#4459)
`BASE_VC_KEYMANAGER_PORT` was not configured separately between runs,
make it configurable and base on `EXECUTOR_NUMBER` like the others.
2023-01-04 18:59:35 +01:00
Etan Kissling 7878e8083b
avoid port re-use across unit test runs (#4374)
In Jenkins CI we run two instances of unit tests concurrently.
This can trigger CI failure when the same port numbers are re-used
by the different test instances. Fixed one more issue of this by
allowing user configuration of the base port number.
2022-11-30 16:29:08 +02:00
Zahary Karadjov 4b9621cd77
Switch to a HTTPS web3-url for the Gnosis network (the wss is currently down) 2022-11-30 12:48:08 +02:00
Zahary Karadjov fbbab3bfef
Add version metric for the VC; Enable VC metrics in the local simulation 2022-11-30 12:47:11 +02:00
Jacek Sieka 04392c236b
bumps (#4352)
* build&style fixes
* use multithreading for LTO outside of make too
* blst 0.3.10 with no significant changes
2022-11-24 20:56:02 +00:00
Zahary Karadjov 26fe3990b9
Show last 50 lines on test failure instead of last 10 2022-11-04 14:30:29 +02:00
zah 865637e8d8
Show last 10 lines from the test log file on test failure (#4284) 2022-11-04 01:11:11 +00:00
tersec f9830836a9
deprecate --terminal-total-difficulty-override; remove launch script for deprecated ropsten (#4241)
* deprecate --terminal-total-difficulty-override; remove launch script for deprecated ropsten

* remove Makefile support for Ropsten
2022-10-24 23:32:52 +03:00
Jacek Sieka fa9c60089c
add fake execution engine server (#4250)
Useful for testing beacon node without running an execution client
(results in an
optimistically synced node)
2022-10-18 22:18:36 +00:00
Etan Kissling 23007eac3b
finalize 🐼 transition in local testnet CI (#4142)
When running local testnets as part of Jenkins CI, add one more epoch
to finalize the merge transition block.
2022-09-19 11:10:47 +00:00
Etan Kissling 3f17ceb0e2
run local finalization testnets with Geth (#4138) 2022-09-18 08:44:20 +03:00
Etan Kissling ae655e7b0f
add `lodestar` to known lib p2p agents (#4108)
Lodestar is switching from `js-libp2p/0.36.2` to `lodestar/version`.
Collect metrics on Lodestar peers following that scheme.
https://github.com/status-im/nimbus-eth2/issues/4106
2022-09-10 17:57:34 +02:00
Etan Kissling 613f4a9a50
accelerate EL sync with LC with `--sync-light-client` (#4041)
When the BN-embedded LC makes sync progress, pass the corresponding
execution block hash to the EL via `engine_forkchoiceUpdatedV1`.
This allows the EL to sync to wall slot while the chain DAG is behind.
Renamed `--light-client` to `--sync-light-client` for clarity, and
`--light-client-trusted-block-root` to `--trusted-block-root` for
consistency with `nimbus_light_client`.

Note that this does not work well in practice at this time:
- Geth sticks to the optimistic sync:
  "Ignoring payload while snap syncing" (when passing the LC head)
  "Forkchoice requested unknown head" (when updating to LC head)
- Nethermind syncs to LC head but does not report ancestors as VALID,
  so the main forward sync is still stuck in optimistic mode:
  "Pre-pivot block, ignored and returned Syncing"

To aid EL client teams in fixing those issues, having this available
as a hidden option is still useful.
2022-08-29 12:16:35 +00:00
zah 8273b3d909
Keep CLI options consistent by removing the '-enable' suffix from the outliers (#3928) 2022-08-05 17:38:26 +02:00
Etan Kissling 3ec7982293
update light client protocol version (#3550)
* Use final `v1` version for light client protocols
* Unhide LC data collection options
* Default enable LC data serving
* rm unneeded import
* Connect to EL on startup
* Add docs for LC based EL sync
2022-07-29 11:45:39 +03:00