Commit Graph

427 Commits

Author SHA1 Message Date
xiaolou86 af6affb5b3
Fix typos (#5835)
* Fix typos

* Update copyright year
2024-01-29 09:37:38 +00:00
Etan Kissling d08e558b88
bump `geth` to `v1.13.11` (#5825)
- https://github.com/ethereum/go-ethereum/releases/tag/v1.13.9
- https://github.com/ethereum/go-ethereum/releases/tag/v1.13.10
- https://github.com/ethereum/go-ethereum/releases/tag/v1.13.11
2024-01-24 15:42:55 +00:00
tersec 6c53dc1e11
automated consensus spec URL updating to v1.4.0-beta.6 (#5804) 2024-01-20 11:19:47 +00:00
jangko 7378d55108
Fix unbound variable error in launch_local_testnet.sh 2024-01-15 09:21:20 +07:00
Zahary Karadjov 05fb7ffff0
Don't require getopt to be installed by brew on macOS for Nix users
Other changes:

* Add/Update copyright banners
* Add post-review comment for https://github.com/status-im/nimbus-eth2/pull/5719
2024-01-13 12:37:41 +02:00
zah 00d05f8170
Use the latest web3signer (#5732)
Other changes:

* Include java in the Nix development shell
  (so users are able to run the web3signer)

* Update the simulation config files to the latest spec
  (so the web3signer accepts them)
2024-01-13 10:37:09 +02:00
Jacek Sieka 62cbdeefc5
verify `genesis_time` more strictly (fixes #1667) (#5694)
Bogus values lead to crashes down the line when timers overflow
2024-01-06 15:26:56 +01:00
Etan Kissling 508f3b6368
add missing `std/` prefix to more imports (#5696)
Bumping some `std` imports to explicitly include the prefix.
Also add explicit `./` prefix for local directory imports.
2024-01-06 07:18:28 +01:00
Etan Kissling a90e1308b5
bump `geth` to `1.13.8` (#5683)
Use latest Geth in local testnet simulation.
2023-12-28 04:12:03 +00:00
tersec 9efb2958ec
automated consensus spec URL updating to v1.4.0-beta.5 (#5647) 2023-12-05 03:34:45 +01:00
tersec 925d531884
test deneb in Jenkins finalization CI (#5627)
* Revert "Revert "test deneb in Jenkins finalization CI (#5625)" (#5626)"

This reverts commit bbf4551ff9.

* always use current Geth stable
2023-11-26 14:38:19 +00:00
tersec bbf4551ff9
Revert "test deneb in Jenkins finalization CI (#5625)" (#5626)
This reverts commit af08c0f47d.
2023-11-26 05:17:46 +00:00
tersec af08c0f47d
test deneb in Jenkins finalization CI (#5625) 2023-11-26 03:37:36 +00:00
Etan Kissling d5fbbd9b88
increase `GENESIS_OFFSET` for local testnet simulation (#5619)
The 40s allotted for setting up a local testnet can sometime be exceeded
when CI is busy. Bumping it by 20s should reduce that risk.

Example, where the entire slots 0 and 1 were missed:

```
{"lvl":"INF","ts":"2023-11-22 18:33:32.000+00:00","msg":"Scheduling first slot action","topics":"beacnde","startTime":"8s999ms800us826ns","nextSlot":2,"timeToNextSlot":"3s199us174ns"}
```
2023-11-22 23:19:54 +00:00
Etan Kissling c81e3a556e
bump `geth` to `1.13.5` in CI (#5618)
Use stable Geth version in CI, the previous unstable one is no longer
available and fails CI builds.
2023-11-22 22:01:40 +01:00
tersec aa6c13cd67
use newer geth testnet binary directory names (#5575)
* use newer geth testnet binary directory names

* linting; Geth requires merged genesis
2023-11-07 16:52:30 +01:00
tersec 726b8fbb39
bump nim-web3 to rm web3_consensus_const_preset (#5515) 2023-10-19 13:12:21 +00:00
tersec 9fd52ebe9f
use consensus spec v1.4.0-beta.3 test vectors (#5512) 2023-10-19 05:14:02 +00:00
Jacek Sieka e64b31986e
Increase default file limit on startup (#5436)
We use file descriptors for validators and sockets and might run out of
either on high-validator setups - increasing the limit here is harmless
and avoids a common limiting factor in setup

Co-authored-by: Etan Kissling <etan@status.im>
2023-09-21 16:08:13 +02:00
Tanguy 4918a4e2e0
Fix direct peers (#5427)
* Fix direct peers

* Support ENRs in DP, use DP in local testnet

* fix docs

* bump libp2p
2023-09-15 18:45:55 +00:00
tersec 91a53e0cb6
capella genesis in local testnet (#5403) 2023-09-11 11:26:21 +02:00
tersec 8e413d9abb
rename --stop-at-epoch flag to --debug-stop-at-epoch (#5361) 2023-08-30 14:15:19 +03:00
Jakub Sokołowski 6162e735dd
launch_local_testnet: add debugging for port conflicts (#5317)
We have been seeing some port conflicts like:
```
[2023-08-15T00:31:47.625Z] Geth 0 failed to start
```
```
$ tail -n1 local-testnet-mainnet/logs/geth.?.txt
==> local-testnet-mainnet/logs/geth.0.txt <==
Fatal: Error starting protocol stack: listen tcp :6801: bind: address already in use

==> local-testnet-mainnet/logs/geth.1.txt <==
Fatal: Error starting protocol stack: listen tcp :6806: bind: address already in use

==> local-testnet-mainnet/logs/geth.2.txt <==
Fatal: Error starting protocol stack: listen tcp :6811: bind: address already in use
```
In order to debug this we'll need to add printing of some extra info
into `unstable` so feature branches include it.

Related: https://github.com/status-im/nimbus-eth2/issues/4575

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-08-18 22:24:40 +00:00
Etan Kissling 6a2bac5cee
need even more log lines for debugging keymanager (#5260)
Keymanager test logs are still cut off, further increase log lines.
2023-08-07 18:02:08 +02:00
Etan Kissling 5b70a686e3
further adjust `test_keymanager_api` logs (#5259)
It is still unclear how `test_keymanager_api` sometimes fails in CI;
further adjust logging parameters.
2023-08-05 13:12:44 +00:00
Etan Kissling ed213538c7
more log lines when jobs fail (#5226)
Increase log lines emitted on test failure to 1000.
2023-07-31 15:03:32 +02:00
tersec ae13b694e4
find/update consensus-spec URLs in *.{c,h} files (#5218)
* find/update consensus-spec URLs in *.{c,h} files

* copyright
2023-07-29 10:23:25 +00:00
Zahary Karadjov e6ed11fc33
Restore the ability to test the Web3Signer (don't enable it in CI yet) 2023-06-03 00:41:57 +03:00
zah 8833acbe23
Add support for using custom remote signers in local sim (#4989)
* Add support for using custom remote signers in local sim

Other changes:

* Enable the Nimbus remote signer in the minimal simulation
* Move all log files into the `logs` folder of the simulation
* Create PID files for all processes and use them during the clean-up
  phase instead of the previous more fragile methods for killing the
  remaining processes.
2023-05-25 15:05:38 +00:00
henridf 573228ffa0
Rename eth1/ -> el/ and eth1_monitor.nim -> el_monitor.nim (#4944) 2023-05-15 05:05:12 +00:00
henridf 5dfd814588
Load trusted setup (#4870)
* Kzg: Load trusted setup

* scripts/launch_local_testnet.sh: set FIELD_ELEMENTS_PER_BLOB

* Use right setup file for mainnet/minimal

* Force rebuild

* Add comment explaining why build with -f
2023-05-11 11:52:44 +03:00
Zahary Karadjov 0d6453aa51
Add an USE_VC env variable affecting the local simulation targets 2023-04-28 00:21:01 +03:00
tersec 228e10f1d9
update engine API URLs from v1.0.0-beta.2 to beta.3 (#4828) 2023-04-17 20:11:28 +00:00
Etan Kissling 450f06566b
accelerate execution layer sync using light client (#4805)
Turn on `--sync-light-client` option by default, now that it has shown
stability in local testnets.
2023-04-10 14:28:46 +00:00
tersec ed148ab69d
fix false positive getopt failure with multiple getopt matches in searched path (#4797)
* fix false positive getopt failure with multiple getopt matches in searched path

* also get launch_local_testnet

* also make_prometheus_config, called from launch_local_testnet
2023-04-08 00:18:29 +00:00
Zahary Karadjov 7f2a3b7130
[local sim] Download the latest nimbus-eth2 when --dl-nimbus-eth2 is used 2023-04-05 19:12:28 +03:00
Zahary Karadjov b4d731a1cc
Make it possible to run the local simulation on nixOS 2023-03-30 19:37:29 +03:00
zah 6539928775
Refer to binaries with their proper extensions on Windows (#4739) 2023-03-17 19:43:52 +01:00
Etan Kissling 2f006d31e7
rename `GETH_EIP_4844_BINARY` > `GETH_DENEB_BINARY` (#4729) 2023-03-12 01:47:24 +00:00
Etan Kissling 969c6f73ae
misc local `EIP4844` > `Deneb` bumps (#4717)
* misc local `EIP4844` > `Deneb` bumps

* fix
2023-03-11 00:28:19 +00:00
Zahary Karadjov 5370a6b67b
Capture the Geth output when running 'make zhejiang' 2023-03-10 11:13:24 +02:00
zah 8771e91d53
Support for driving multiple EL nodes from a single Nimbus BN (#4465)
* Support for driving multiple EL nodes from a single Nimbus BN

Full list of changes:

* Eth1Monitor has been renamed to ELManager to match its current
  responsibilities better.

* The ELManager is no longer optional in the code (it won't have
  a nil value under any circumstances).

* The support for subscribing for headers was removed as it only
  worked with WebSockets and contributed significant complexity
  while bringing only a very minor advantage.

* The `--web3-url` parameter has been deprecated in favor of a
  new `--el` parameter. The new parameter has a reasonable default
  value and supports specifying a different JWT for each connection.
  Each connection can also be configured with a different set of
  responsibilities (e.g. download deposits, validate blocks and/or
  produce blocks). On the command-line, these properties can be
  configured through URL properties stored in the #anchor part of
  the URL. In TOML files, they come with a very natural syntax
  (althrough the URL scheme is also supported).

* The previously scattered EL-related state and logic is now moved
  to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim`
  in a follow-up commit). State is assigned properly either to the
  `ELManager` or the to individual `ELConnection` objects where
  appropriate.

  The ELManager executes all Engine API requests against all attached
  EL nodes, in parallel. It compares their results and if there is a
  disagreement regarding the validity of a certain payload, this is
  detected and the beacon node is protected from publishing a block
  with a potential execution layer consensus bug in it.

  The BN provides metrics per EL node for the number of successful or
  failed requests for each type Engine API requests. If an EL node
  goes offline and connectivity is resoted later, we report the
  problem and the remedy in edge-triggered fashion.

* More progress towards implementing Deneb block production in the VC
  and comparing the value of blocks produced by the EL and the builder
  API.

* Adds a Makefile target for the zhejiang testnet
2023-03-05 01:40:21 +00:00
Jakub Sokołowski 2f36f15b20
launch_local_testnet.sh: fix terminating Geth (#4684)
Also improve logging and list jobs after cleanup.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-03-01 18:27:55 +02:00
Jakub Sokołowski f46ed12f04
launch_local_testnet.sh: re-add targetting parent PID (#4680)
Otherwise we kill of other unrelated processes.

Fix for bug introduced most probably in:
https://github.com/status-im/nimbus-eth2/pull/4551

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-02-28 10:39:39 +00:00
Jakub Sokołowski f2a09802b1
launch_local_testnet.sh: print processes first
Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-02-28 11:24:20 +01:00
Jakub Sokołowski 02437bb365
launch_local_testnet.sh: show which killed processes
First step in debugging issue most probably re-introduced by:
https://github.com/status-im/nimbus-eth2/pull/4551

Which causes the finalization tests script to kill other processes
unrelated to the given CI job.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-02-28 11:24:17 +01:00
Kim De Mey 14f4100e1b
Fix Nimbus EL node index in local testnet script (#4671) 2023-02-27 20:39:50 +01:00
zah 6036f2e7d7
Local sim impovements (#4551)
* Local sim impovements

* Added support for running Capella and EIP-4844 simulations
  by downloading the correct version of Geth.

* Added support for using Nimbus remote signer and Web3Signer.
  Use 2 out of 3 threshold signing configuration in the mainnet
  configuration and regular remote signing in the minimal one.

* The local testnet simulation can now use a payload builder.
  This is currently not activated in CI due to lack of automated
  procedures for installing third-party relays or builders.

  You are adviced to use mergemock for now, but for most realistic
  results, we can create a simple builder based on the nimbus-eth1
  codebase that will be able to propose transactions from the regular
  network mempool.

* Start the simulation from a merged state. This would allow us
  to start removing pre-merge functionality such as the gossip
  subsciption logic. The commit also removes the merge-forcing
  hack installed after the TTD removal.

* Consolidate all the tools used in the local simulation into a
  single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
Jakub Sokołowski 3d3d17aad9
after_install: use . not source in bourne shell (#4639)
Another dumb mistake when using bourne shell:
```
/var/lib/dpkg/info/nimbus-beacon-node.postinst: 23: source: not found
```

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-02-17 14:26:23 +01:00
Jakub Sokołowski 7d599185e0
use Bourne shell as the actual postinst script does (#4638)
The `postinst` wrapper script into which these scripts are embedded as
`after_upgrade` and `after_install` functions are executed using Bourne
shell(`sh`), so we cannot use the Bash specific `[[ ]]` test or it fails:
```
/var/lib/dpkg/info/nimbus-beacon-node.postinst: 22: [[: not found
```

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-02-17 14:13:28 +02:00