A fix for a bug triggered by recent `Jenkinsfile` refactoring done in:
https://github.com/status-im/nimbus-eth2/pull/3827
Which due to a big in Jenkins Throttling plugin caused jobs to start
running in parallel on the same host despite global configuration that
is supposed to block this:
https://issues.jenkins.io/browse/JENKINS-49173https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/68
An attempt to fix this was made in this PR:
https://github.com/status-im/nimbus-eth2/pull/3913
But it was ineffective due to bugs in the Throttle plugin.
As a result semi-random testnet launches would fail with errors like this:
```
./scripts/launch_local_testnet.sh: line 1026: 58977 Killed: 9 ${BEACON_NODE_COMMAND} ...
```
The culprit was the old process cleanup in `scripts/launch_local_testnet.sh`:
```
+ make local-testnet-mainnet
Found old process listening on port 7001, with PID 58977. Killing it.
Found old process listening on port 7002, with PID 59024. Killing it.
Found old process listening on port 7003, withu PID 59027. Killing it.
Found old process listening on port 7004, with PID 59030. Killing it.
```
Which was triggered due to use of immediate assignment for `EXECUTOR_NUMBER`:
```
EXECUTOR_NUMBER := 0
```
Which cause the `EXECUTOR_NUMBER` value set by Jenkins to be ignored.
For more details see:
https://www.gnu.org/software/make/manual/html_node/Flavors.html#Flavors
Signed-off-by: Jakub Sokołowski <jakub@status.im>
The ports for the concurrently executing REST and Minimal testnet clash,
leading to some CI failures since #3827 introduced further concurrency.
Adjusting the ports to be distinct across various tests should fix this.
LC cancels concurrent requests if one peer sent a correct response and
waiting for other peers is no longer useful. On the server side this
resulted in a descore (-500) and a likely disconnect. The behaviour is
changed to match `UnexpectedEOF`, `PotentiallyExpectedEOF` handling that
return an error response without disconnecting from the peer.
`mkdocs` works with markdown similar to `mdbook` but is generally more
pleasing to the eye and has several nice UX features.
This PR does the bulk of the transition - likely, a followup would be
needed to fully make use of the extra features and navigation.
Book pages have been kept url-compatible, meaning that for the most
part, old links should continue to work!
Co-authored-by: Etan Kissling <etan@status.im>
Changes:
- Name local testnet output folders same as the `make` taget.
- Move both `Jenkinsfile`s to `ci` folder to avoid cluttering repo root.
- Separate builds by platform so logs from macos and linux hosts don't get mixed.
- Detect platform and architecture from Jenkins Job path to use one Jenkinsfile.
- Divide shell commands into as many stages as possible to make debugging easier.
- Generalize running testnets via a `launchLocalTestnet()` Groovy method.
- Handle uploading of results of running testnets stage-by-stage basis.
- Use `catchError()` to upload test results while marking job as failed.
- Abort previously started PR build jobs usin `disableConcurrentBuilds()`.
- Throttle jobs using the new `throttleJobProperty()` function.
Builds:
- https://ci.status.im/job/nimbus/job/nimbus-eth2/job/platforms/job/linux/job/x86_64/
- https://ci.status.im/job/nimbus/job/nimbus-eth2/job/platforms/job/macos/job/x86_64/
- https://ci.status.im/job/nimbus/job/nimbus-eth2/job/platforms/job/macos/job/aarch64/
Signed-off-by: Jakub Sokołowski <jakub@status.im>
* update docs
* introduce mdbook-admonish for nice looking callouts
* new section on data directory
* recommend source build for advanced users and direct the rest to
binaries
* more strongly highlight that execution client is needed
* write an actual deposit guide
* remove cruft / fix links / etc
When there is heavy forking, proposals may get missed due to including
attestations from different forks that later fail verification.
Checking attestation signatures when building blocks should fix this.
The light client sync protocol employs heuristics to ensure it does not
become stuck during non-finality or low sync committee participation.
These can enable use cases that prefer availability of recent data
over security. For our syncing use case, though, security is preferred.
An option is added to light client processor to configure this tradeoff.
* Re-enabled requireAllFields after a fix in nim-json-serialization
The problem was that `Option[T]` fields were not treated as optional
when requireAllFields is set to true. This is now fixed in NJS.
* Add makefile targets for recreating the Jenkins simulation runs
* Fix a discrepancy with the REST spec
It's not quite clear why this condition was triggered in the local
simulation, but it seems a viable scenario after the Keymanager API
is integrated in the validator client.
The user can temporarily remove all validator keys from a running
client before adding another set of keys.
Adds the `--web3-url` launch argument to `nimbus_light_client` to enable
driving the EL with the optimistic head obtained from LC sync protocol.
This will keep issuing `newPayload` / `forkChoiceUpdated` requests for
new blocks, marking them as optimistic. `ZERO_HASH` is reported as the
finalized block for now.
#3864 introduced a regression by turning on `requireAllFields` globally
for JSON parsing. Certain endpoints such as `RestSyncInfo` have optional
fields that do not parse correctly without additional changes. This is
reverted for now to restore previous behaviour and unblock CI testing.
Other changes:
* The Keymanager error responses differ from the Beacon API responses.
'keymanagerApiError' replaces the former usages of 'jsonError'.
* Return status code 401 and 403 for authorization errors in accordance
to the spec.
* Eliminate inconsistencies in the REST JSON parsing. Some of the code
paths allowed missing fields.
* Added logging of serialization failure details at DEBUG level.
Jakub Sokołowski
tersec
zah
Taneli Hukkinen
Jacek Sieka
Etan Kissling
Miran
Eugene Kabanov
Zahary Karadjov
kayla-henrie