Commit Graph

5960 Commits

Author SHA1 Message Date
Etan Kissling 13f436bc24
explicit `int` array indices for Nim 2.0 (#5306)
In Nim, when using a `distinct` capacity for an array, `int` lookups
don't work. Therefore, define the array with `int` capacity.
2023-08-17 17:26:08 +00:00
Jacek Sieka 49729e1ef3
prevent concurrent `storeBlock` calls (fixes #5285) (#5295)
When a block is introduced to the system both via REST and gossip at the
same time, we will call `storeBlock` from two locations leading to a
dupliace check race condition as we wait for the EL.

This issue may manifest in particular when using an external block
builder that itself publishes the block onto the gossip network.

* refactor enqueue flow
* simplify calling `addBlock`
* complete request manager verifier future for blobless blocks
* re-verify parent conditions before adding block

among other things, it might have gone stale or finalized between one
call and the other
2023-08-17 15:12:37 +02:00
Jacek Sieka 5d936c24e4
fix batch validation when `--num-threads=1` (#5303)
without a dedicated taskpools thread, validation will never happen!
2023-08-17 10:58:13 +02:00
tersec 0d1614326d
16-byte align ValidatorSig and TrustedSig (#5301) 2023-08-17 06:27:26 +00:00
tersec 22b7b721d0
adjust digest hashing not to care about alignment; increase ValidatorPubKey alignment (#5296) 2023-08-17 03:06:21 +00:00
Jacek Sieka a150bc93a6
blst: 0.3.11 (#5300)
```
Essential changes:

    security bugfix in Go bindings
    run-time switch in portable build
    no-std support [exercised with limited stack size of 56K]
    serde support
blst_miller_loop_n interface, with parallelized Rust and Go
counterparts
    optional blst_fr_pentaroot subroutine
    Emscripten bindings
    hardened security on non-asm platforms
```
2023-08-16 18:13:58 +00:00
Etan Kissling 1fa3941aa5
set `WITH_UBSAN` correctly when running on GA (#5299)
GitHub actions script was not setting `WITH_UBSAN` correctly. Fix it.
2023-08-16 16:56:05 +00:00
Eugene Kabanov 4a84457376
VC: Fix API timeouts issue. (#5297)
* Fix VC timeouts issue.
Add tests.

* Update AllTests.

* Remove assertion, which could crash VC when API call was interrupted by some higher logic.
2023-08-16 16:14:39 +00:00
Jacek Sieka ddddfc41aa
bearssl: bump to 0.2.1 (#5298) 2023-08-16 15:42:01 +00:00
tersec 230626306a
use fcU V3 in Deneb/Cancun (#5293) 2023-08-15 23:00:35 +00:00
tersec c3584a9ea7
fix regression in remote signing REST schema (#5294) 2023-08-15 22:39:12 +00:00
Eugene Kabanov 025b0e3f04
VC: Refactor sync committee duties and selection proofs calculation. (#5184)
* Initial commit.

* Fix issues in fill*() procedures.

* Enable sync committee processing.

* Address review comments.

* Address remaining review comments.

* Add sync committee selection proof pruning mechanism.

* Fix recursion issue.
2023-08-15 19:00:27 +02:00
Jacek Sieka b39e71335d
ssz: bump for 10% hash_tree_root perf increase, json loading fix (#5292)
* move file loading out of timed ncli section to focus benchmark on
deserialization rather than disk loading
* JSON loading fix included in bump, but does not practically happen
since we generally don't load hashlist from JSON
2023-08-15 13:28:33 +00:00
Eugene Kabanov 255131d331
Bump chronos. (#5281)
Add current/last connection query to the connection information.
2023-08-15 13:14:34 +02:00
tersec af37a96dbd
don't send fcUs every block if in lc-opt regime and block putatively finalized (#5248) 2023-08-15 09:27:56 +00:00
dependabot[bot] 6de3c49512
Bump tornado from 6.3.2 to 6.3.3 in /ncli (#5291)
Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.3.2 to 6.3.3.
- [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst)
- [Commits](https://github.com/tornadoweb/tornado/compare/v6.3.2...v6.3.3)

---
updated-dependencies:
- dependency-name: tornado
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-14 23:53:30 +00:00
Jacek Sieka f77548310f
avoid quadratic attestation queue iteration (#5288)
every attestation is processed with a new wall time so we end up
iterating over all attestations for every attestation we queue - this is
4% of cpu time on a subscribe-all-subnets node

* remove redundant zero checks - block root must be an existing block
and therefore cannot be zero
* simplify "hasn't-voted" check to root only (isZeroMemory is dubiously
implemented for objects)
2023-08-14 16:48:30 +03:00
tersec d171303133
update some consensus spec URLs to v1.4.0-beta.1 (#5287) 2023-08-12 10:38:06 +00:00
tersec 6cd8f3ba55
Update constants/presets for consistency with consensus-specs v1.4.0 (#5284) 2023-08-12 03:10:12 +00:00
henridf 9efd26c2e9
Add support for POST /eth/v2/beacon/blocks (#5214)
* Add support for POST /eth/v2/beacon/blocks

* More descriptive errors

* Address review feedback

* Return 500 (not 400) for a missing implementation case
2023-08-12 03:08:54 +00:00
henridf 9ceed40090
Prune blobs (#5283)
* Prune blobs

* Address review feedback
2023-08-11 00:15:34 +00:00
Jacek Sieka 3c9fc49411
reduce ncli/research imports (#5275)
* reduce ncli/research imports

avoids ssl, most of chronos etc, speeds up compiles

* copyright
2023-08-10 14:52:49 +02:00
Jacek Sieka a154a42205
bump ssz (small speedup) (#5282) 2023-08-09 19:40:57 +00:00
Etan Kissling 973e9e9a98
speed up LC signature verification (#5268)
We know the aggregate publickey of a fully participating sync committee.
Because participation is typically very high (>95%), it is faster to
start from that aggregate publickey and subtract the individual keys of
non-participants, than summing up all the participating pubkeys.
2023-08-09 08:50:07 +02:00
tersec 8e398001fe
bump nimcrypto to f118a5f86b04fbfd388b584d749f93bb3795f88f (#5279) 2023-08-09 06:40:03 +00:00
tersec 85e1976ac3
automated consensus spec URL updating to v1.4.0-beta.1 (#5280) 2023-08-09 03:58:47 +00:00
tersec 8c3d1cd3aa
ensure `ValidatorPubKey` int-compatibly aligned to fix UB (#5277)
* ensure ValidatorPubKey int-compatibly aligned

* use sizeof(Hash) explicitly
2023-08-08 19:01:32 +00:00
tersec f8520c8b1c
use v1.4.0 stability subnets by default (#5276) 2023-08-08 19:01:04 +00:00
Etan Kissling 2b57a221de
bump `nim-blscurve` to `337ab727eb8a38a0a71725f6f01ce55563f355c0` (#5274)
- bump blst
- add sig aggregation bench
- allow subtracting from aggregate for when participation is high
- add `{.push: raises: [].}` annotation
2023-08-08 12:20:08 +00:00
tersec 18061f7991
Remove most remaining portions of Bellatrix Builder API (#5272) 2023-08-08 00:05:17 +00:00
tersec 3e2c0a64df
always use --mm:refc, even under Nim 2.0 (#5273) 2023-08-08 02:04:41 +02:00
Etan Kissling 41634f6385
avoid port 8301 on Jenkins (#5271)
Port 8301 is used by Consul: https://github.com/status-im/nimbus-eth2/issues/4575#issuecomment-1667490331

Avoid using it in the local testnet simulation when EXECUTOR_NUMBER == 2
2023-08-07 19:58:24 +00:00
Etan Kissling bb672a7e90
bump Jenkins build timeout (#5270)
With the gnosis binary, building all targets sometimes exceeds the 40m.
Bump timeout to 50m.

```
Cancelling nested steps due to timeout
Sending interrupt signal to process
gcc: fatal error: Terminated signal terminated program lto1
compilation terminated.
make[2]: *** [/tmp/cctcWxAN.mk:134: /tmp/cciCtCWJ.ltrans44.ltrans.o] Error 1
make[2]: *** Waiting for unfinished jobs....
gcc: fatal error: Terminated signal terminated program lto1
compilation terminated.
make[2]: *** [/tmp/cctcWxAN.mk:155: /tmp/cciCtCWJ.ltrans51.ltrans.o] Error 1
lto-wrapper: fatal error: make returned 2 exit status
compilation terminated.
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make[1]: *** [nimcache/release/nimbus_beacon_node_gnosis/nimbus_beacon_node_gnosis.makefile:3148: build] Error 1
make: *** [Makefile:699: gnosis-build] Error 2
```
2023-08-07 19:42:28 +00:00
Etan Kissling 6a2bac5cee
need even more log lines for debugging keymanager (#5260)
Keymanager test logs are still cut off, further increase log lines.
2023-08-07 18:02:08 +02:00
tersec b9920a3a59
don't overwrite immutable parts of already-present validators (#5264) 2023-08-07 14:52:42 +00:00
Etan Kissling d7afa1c78a
add helpers for processing EL block header to `libnimbus_lc.a` (#5199)
To obtain the correct `transactions_root` and `withdrawals_root`,
it is necessary to process execution block header. Light client updates
don't contain the correct MPT roots.
2023-08-07 14:23:44 +02:00
tersec 2c424cac87
Merkle is someone's name (#5265) 2023-08-07 10:06:47 +00:00
andri lim e194d7b4ab
bump ssz-serialization to 4edfffe2b8d338444ef0b754954fba3b9fe78b9d (#5266) 2023-08-07 07:45:34 +00:00
Etan Kissling c26e8132fc
fix ubsan for `test_libnimbus_lc` (#5258)
Pass `-fsanitize=undefined` flag when compiling `test_libnimbus_lc`
when building in ubsan mode.
2023-08-07 07:57:56 +02:00
Etan Kissling 6d24e73f8d
adjust keymanager test port base (#5263)
Port 9952 sometimes seems blocked, maybe by another Status project
being tested on the same host, or by UPS process, or maybe it needs
SO_REUSEPORT as well?

Switch to 9962 for now to see if that improves things.

```
{"lvl":"DBG","ts":"2023-08-06 13:44:56.834+03:00","msg":"Initializing ELManager","topics":"beacnde","tid":231629305,"file":"el_manager.nim:1823","depositContractBlockNumber":1,"depositContractBlockHash":"00000000"}
{"lvl":"ERR","ts":"2023-08-06 13:44:56.834+03:00","msg":"Could not create HTTP server instance","topics":"beacnde","tid":231629305,"file":"server.nim:50","address":{"family":"IPv4","address_v4":[127,0,0,1],"port":9952},"serverIdent":"nim-presto/0.0.3 (arm64/macosx)","serverFlags":["NotifyDisconnect","QueryCommaSeparatedArray"],"socketFlags":["ReuseAddr"],"serverUri":{"scheme":"","username":"","password":"","hostname":"","port":"","path":"","query":"","anchor":"","opaque":false,"isIpv6":false},"maxConnections":-1,"backlogSize":100,"bufferSize":4096,"httpHeadersTimeout":"15250w1d23h47m16s854ms775us807ns","maxHeadersSize":131072,"maxRequestBodySize":16777216,"err":"(48) Address already in use"}
{"lvl":"NOT","ts":"2023-08-06 13:44:56.835+03:00","msg":"Rest server could not be started","topics":"beacnde","tid":231629305,"file":"nimbus_binary_common.nim:370","address":"127.0.0.1:9952","reason":"Could not create HTTP server instance"}
```
2023-08-07 04:58:50 +02:00
tersec d97027fb43
remove less-than-useful warning (#5245)
* remove less-than-useful warning

* remove redundant check for node.config.payloadBuilderEnable
2023-08-06 15:12:32 +00:00
tersec 2ae2d0a042
remove Bellatrix Builder API (Signed)BuilderBid (#5262)
* remove Bellatrix Builder API (Signed)BuilderBid

* stub out BlindedBeaconBlock
2023-08-06 09:20:43 +00:00
Etan Kissling 5b70a686e3
further adjust `test_keymanager_api` logs (#5259)
It is still unclear how `test_keymanager_api` sometimes fails in CI;
further adjust logging parameters.
2023-08-05 13:12:44 +00:00
Etan Kissling ee00fb3d80
replace copy with symlink (#5252)
One of the test files is accidentally a copy instead of a symlink.
2023-08-05 11:06:55 +00:00
Etan Kissling e9940115e4
switch from `create`/`dealloc` to `new+GC_ref`/`GC_unref` (#5238)
Using `create` with objects containing managed objects is broken:
https://github.com/nim-lang/Nim/issues/22341

Switch to a safer pattern based on `new+GC_ref`/`GC_unref` instead.
2023-08-05 09:26:27 +02:00
tersec 1f5dd83fbd
unplumbing Bellatrix builder API support (#5203) 2023-08-05 05:13:08 +00:00
tersec d542410a17
remove more Bellatrix-only Builder API support infrastructure (#5251) 2023-08-05 00:11:51 +00:00
Etan Kissling 4cf1a7e791
further reduce CI keymanager test log spam (#5250)
Remove verbose `gossip_eth2` logs not relevant for `test_keymanager_api`
from the log to debug CI issue.
2023-08-04 22:16:19 +00:00
tersec 001fbd0970
Revert "Revert "correctly align genesis size (#5244)" (#5253)" (#5254)
This reverts commit 26dcee111b.
2023-08-04 20:49:12 +00:00
tersec 26dcee111b
Revert "correctly align genesis size (#5244)" (#5253)
This reverts commit 706ce01bb1.
2023-08-04 20:11:20 +00:00