5809 Commits

Author SHA1 Message Date
Jacek Sieka
49729e1ef3
prevent concurrent storeBlock calls (fixes #5285) (#5295)
When a block is introduced to the system both via REST and gossip at the
same time, we will call `storeBlock` from two locations leading to a
dupliace check race condition as we wait for the EL.

This issue may manifest in particular when using an external block
builder that itself publishes the block onto the gossip network.

* refactor enqueue flow
* simplify calling `addBlock`
* complete request manager verifier future for blobless blocks
* re-verify parent conditions before adding block

among other things, it might have gone stale or finalized between one
call and the other
2023-08-17 15:12:37 +02:00
Jacek Sieka
5d936c24e4
fix batch validation when --num-threads=1 (#5303)
without a dedicated taskpools thread, validation will never happen!
2023-08-17 10:58:13 +02:00
tersec
0d1614326d
16-byte align ValidatorSig and TrustedSig (#5301) 2023-08-17 06:27:26 +00:00
tersec
22b7b721d0
adjust digest hashing not to care about alignment; increase ValidatorPubKey alignment (#5296) 2023-08-17 03:06:21 +00:00
Jacek Sieka
a150bc93a6
blst: 0.3.11 (#5300)
```
Essential changes:

    security bugfix in Go bindings
    run-time switch in portable build
    no-std support [exercised with limited stack size of 56K]
    serde support
blst_miller_loop_n interface, with parallelized Rust and Go
counterparts
    optional blst_fr_pentaroot subroutine
    Emscripten bindings
    hardened security on non-asm platforms
```
2023-08-16 18:13:58 +00:00
Etan Kissling
1fa3941aa5
set WITH_UBSAN correctly when running on GA (#5299)
GitHub actions script was not setting `WITH_UBSAN` correctly. Fix it.
2023-08-16 16:56:05 +00:00
Eugene Kabanov
4a84457376
VC: Fix API timeouts issue. (#5297)
* Fix VC timeouts issue.
Add tests.

* Update AllTests.

* Remove assertion, which could crash VC when API call was interrupted by some higher logic.
2023-08-16 16:14:39 +00:00
Jacek Sieka
ddddfc41aa
bearssl: bump to 0.2.1 (#5298) 2023-08-16 15:42:01 +00:00
tersec
230626306a
use fcU V3 in Deneb/Cancun (#5293) 2023-08-15 23:00:35 +00:00
tersec
c3584a9ea7
fix regression in remote signing REST schema (#5294) 2023-08-15 22:39:12 +00:00
Eugene Kabanov
025b0e3f04
VC: Refactor sync committee duties and selection proofs calculation. (#5184)
* Initial commit.

* Fix issues in fill*() procedures.

* Enable sync committee processing.

* Address review comments.

* Address remaining review comments.

* Add sync committee selection proof pruning mechanism.

* Fix recursion issue.
2023-08-15 19:00:27 +02:00
Jacek Sieka
b39e71335d
ssz: bump for 10% hash_tree_root perf increase, json loading fix (#5292)
* move file loading out of timed ncli section to focus benchmark on
deserialization rather than disk loading
* JSON loading fix included in bump, but does not practically happen
since we generally don't load hashlist from JSON
2023-08-15 13:28:33 +00:00
Eugene Kabanov
255131d331
Bump chronos. (#5281)
Add current/last connection query to the connection information.
2023-08-15 13:14:34 +02:00
tersec
af37a96dbd
don't send fcUs every block if in lc-opt regime and block putatively finalized (#5248) 2023-08-15 09:27:56 +00:00
dependabot[bot]
6de3c49512
Bump tornado from 6.3.2 to 6.3.3 in /ncli (#5291)
Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.3.2 to 6.3.3.
- [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst)
- [Commits](https://github.com/tornadoweb/tornado/compare/v6.3.2...v6.3.3)

---
updated-dependencies:
- dependency-name: tornado
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-14 23:53:30 +00:00
Jacek Sieka
f77548310f
avoid quadratic attestation queue iteration (#5288)
every attestation is processed with a new wall time so we end up
iterating over all attestations for every attestation we queue - this is
4% of cpu time on a subscribe-all-subnets node

* remove redundant zero checks - block root must be an existing block
and therefore cannot be zero
* simplify "hasn't-voted" check to root only (isZeroMemory is dubiously
implemented for objects)
2023-08-14 16:48:30 +03:00
tersec
d171303133
update some consensus spec URLs to v1.4.0-beta.1 (#5287) 2023-08-12 10:38:06 +00:00
tersec
6cd8f3ba55
Update constants/presets for consistency with consensus-specs v1.4.0 (#5284) 2023-08-12 03:10:12 +00:00
henridf
9efd26c2e9
Add support for POST /eth/v2/beacon/blocks (#5214)
* Add support for POST /eth/v2/beacon/blocks

* More descriptive errors

* Address review feedback

* Return 500 (not 400) for a missing implementation case
2023-08-12 03:08:54 +00:00
henridf
9ceed40090
Prune blobs (#5283)
* Prune blobs

* Address review feedback
2023-08-11 00:15:34 +00:00
Jacek Sieka
3c9fc49411
reduce ncli/research imports (#5275)
* reduce ncli/research imports

avoids ssl, most of chronos etc, speeds up compiles

* copyright
2023-08-10 14:52:49 +02:00
Jacek Sieka
a154a42205
bump ssz (small speedup) (#5282) 2023-08-09 19:40:57 +00:00
Etan Kissling
973e9e9a98
speed up LC signature verification (#5268)
We know the aggregate publickey of a fully participating sync committee.
Because participation is typically very high (>95%), it is faster to
start from that aggregate publickey and subtract the individual keys of
non-participants, than summing up all the participating pubkeys.
2023-08-09 08:50:07 +02:00
tersec
8e398001fe
bump nimcrypto to f118a5f86b04fbfd388b584d749f93bb3795f88f (#5279) 2023-08-09 06:40:03 +00:00
tersec
85e1976ac3
automated consensus spec URL updating to v1.4.0-beta.1 (#5280) 2023-08-09 03:58:47 +00:00
tersec
8c3d1cd3aa
ensure ValidatorPubKey int-compatibly aligned to fix UB (#5277)
* ensure ValidatorPubKey int-compatibly aligned

* use sizeof(Hash) explicitly
2023-08-08 19:01:32 +00:00
tersec
f8520c8b1c
use v1.4.0 stability subnets by default (#5276) 2023-08-08 19:01:04 +00:00
Etan Kissling
2b57a221de
bump nim-blscurve to 337ab727eb8a38a0a71725f6f01ce55563f355c0 (#5274)
- bump blst
- add sig aggregation bench
- allow subtracting from aggregate for when participation is high
- add `{.push: raises: [].}` annotation
2023-08-08 12:20:08 +00:00
tersec
18061f7991
Remove most remaining portions of Bellatrix Builder API (#5272) 2023-08-08 00:05:17 +00:00
tersec
3e2c0a64df
always use --mm:refc, even under Nim 2.0 (#5273) 2023-08-08 02:04:41 +02:00
Etan Kissling
41634f6385
avoid port 8301 on Jenkins (#5271)
Port 8301 is used by Consul: https://github.com/status-im/nimbus-eth2/issues/4575#issuecomment-1667490331

Avoid using it in the local testnet simulation when EXECUTOR_NUMBER == 2
2023-08-07 19:58:24 +00:00
Etan Kissling
bb672a7e90
bump Jenkins build timeout (#5270)
With the gnosis binary, building all targets sometimes exceeds the 40m.
Bump timeout to 50m.

```
Cancelling nested steps due to timeout
Sending interrupt signal to process
gcc: fatal error: Terminated signal terminated program lto1
compilation terminated.
make[2]: *** [/tmp/cctcWxAN.mk:134: /tmp/cciCtCWJ.ltrans44.ltrans.o] Error 1
make[2]: *** Waiting for unfinished jobs....
gcc: fatal error: Terminated signal terminated program lto1
compilation terminated.
make[2]: *** [/tmp/cctcWxAN.mk:155: /tmp/cciCtCWJ.ltrans51.ltrans.o] Error 1
lto-wrapper: fatal error: make returned 2 exit status
compilation terminated.
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make[1]: *** [nimcache/release/nimbus_beacon_node_gnosis/nimbus_beacon_node_gnosis.makefile:3148: build] Error 1
make: *** [Makefile:699: gnosis-build] Error 2
```
2023-08-07 19:42:28 +00:00
Etan Kissling
6a2bac5cee
need even more log lines for debugging keymanager (#5260)
Keymanager test logs are still cut off, further increase log lines.
2023-08-07 18:02:08 +02:00
tersec
b9920a3a59
don't overwrite immutable parts of already-present validators (#5264) 2023-08-07 14:52:42 +00:00
Etan Kissling
d7afa1c78a
add helpers for processing EL block header to libnimbus_lc.a (#5199)
To obtain the correct `transactions_root` and `withdrawals_root`,
it is necessary to process execution block header. Light client updates
don't contain the correct MPT roots.
2023-08-07 14:23:44 +02:00
tersec
2c424cac87
Merkle is someone's name (#5265) 2023-08-07 10:06:47 +00:00
andri lim
e194d7b4ab
bump ssz-serialization to 4edfffe2b8d338444ef0b754954fba3b9fe78b9d (#5266) 2023-08-07 07:45:34 +00:00
Etan Kissling
c26e8132fc
fix ubsan for test_libnimbus_lc (#5258)
Pass `-fsanitize=undefined` flag when compiling `test_libnimbus_lc`
when building in ubsan mode.
2023-08-07 07:57:56 +02:00
Etan Kissling
6d24e73f8d
adjust keymanager test port base (#5263)
Port 9952 sometimes seems blocked, maybe by another Status project
being tested on the same host, or by UPS process, or maybe it needs
SO_REUSEPORT as well?

Switch to 9962 for now to see if that improves things.

```
{"lvl":"DBG","ts":"2023-08-06 13:44:56.834+03:00","msg":"Initializing ELManager","topics":"beacnde","tid":231629305,"file":"el_manager.nim:1823","depositContractBlockNumber":1,"depositContractBlockHash":"00000000"}
{"lvl":"ERR","ts":"2023-08-06 13:44:56.834+03:00","msg":"Could not create HTTP server instance","topics":"beacnde","tid":231629305,"file":"server.nim:50","address":{"family":"IPv4","address_v4":[127,0,0,1],"port":9952},"serverIdent":"nim-presto/0.0.3 (arm64/macosx)","serverFlags":["NotifyDisconnect","QueryCommaSeparatedArray"],"socketFlags":["ReuseAddr"],"serverUri":{"scheme":"","username":"","password":"","hostname":"","port":"","path":"","query":"","anchor":"","opaque":false,"isIpv6":false},"maxConnections":-1,"backlogSize":100,"bufferSize":4096,"httpHeadersTimeout":"15250w1d23h47m16s854ms775us807ns","maxHeadersSize":131072,"maxRequestBodySize":16777216,"err":"(48) Address already in use"}
{"lvl":"NOT","ts":"2023-08-06 13:44:56.835+03:00","msg":"Rest server could not be started","topics":"beacnde","tid":231629305,"file":"nimbus_binary_common.nim:370","address":"127.0.0.1:9952","reason":"Could not create HTTP server instance"}
```
2023-08-07 04:58:50 +02:00
tersec
d97027fb43
remove less-than-useful warning (#5245)
* remove less-than-useful warning

* remove redundant check for node.config.payloadBuilderEnable
2023-08-06 15:12:32 +00:00
tersec
2ae2d0a042
remove Bellatrix Builder API (Signed)BuilderBid (#5262)
* remove Bellatrix Builder API (Signed)BuilderBid

* stub out BlindedBeaconBlock
2023-08-06 09:20:43 +00:00
Etan Kissling
5b70a686e3
further adjust test_keymanager_api logs (#5259)
It is still unclear how `test_keymanager_api` sometimes fails in CI;
further adjust logging parameters.
2023-08-05 13:12:44 +00:00
Etan Kissling
ee00fb3d80
replace copy with symlink (#5252)
One of the test files is accidentally a copy instead of a symlink.
2023-08-05 11:06:55 +00:00
Etan Kissling
e9940115e4
switch from create/dealloc to new+GC_ref/GC_unref (#5238)
Using `create` with objects containing managed objects is broken:
https://github.com/nim-lang/Nim/issues/22341

Switch to a safer pattern based on `new+GC_ref`/`GC_unref` instead.
2023-08-05 09:26:27 +02:00
tersec
1f5dd83fbd
unplumbing Bellatrix builder API support (#5203) 2023-08-05 05:13:08 +00:00
tersec
d542410a17
remove more Bellatrix-only Builder API support infrastructure (#5251) 2023-08-05 00:11:51 +00:00
Etan Kissling
4cf1a7e791
further reduce CI keymanager test log spam (#5250)
Remove verbose `gossip_eth2` logs not relevant for `test_keymanager_api`
from the log to debug CI issue.
2023-08-04 22:16:19 +00:00
tersec
001fbd0970
Revert "Revert "correctly align genesis size (#5244)" (#5253)" (#5254)
This reverts commit 26dcee111b4831ca577f3dec818415985ed52920.
2023-08-04 20:49:12 +00:00
tersec
26dcee111b
Revert "correctly align genesis size (#5244)" (#5253)
This reverts commit 706ce01bb180b4cfbf19ece642fbde17f7a68de9.
2023-08-04 20:11:20 +00:00
Jacek Sieka
706ce01bb1
correctly align genesis size (#5244) 2023-08-04 18:08:15 +03:00