Commit Graph

5960 Commits

Author SHA1 Message Date
Jacek Sieka 706ce01bb1
correctly align genesis size (#5244) 2023-08-04 18:08:15 +03:00
tersec 1c55ef7345
note summary data about invalid baked-in states (#5247) 2023-08-04 07:36:42 +02:00
Etan Kissling 8f18491131
adjust log level to reduce ci log spam (#5232)
In keymanager test, logs are currently not analyzable due to large
number of `libp2p gossipsub` logs. Hide those.
2023-08-03 23:09:08 +00:00
Etan Kissling 40e9bdc220
update port assignments to support more Jenkins executors (#5243)
Some Linux hosts have 3 concurrent executors. To avoid job failure
due to network port conflicts, adjust the port assignments.
2023-08-03 20:43:45 +00:00
Etan Kissling f0774564a8
bump `nim-web3` to `04f56c593a035af1b7bebdc726543a2a73826412` (#5241)
- fix `TransactionObject` type up through mainnet
- fix `Quantity` parsing on 32-bit platform
2023-08-03 17:18:57 +00:00
Etan Kissling 50dfd846f0
fix inconsistent capitalization (#5240)
Two cases of inconsistent capitalization, fix them.
2023-08-03 18:53:32 +02:00
Jacek Sieka b8a32419b8
async batch verification (+40% sig verification throughput) (#5176)
* async batch verification

When batch verification is done, the main thread is blocked reducing
concurrency.

With this PR, the new thread signalling primitive in chronos is used to
offload the full batch verification process to a separate thread
allowing the main threads to continue async operations while the other
threads verify signatures.

Similar to previous behavior, the number of ongoing batch verifications
is capped to prevent runaway resource usage.

In addition to the asynchronous processing, 3 addition changes help
drive throughput:

* A loop is used for batch accumulation: this prevents a stampede of
small batches in eager mode where both the eager and the scheduled batch
runner would pick batches off the queue, prematurely picking "fresh"
batches off the queue
* An additional small wait is introduced for small batches - this helps
create slightly larger batches which make better used of the increased
concurrency
* Up to 2 batches are scheduled to the threadpool during high pressure,
reducing startup latency for the threads

Together, these changes increase attestation verification throughput
under load up to 30%.

* fixup

* Update submodules

* fix blst build issues (and a PIC warning)

* bump

---------

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2023-08-03 11:36:45 +03:00
tersec 0a4036baee
bump nim-eth to d217d309eb12e39d2a642feae1d9d27157b88352 (#5237) 2023-08-03 02:13:36 +00:00
Jacek Sieka 5bc48acc36
reduce memory allocations during state transition (#5235)
This PR removes a few hundred thousand temporary seq allocations during
state transition - in particular, the flag seq was allocated per
validator while committees are computed per attestation.
2023-08-03 01:03:40 +02:00
henridf 28194468c9
Rename "data gas" to "blob gas" (#5216)
* Rename data gas to blob gas

* Update vendor/nim-eth and vendor/nim-web3
2023-08-02 22:07:57 +00:00
Miran a15cc10463
update FAQ (#5211)
* update FAQ

* address review comments
2023-08-02 11:53:24 +00:00
henridf 1da82a5dcd
Remove async blob queue from Request Manager (#5198)
* Remove async blob queue from Request Manager

* Review feedback

* Review feedback
2023-08-01 22:39:14 +02:00
Etan Kissling 32e8bfe911
bump `nim-confutils` to `dbe8d61f7fbb1cb9d74b38f9012406c8071aa9e9` (#5228)
- return empty completion lines on unsupported `COMP_POINT`
- require `secondarySources` to be `gcsafe`
2023-08-01 01:23:55 +00:00
Etan Kissling 66160d986f
bump `gnosis-chain-configs` to `b90374a1c63703db8235fcdb65aff2e909bc42b5` (#5227)
- fix: bootstrap_nodes.txt
2023-08-01 01:23:46 +00:00
tersec e9a17fb99f
remove workaround for long-since-fixed Nim bug (#5219) 2023-07-31 22:20:09 +00:00
tersec 909a7c2d07
use v1.4.0-beta.1 consensus spec tests (#5230) 2023-07-31 21:51:06 +00:00
Etan Kissling f8ba2e3882
generalize `ShufflingRef` acceleration logic (#5197) (#5204)
Split up the `ShufflingRef` acceleration logic into generically usable
parts and attester shuffling specific parts. The generic parts could be
used to accelerate other purposes, e.g., REST `/states/xxx/randao` API.
2023-07-31 18:12:15 +00:00
tersec 846e7c585b
Revert "Revert "generalize `ShufflingRef` acceleration logic (#5197)" (#5223)" (#5225)
This reverts commit 2ab4592a31.
2023-07-31 13:11:45 +00:00
Etan Kissling ed213538c7
more log lines when jobs fail (#5226)
Increase log lines emitted on test failure to 1000.
2023-07-31 15:03:32 +02:00
tersec 2ab4592a31
Revert "generalize `ShufflingRef` acceleration logic (#5197)" (#5223)
This reverts commit eb3a30655b.
2023-07-31 08:05:32 +02:00
tersec ae13b694e4
find/update consensus-spec URLs in *.{c,h} files (#5218)
* find/update consensus-spec URLs in *.{c,h} files

* copyright
2023-07-29 10:23:25 +00:00
Eugene Kabanov 219916162e
Add timeout on test. (#5215)
* Add timeout on test.

* Improved version.
2023-07-26 23:16:40 +03:00
Philippe Schommers 74c993d616
feat: add support for Chiado (#5208) 2023-07-26 15:54:16 +03:00
Jacek Sieka e8379389e7
speed up state/block loading (#5207)
* speed up state/block loading

When loading blocks and states from db/era, we currently redundantly
check their CRC32 - for a state, this costs 50ms of loading time
presently (110mb uncompressed size) on a decent laptop.

* remove `maxDecompressedDbRecordSize` - not actually used on recent
data since we store the framed format - also, we're in luck: we blew
past the limit quite some time ago
* fix obsolete exception-based error checking
* avoid `zeroMem` when reading from era store

see https://github.com/status-im/nim-snappy/pull/22 for benchmarks

* bump snappy
2023-07-26 10:47:46 +03:00
Eugene Kabanov df80ae68fa
REST server metrics and connections API endpoints. (#5193)
* Add new REST endpoints to monitor REST server connections and new chronos metrics.

* Bump head versions of chronos and presto.

* Bump chronos with regression fix.

* Remove outdated tests which was supposed to test pipeline mode.

* Disable pipeline mode in resttest.

* Update copyright year.

* Upgrade test_signing_node to start use AsyncProcess instead of std library's osproc.
Bump chronos to check graceful shutdown.

* Update AllTests.

* Bump chronos.
2023-07-21 16:54:09 +02:00
Jacek Sieka bf400d37ab
bump: ssz, stew
* https://github.com/status-im/nim-ssz-serialization/pull/53
* avoid some `zeroMem`
2023-07-21 15:21:59 +02:00
dependabot[bot] a18bd40208
Bump pygments from 2.12.0 to 2.15.0 in /docs (#5206)
Bumps [pygments](https://github.com/pygments/pygments) from 2.12.0 to 2.15.0.
- [Release notes](https://github.com/pygments/pygments/releases)
- [Changelog](https://github.com/pygments/pygments/blob/master/CHANGES)
- [Commits](https://github.com/pygments/pygments/compare/2.12.0...2.15.0)

---
updated-dependencies:
- dependency-name: pygments
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-20 14:25:45 +00:00
dependabot[bot] 1430981d25
Bump pygments from 2.11.2 to 2.15.0 in /ncli (#5205)
Bumps [pygments](https://github.com/pygments/pygments) from 2.11.2 to 2.15.0.
- [Release notes](https://github.com/pygments/pygments/releases)
- [Changelog](https://github.com/pygments/pygments/blob/master/CHANGES)
- [Commits](https://github.com/pygments/pygments/compare/2.11.2...2.15.0)

---
updated-dependencies:
- dependency-name: pygments
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-20 14:25:18 +00:00
Etan Kissling eb3a30655b
generalize `ShufflingRef` acceleration logic (#5197)
Split up the `ShufflingRef` acceleration logic into generically usable
parts and attester shuffling specific parts. The generic parts could be
used to accelerate other purposes, e.g., REST `/states/xxx/randao` API.
2023-07-20 10:25:39 +02:00
tersec 81c989660a
remove Bellatrix builder API support (#5162)
Co-authored-by: Etan Kissling <etan@status.im>
2023-07-20 04:37:54 +00:00
Etan Kissling 2954e7fac7
bump `nim-web3` to `9795b85a390ae02a931274ae92b3e67c8df6a0b8` (#5201)
- add `mixHash`, `withdrawals` fields to `BlockObject`
2023-07-20 02:02:12 +02:00
Etan Kissling da5bb96856
avoid Nim 2.0 issue when loading config (#5202)
To avoid Nim 2.0 issue https://github.com/nim-lang/Nim/issues/22284,
explicitly specify `ref SecondarySources` instead of using `auto`,
and add `{.raises.}` annotation
2023-07-19 20:26:30 +02:00
Zahary Karadjov 3eb1729d15
Fix a typo 2023-07-19 19:36:15 +03:00
Zahary Karadjov 60f0a2f6a6
Version 23.7.0 2023-07-19 15:35:49 +03:00
Etan Kissling 722b7f6f6b
move lint checks to separate job (#5200)
Running the lint checks separately allows running tests to check code
correctness even when targeting non-master branches or having outdated
copyright headers.
2023-07-19 13:55:23 +02:00
Etan Kissling 971b4483c5
add `libnimbus_lc.a` C library (#5122)
Add a new C library for processing light client data based on the Nimbus
implementation. This can be used from other, non-Nimbus components.
2023-07-19 09:48:39 +02:00
Etan Kissling 346d05a95d
increase `getPayloadFromSingleEL` timeout (#5194)
When producing a local block and `runProposalForkchoiceUpdated` was
missed, `getPayloadFromSingleEL` adds additional ~500ms of latency.
Quick fix to avoid missing blocks to that.
2023-07-18 23:29:50 +03:00
Jacek Sieka b3b5238434
disable startup pruning (#5191)
it has been shown to cause long startup times - a better strategy is
needed
2023-07-18 23:29:23 +03:00
Etan Kissling dd35d2d309
delay `onSlotEnd` if there are duties (#5196)
We currently call `onSlotEnd` whenever all in-BN validator duties are
completed. VC validator duties are not awaited. When `onSlotEnd` is
processed close to the slot start, a VC may therefore miss duties.
Adding a delay before `onSlotEnd` improves this situation.
The logic can be optimized further if `ActionTracker` would track
`knownValidators` from REST separately from in-process ones.
2023-07-18 20:55:36 +02:00
Etan Kissling f98c33ad03
generalize `commonAncestor` function to `BlockId` (#5192)
To enable additional use cases, e.g., `/states/###/randao` beacon API,
`ShufflingRef` acceleration logic needs to be able to operate on parts
of the DAG that do not have `BlockRef`. Changing `commonAncestor` to
act on `BlockId` instead of `BlockRef` is a step toward that and also
simplifies the logic some more.
2023-07-18 17:37:53 +02:00
tersec 3a818ecb93
fall back to non-fcu fork choice on epoch boundaries (#5195)
* fall back to non-fcu fork choice on epoch boundaries

* Future[bool]

* fix

* Update beacon_chain/consensus_object_pools/consensus_manager.nim

Co-authored-by: Etan Kissling <etan@status.im>

* make things consistent with Opt[void] return

---------

Co-authored-by: Etan Kissling <etan@status.im>
2023-07-17 22:30:38 +02:00
Miran 92da61ce88
update the RPi guide regarding overclocking (#5135) 2023-07-17 10:26:38 +02:00
Etan Kissling 2efc44a8ab
accelerate RANDAO computation for post-merge blocks (#5190)
Post-merge blocks contain all information to directly obtain RANDAO
without having to load any additional info. Take advantage of that to
further accelerate `ShufflingRef` computation. Note that it is still
necessary to verify that `blck` / `state` share a sufficiently recent
ancestor for the purpose of computing attester shufflings.

- new: 243.71s, 239.67s, 237.32s, 238.36s, 239.57s
- old: 251.33s, 234.29s, 249.28s, 237.03s, 236.78s
2023-07-15 22:16:56 +02:00
tersec 565edfa351
remove unused code (#5158)
* remove unused code

* add copyright header
2023-07-15 16:30:52 +00:00
Etan Kissling f1963bf599
allow initializing test DBs from later forks (#5186)
Running `makeTestDB` in tests currently always initializes DB with a
`phase0` state, preventing tests that configure a fork schedule that
starts in a different fork from working properly. Fix that by upgrading
the genesis state to whatever fork the fork schedule starts with.
2023-07-14 09:35:58 +02:00
tersec 703eead2e6
ensure the arrays in BeaconStateDiff are int-indexable in Nim 2.0 (#5189) 2023-07-13 23:25:10 +02:00
tersec c549fe6a65
bump nim-json-rpc to 32200345f80a3b5f72e15d1b4e9363363d95a1cd (#5187) 2023-07-12 22:44:17 +02:00
tersec 0be5b5b8f2
handle distinct nil more rigorously (#5180) 2023-07-12 17:33:19 +00:00
Etan Kissling 74bb4b1411
simplify RANDAO recovery in `ShufflingRef` acceleration (#5183)
Current RANDAO recovery logic is quite complex as it optimizes for the
minimum amount of database reads. Loading blocks isn't the bottleneck
though, so rather make the implementation more concise by avoiding the
complex strategy planning step. Note that this also prepares for an even
faster implementation for post-merge blocks in the future that extracts
RANDAO from `ExecutionPayload` directly if available, so even in cases
where efficiency is slightly lower, only historical data is affected.

`time nim c -r tests/test_blockchain_dag` (cached binary):

- new: 145.45s, 133.59s, 144.65s, 127.69s, 136.14s
- old: 149.15s, 150.84s, 135.77s, 137.49s, 133.89s
2023-07-12 17:27:05 +02:00
Jacek Sieka 0660ffcd3e
fix debug logging 2023-07-11 21:48:27 +02:00