* async batch verification
When batch verification is done, the main thread is blocked reducing
concurrency.
With this PR, the new thread signalling primitive in chronos is used to
offload the full batch verification process to a separate thread
allowing the main threads to continue async operations while the other
threads verify signatures.
Similar to previous behavior, the number of ongoing batch verifications
is capped to prevent runaway resource usage.
In addition to the asynchronous processing, 3 addition changes help
drive throughput:
* A loop is used for batch accumulation: this prevents a stampede of
small batches in eager mode where both the eager and the scheduled batch
runner would pick batches off the queue, prematurely picking "fresh"
batches off the queue
* An additional small wait is introduced for small batches - this helps
create slightly larger batches which make better used of the increased
concurrency
* Up to 2 batches are scheduled to the threadpool during high pressure,
reducing startup latency for the threads
Together, these changes increase attestation verification throughput
under load up to 30%.
* fixup
* Update submodules
* fix blst build issues (and a PIC warning)
* bump
---------
Co-authored-by: Zahary Karadjov <zahary@gmail.com>
* speed up state/block loading
When loading blocks and states from db/era, we currently redundantly
check their CRC32 - for a state, this costs 50ms of loading time
presently (110mb uncompressed size) on a decent laptop.
* remove `maxDecompressedDbRecordSize` - not actually used on recent
data since we store the framed format - also, we're in luck: we blew
past the limit quite some time ago
* fix obsolete exception-based error checking
* avoid `zeroMem` when reading from era store
see https://github.com/status-im/nim-snappy/pull/22 for benchmarks
* bump snappy
* Add new REST endpoints to monitor REST server connections and new chronos metrics.
* Bump head versions of chronos and presto.
* Bump chronos with regression fix.
* Remove outdated tests which was supposed to test pipeline mode.
* Disable pipeline mode in resttest.
* Update copyright year.
* Upgrade test_signing_node to start use AsyncProcess instead of std library's osproc.
Bump chronos to check graceful shutdown.
* Update AllTests.
* Bump chronos.
> sqlite3: better error message
> Use results Opt for discv5 getNode and resolve
> implement EIP-4844: Shard Blobs Transactions
> Enable tests with also json sink and fix failures
> Style fixes and comment improvements on uTP code
> fixes related to stint breaking changes
> Fix potentential uTP socket leak for socket in SynSent state
> Avoid double socket destroy on timed-out uTP SYN-ACK
> remove stddefects import
> use correct exception in `parseCmdArg`
* bump `nim-zlib` to `f34ca261efd90f118dc1647beefd2f7a69b05d93`
- ignore *.exe
- nim 1.6 adjustment
- update github action to v3
- use Nim 2.0 in CI
- test both refc and ORC in post-1.6 Nim versions
- remove Nim 1.2 and 1.4 support
* bump `nim-websock` for compatibility
`nim-websock` `fea05cde8b123b38d1a0a8524b77efbc84daa848` is needed
- reduce compiler warnings
- readFieldIMPL workaround when orc enabled
- remove nimble.lock
- remove nim 1.2 and nim 1.4 from ci
- remove Defect raises from makeFieldReadersTable
- another attempt to reduce compiler warnings
- More fix to exception tracking and generic_suite
- bump version to 0.2.0
- avoid implicit synthesis of `CaseTransition`
- make nimble file consistent with test matrix
- workaround for `NimYAML` issue
* Refactor api.nim to provide more informative failure reasons.
Distinct between unexpected data and unexpected code.
Deprecate Option[T] usage.
* Fix 400 for produceBlindedBlock().
Get proper string conversion for strategy.
* Fix SSZ encoded versions of ProduceBlockResponseV2, ProduceBlockResponseV2 can be received and decoded.
Fix done() warnings.
Bump presto.
* Fix compilation error with new presto.
Use TcpNoDelay option for Web3Signer.
* Fix produceBlockV2() should provide SSZ responses too.
* Address block encoding issue.
* Fix signing test.
* Bump presto.
* Address review comments.
- Generate lock file
- v0.10.3
- fix stylecheck to be compatible with Nim 1.2 and 1.6
- update to supported macOS GitHub CI image
- switch GitHub Actions CI images from deprecated Ubuntu 18.04 to 20.04
- Fix nim 2.0 bare except warnings
- fix nim 1.6 bare except warnings
- reduce compiler warnings
- Fix bare expect warning
- Remove appveyor and travis scripts
- upgrade github actions to v3
- remove nimble.lock, which is outdated and block Nim CI
- remove support for Nim v1.2 and v1.4
- use --mm: instead of --gc: and remove stddefects import
- annotate `parseCmdArg` with `{.raises.}`
- ignore exceptions while writing to stdout/stderr
- handle `terminalWidth` exception
- catch exceptions in `appInvocation`
- add support for generic union
- switch CI to the supported versions of ubuntu and macos
- install openssl on macos for Nim devel
- update github actions to v3
- improve union validation
- enable ssSize of union
- implement SSZ Optionals (EIP-6475)
- implement sszSize of Optionals
- use Nim 2.0 in CI
- test both refc and ORC in post-1.6 Nim versions
- stop testing Nim pre-1.6 in CI
- update for latest `Optional` spec
- avoid ptr_arith
- remove redundant when's because Nim 1.6+ is always true
- rm unused `SszMaxSizeExceeded`
- remove Defect raises
- fix nim devel error: 'result' requires explicit initialization
- Use Nim 2.0 in CI
- test both refc and ORC in post-1.6 Nim versions
- fix makeFieldReadersTable crash when orc enabled.
- remove appveyor and travis scripts
* Refactor nimbus_signing_node to support Unix signals.
* Fix SN unable to close REST server properly.
* Fix `keys`, `deposit` and `validator_registration` endpoints issues.
Add getValidatorExitSignature() and getDepositMessageSignature() to validator_pool.
* Add /reload endpoint and implementation.
Fix signData to not cancel `timer`.
Fix validator_pool should clear attachedValidators table.
* Diva protocol enhancement implementation.
* Support for driving multiple EL nodes from a single Nimbus BN
Full list of changes:
* Eth1Monitor has been renamed to ELManager to match its current
responsibilities better.
* The ELManager is no longer optional in the code (it won't have
a nil value under any circumstances).
* The support for subscribing for headers was removed as it only
worked with WebSockets and contributed significant complexity
while bringing only a very minor advantage.
* The `--web3-url` parameter has been deprecated in favor of a
new `--el` parameter. The new parameter has a reasonable default
value and supports specifying a different JWT for each connection.
Each connection can also be configured with a different set of
responsibilities (e.g. download deposits, validate blocks and/or
produce blocks). On the command-line, these properties can be
configured through URL properties stored in the #anchor part of
the URL. In TOML files, they come with a very natural syntax
(althrough the URL scheme is also supported).
* The previously scattered EL-related state and logic is now moved
to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim`
in a follow-up commit). State is assigned properly either to the
`ELManager` or the to individual `ELConnection` objects where
appropriate.
The ELManager executes all Engine API requests against all attached
EL nodes, in parallel. It compares their results and if there is a
disagreement regarding the validity of a certain payload, this is
detected and the beacon node is protected from publishing a block
with a potential execution layer consensus bug in it.
The BN provides metrics per EL node for the number of successful or
failed requests for each type Engine API requests. If an EL node
goes offline and connectivity is resoted later, we report the
problem and the remedy in edge-triggered fashion.
* More progress towards implementing Deneb block production in the VC
and comparing the value of blocks produced by the EL and the builder
API.
* Adds a Makefile target for the zhejiang testnet
* Local sim impovements
* Added support for running Capella and EIP-4844 simulations
by downloading the correct version of Geth.
* Added support for using Nimbus remote signer and Web3Signer.
Use 2 out of 3 threshold signing configuration in the mainnet
configuration and regular remote signing in the minimal one.
* The local testnet simulation can now use a payload builder.
This is currently not activated in CI due to lack of automated
procedures for installing third-party relays or builders.
You are adviced to use mergemock for now, but for most realistic
results, we can create a simple builder based on the nimbus-eth1
codebase that will be able to propose transactions from the regular
network mempool.
* Start the simulation from a merged state. This would allow us
to start removing pre-merge functionality such as the gossip
subsciption logic. The commit also removes the merge-forcing
hack installed after the TTD removal.
* Consolidate all the tools used in the local simulation into a
single `ncli_testnet` binary.
Other changes:
Renamed the `EIP_4844_FORK_*` config constants to `DENEB_FORK_*` as
this matches the latest spec and it's already used in the official
Sepolia config.
Jacek Sieka
tersec
henridf
Etan Kissling
Eugene Kabanov
Tanguy
zah
Kim De Mey
Zahary Karadjov
virchau13
Philippe Schommers