This PR allows sharing the pubkey data between validators by using a
thread-local cache for pubkey data, netting about a 400mb mem usage
reduction on holesky due to us keeping 3 permanent + several ephemeral
state copies in memory at all times and each state copy holding a full
validator.
The PR also introduces a hash cache for the key which gives ~14% speedup
for a full state `hash_tree_root` - the key makes up for a large part of
the `Validator` htr time.
Finally, the time it takes to copy a state goes down as well from ~80m
ms to ~60, for reasons similar to htr.
We use a `ptr` even if a `ref` could in theory have been used - there is
not much practical benefit to a `ref` (given it's mutable) while a `ptr`
is cheaper and easier to copy (when copying temporary states).
We could go further and cache a cooked pubkey but it turns out this is
quite intrusive - in all the relevant places, we're already using a
cooked key from the immutable validator data so there are no immediate
performance gains of doing so while managing the compressed -> cooked
key mapping would become more difficult - something for a future PR
perhaps.
Co-authored-by: Etan Kissling <etan@status.im>
- Refactor p2pProtocol internals
- Fix chronos related deprecated warnings in discv5 and uTP
- normalise nimble, ci
- Fix chronos related deprecated warnings in uTP code part II
- Remove usage of stew/shim/net
- Upgrade github actions to v4
- Fix two bugs in Receipts RLP encoding/decoding
- Move Ethereum specific RLP encoding tests under tests/common
- Reduce compiler warnings in rlp
- test refc in CI in Nim 2.0 and later
- test Nim 2.0 in CI and use non-EOL macOS version
- Use asyncraises in p2p
- Don't auto write p2pProtocol macro expansion to file
- Fix improper yield usage in rlpx and refine exception handling
- Client also handle error message if id is null
- Client pass meaningful error to newFut when processMessage failed
- Refactoring: extract rpc handler from HTTP and WebSocket server
- Use pragma push/pop pair to disable warning
- http server better exception handling
- Fix CI badge url
- Upgrade github actions to v4
- Revert "Fix CI badge url"
- HttpAuthHook use async raises
- Move CancelledError handling to outer try/except of RpcWebsocketServer
- Implement RPC batch call both in servers and clients
- v0.4.0
- Should compile if chronicles log turned on
- Add framework to support more optional types
- v0.4.2
- test refc in CI in Nim 2.0 and later
- use non-EOL macOS version for GitHub Actions CI
- avoid failing uninitialized `Future`
- Improve batch call example and wrapper comments
- Fix ws and socket client error handling and add test to #212
- Add build test with chronicles to json enabled
- update instructions for tracking upstream MIRACL Core
- bump `bls12-381-tests` to `v0.1.2`
- bump `miracl-core` to `0f67878bee7c4108405deb2b0b5e4e58d1ae30fc`
- test refc in CI in Nim 2.0 and later
- rename `milagro.nims` -> `miracl.nims`
- rename `milagro.nim` -> `miracl.nim`
- rename `milagro(Path|_func)` -> `miracl(Path|_func)`
- rename `milagro` references -> `miracl` in documentation
- Synchronize order of `blobGasUsed` / `excessBlobGas` in test
- Reduce compiler warnings
- Upgrade github actions to v4
- v0.3.2
- re-add support for `fromBlock`, `toBlock`, `blockHash` in `eth_getLogs`
- Feature flag for raises support
- HTTP server middleware implementation
- Fix examples documentation
- check leaks after every test
- deprecate `callback=`, UDP fixes
- Improve parseList and parseInlineTable strictness
- v0.2.10
- Switch to llvm-mingw for faster Windows CI
- Add configureTomlDeserialization to README.md
- Put array/inline table nonstandard behavior behind flag
- Unify parseList and parseArray implementation
- v0.2.12
- Fix a bunch of compiler hints and warnings in uTP and discv5
- Fix missing std/times import for the metrics 0.0.1 case
- Fix for uTP issues with latest chronos
- Clean-up, correct and clarify utp_protocol tests
- better async timeout wait
- Adjust test names and comments for `blobGasUsed` field
- Add data over multiple sockets uTP test
- Add uTP over discv5 test and small uTP performance improvements
Etan Kissling (2):
remove unused `skip0xPrefix`
keep the internal count helper
Will (1):
Bugfix/nully values (#61)
Yuriy Glukhov (5):
Contract constructor support
Fixed compilation error in exec function
Added string encoding
Fixed source->from field of EthCall
More flexibility to contract DSL, Async contract caller
jangko (5):
Reduce compiler warnings when using Nim v2
Migrate to json-serialization
Add tests of json rpc marshalled types
Resolve contract_dsl ambiguity
Event handler passing around JsonString instead of JsonNode
Share encoder between json-rpc and chronicles (#119)
Simplify generic constraint of rpc and chronicles encoders
Feature/execution api spec (#69)
v0.3.0
bump nim-json-rpc to a6475e49b26d3afc58aaa3d67621c94eafef8efb
coffeepots (1):
Use nim-json-serialization for RPCs (#172)
jangko (10):
Add copyright to source file
Remove StringOfJson
Fix optional parameter parsing fails in rpc macro with generics
Rename jrpc_sys module back to jsonmarshal
Reenable test hhtps
Add test for createRpcSigsFromNim and createSingleRpcSig
Let the OS choose the port for tests
Add onProcessMessage hook to client
Fix example in the README.md
Move errors module back to json_rpc folder
Upgrade rpc router internals (#178)
RPC server handle null return value correctly
v0.3.0
kdeme (1):
Add example test case that currently fails the Option parsing
- use still-supported versions of Ubuntu and macOS in GitHub Actions CI
- try py3 for autobahn
- Remove Nim 1.2
- Handle cancellation in close
- make nimble file consistent with CI setup
- replace `utils` module with direct `generate` calls
- allow manually triggering CI
- Fix grammatical errors of README.md
- Fix breaking changes introduces by upstream nim-serialization
- Remove unreachable code
- Fix cache key of Nim binaries
- Increase stack size of Windows
- Add note to increase stack size on Windows
- add `{.raises.}` annotation to `writeValue`
- add more missing `{.raises.}` annotations
- version 0.2.8
- Add table of contents to README.md
- Add JsonVoid, JsonNumber, and JsonValueRef to JsonWriter
- Add JsonWriter helper procs to README.md
- v0.2.2
- Improve nested structure depth limit check
- Fix JsonValueRef nil comparison
- Reduce declared but not used warning
- Restore accidentally deleted raises pragma
- Switch to mingw-llvm for faster Windows CI
- Increase stack size on Windows when running CI
This PR causes a few new warnings to appear - these are harmless but
will need addressing separately as they span several libraries.
* new asyncraises syntax
* asyncraises support in several modules
* `sink` usage reduces spurious copying
* `?` compatiblity for `async` + `results`
* remove `-d:chronosStrictException` (obsolete)
The `eth2-networks` repo often receives metadata updates with a delay.
Switch to `goerli` repo to obtain the latest config (Dencun scheduling)
when it is updated. This is in line with how Sepolia / Holesky work.
- https://github.com/eth-clients/goerli/pull/178
This requires all object types to be explicitly white-listed for
default serialization. The PR makes the minimal changes, although
a number of similar mechanisms in eth2_rest_serialization can now
be removed.
* Bump nim-eth: Change block timestamp from std.Time to distinct uint64
Also change tx.maxFeePerBlobGas from GasInt to UInt256
following Cancun latest spec
* Fix EthTime.now from func to proc due to sideeffects
- Add IP and location information to mainnet bootnodes
- Change IP and location of a Teku bootstrap node
- Add lighthouse's ipv6 compatible bootnodes
- Adding new Teku mainnet bootnodes
- Remove the ENRs from eth2 networks
* implement EIP-7514 for Deneb: Add Max Epoch Churn Limit
Cap activations per epoch according to EIP-7514:
- https://eips.ethereum.org/EIPS/eip-7514
- https://github.com/ethereum/consensus-specs/pull/3499
* apply proposer boost to first block in case of equivocation
Implement spec changes to fork choice; this only affects equivocation
when multiple blocks are signed for the same slot. Regular operation
is not changed.
- https://github.com/ethereum/consensus-specs/pull/3352
* bump test vectors to v1.4.0-beta.2-hotfix
---------
Co-authored-by: tersec <tersec@users.noreply.github.com>
The 'vendor/holesky/public-keys/all.txt' file is not used by Nimbus,
but sometimes fails to download due to payment issues in `@eth-clients`.
Skipping the download of that file should reduce that risk.
```
Downloading public-keys/all.txt (240 MB)
Error downloading object: public-keys/all.txt (ba977b5): Smudge error: Error downloading public-keys/all.txt (ba977b5b1da180914c934623fce187860e1b54cff939e6208533b2cb5f589e07): batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.
```
* bump nbs: Patch libnatpmp.a rule to support llvm-mingw in targets.mk
* bump nbs: Allow passing config to submodule initialization in `update-common`
* Add metadata for the Holesky network
* Add copyright banner to the new Nim module
* Working version
* Bump Chronos to fix downloading from Github
* Add checksum check of the downloaded file
* Clean up debugging code and obsolete imports
Suite names were not being used because `test` has to have access to it
during instantiation - this PR cleans things up a little while at the
same time upgrading unittest2.
* bump `nim-chronicles` to `71556f84af16cd275a8ae4c2390060481c6fedf7`
- remove Nim 1.2 and 1.4 support; test Nim 2.0 in CI
- suppress some more `IOError` when logging
* bump `nim-json-serialization` to `85b7ea093cb85ee4f433a617b97571bd709d30df`
- add `readValue` implementations for case objects
- add `{.raises.}` annotation to `writeValue`
```
Essential changes:
security bugfix in Go bindings
run-time switch in portable build
no-std support [exercised with limited stack size of 56K]
serde support
blst_miller_loop_n interface, with parallelized Rust and Go
counterparts
optional blst_fr_pentaroot subroutine
Emscripten bindings
hardened security on non-asm platforms
```
* move file loading out of timed ncli section to focus benchmark on
deserialization rather than disk loading
* JSON loading fix included in bump, but does not practically happen
since we generally don't load hashlist from JSON
* async batch verification
When batch verification is done, the main thread is blocked reducing
concurrency.
With this PR, the new thread signalling primitive in chronos is used to
offload the full batch verification process to a separate thread
allowing the main threads to continue async operations while the other
threads verify signatures.
Similar to previous behavior, the number of ongoing batch verifications
is capped to prevent runaway resource usage.
In addition to the asynchronous processing, 3 addition changes help
drive throughput:
* A loop is used for batch accumulation: this prevents a stampede of
small batches in eager mode where both the eager and the scheduled batch
runner would pick batches off the queue, prematurely picking "fresh"
batches off the queue
* An additional small wait is introduced for small batches - this helps
create slightly larger batches which make better used of the increased
concurrency
* Up to 2 batches are scheduled to the threadpool during high pressure,
reducing startup latency for the threads
Together, these changes increase attestation verification throughput
under load up to 30%.
* fixup
* Update submodules
* fix blst build issues (and a PIC warning)
* bump
---------
Co-authored-by: Zahary Karadjov <zahary@gmail.com>
* speed up state/block loading
When loading blocks and states from db/era, we currently redundantly
check their CRC32 - for a state, this costs 50ms of loading time
presently (110mb uncompressed size) on a decent laptop.
* remove `maxDecompressedDbRecordSize` - not actually used on recent
data since we store the framed format - also, we're in luck: we blew
past the limit quite some time ago
* fix obsolete exception-based error checking
* avoid `zeroMem` when reading from era store
see https://github.com/status-im/nim-snappy/pull/22 for benchmarks
* bump snappy
* Add new REST endpoints to monitor REST server connections and new chronos metrics.
* Bump head versions of chronos and presto.
* Bump chronos with regression fix.
* Remove outdated tests which was supposed to test pipeline mode.
* Disable pipeline mode in resttest.
* Update copyright year.
* Upgrade test_signing_node to start use AsyncProcess instead of std library's osproc.
Bump chronos to check graceful shutdown.
* Update AllTests.
* Bump chronos.
> sqlite3: better error message
> Use results Opt for discv5 getNode and resolve
> implement EIP-4844: Shard Blobs Transactions
> Enable tests with also json sink and fix failures
> Style fixes and comment improvements on uTP code
> fixes related to stint breaking changes
> Fix potentential uTP socket leak for socket in SynSent state
> Avoid double socket destroy on timed-out uTP SYN-ACK
> remove stddefects import
> use correct exception in `parseCmdArg`
* bump `nim-zlib` to `f34ca261efd90f118dc1647beefd2f7a69b05d93`
- ignore *.exe
- nim 1.6 adjustment
- update github action to v3
- use Nim 2.0 in CI
- test both refc and ORC in post-1.6 Nim versions
- remove Nim 1.2 and 1.4 support
* bump `nim-websock` for compatibility
`nim-websock` `fea05cde8b123b38d1a0a8524b77efbc84daa848` is needed
- reduce compiler warnings
- readFieldIMPL workaround when orc enabled
- remove nimble.lock
- remove nim 1.2 and nim 1.4 from ci
- remove Defect raises from makeFieldReadersTable
- another attempt to reduce compiler warnings
- More fix to exception tracking and generic_suite
- bump version to 0.2.0
- avoid implicit synthesis of `CaseTransition`
- make nimble file consistent with test matrix
- workaround for `NimYAML` issue
* Refactor api.nim to provide more informative failure reasons.
Distinct between unexpected data and unexpected code.
Deprecate Option[T] usage.
* Fix 400 for produceBlindedBlock().
Get proper string conversion for strategy.
* Fix SSZ encoded versions of ProduceBlockResponseV2, ProduceBlockResponseV2 can be received and decoded.
Fix done() warnings.
Bump presto.
* Fix compilation error with new presto.
Use TcpNoDelay option for Web3Signer.
* Fix produceBlockV2() should provide SSZ responses too.
* Address block encoding issue.
* Fix signing test.
* Bump presto.
* Address review comments.
- Generate lock file
- v0.10.3
- fix stylecheck to be compatible with Nim 1.2 and 1.6
- update to supported macOS GitHub CI image
- switch GitHub Actions CI images from deprecated Ubuntu 18.04 to 20.04
- Fix nim 2.0 bare except warnings
- fix nim 1.6 bare except warnings
- reduce compiler warnings
- Fix bare expect warning
- Remove appveyor and travis scripts
- upgrade github actions to v3
- remove nimble.lock, which is outdated and block Nim CI
- remove support for Nim v1.2 and v1.4
- use --mm: instead of --gc: and remove stddefects import
- annotate `parseCmdArg` with `{.raises.}`
- ignore exceptions while writing to stdout/stderr
- handle `terminalWidth` exception
- catch exceptions in `appInvocation`
- add support for generic union
- switch CI to the supported versions of ubuntu and macos
- install openssl on macos for Nim devel
- update github actions to v3
- improve union validation
- enable ssSize of union
- implement SSZ Optionals (EIP-6475)
- implement sszSize of Optionals
- use Nim 2.0 in CI
- test both refc and ORC in post-1.6 Nim versions
- stop testing Nim pre-1.6 in CI
- update for latest `Optional` spec
- avoid ptr_arith
- remove redundant when's because Nim 1.6+ is always true
- rm unused `SszMaxSizeExceeded`
- remove Defect raises
- fix nim devel error: 'result' requires explicit initialization
- Use Nim 2.0 in CI
- test both refc and ORC in post-1.6 Nim versions
- fix makeFieldReadersTable crash when orc enabled.
- remove appveyor and travis scripts
* Refactor nimbus_signing_node to support Unix signals.
* Fix SN unable to close REST server properly.
* Fix `keys`, `deposit` and `validator_registration` endpoints issues.
Add getValidatorExitSignature() and getDepositMessageSignature() to validator_pool.
* Add /reload endpoint and implementation.
Fix signData to not cancel `timer`.
Fix validator_pool should clear attachedValidators table.
* Diva protocol enhancement implementation.
* Support for driving multiple EL nodes from a single Nimbus BN
Full list of changes:
* Eth1Monitor has been renamed to ELManager to match its current
responsibilities better.
* The ELManager is no longer optional in the code (it won't have
a nil value under any circumstances).
* The support for subscribing for headers was removed as it only
worked with WebSockets and contributed significant complexity
while bringing only a very minor advantage.
* The `--web3-url` parameter has been deprecated in favor of a
new `--el` parameter. The new parameter has a reasonable default
value and supports specifying a different JWT for each connection.
Each connection can also be configured with a different set of
responsibilities (e.g. download deposits, validate blocks and/or
produce blocks). On the command-line, these properties can be
configured through URL properties stored in the #anchor part of
the URL. In TOML files, they come with a very natural syntax
(althrough the URL scheme is also supported).
* The previously scattered EL-related state and logic is now moved
to `eth1_monitor.nim` (this module will be renamed to `el_manager.nim`
in a follow-up commit). State is assigned properly either to the
`ELManager` or the to individual `ELConnection` objects where
appropriate.
The ELManager executes all Engine API requests against all attached
EL nodes, in parallel. It compares their results and if there is a
disagreement regarding the validity of a certain payload, this is
detected and the beacon node is protected from publishing a block
with a potential execution layer consensus bug in it.
The BN provides metrics per EL node for the number of successful or
failed requests for each type Engine API requests. If an EL node
goes offline and connectivity is resoted later, we report the
problem and the remedy in edge-triggered fashion.
* More progress towards implementing Deneb block production in the VC
and comparing the value of blocks produced by the EL and the builder
API.
* Adds a Makefile target for the zhejiang testnet
* Local sim impovements
* Added support for running Capella and EIP-4844 simulations
by downloading the correct version of Geth.
* Added support for using Nimbus remote signer and Web3Signer.
Use 2 out of 3 threshold signing configuration in the mainnet
configuration and regular remote signing in the minimal one.
* The local testnet simulation can now use a payload builder.
This is currently not activated in CI due to lack of automated
procedures for installing third-party relays or builders.
You are adviced to use mergemock for now, but for most realistic
results, we can create a simple builder based on the nimbus-eth1
codebase that will be able to propose transactions from the regular
network mempool.
* Start the simulation from a merged state. This would allow us
to start removing pre-merge functionality such as the gossip
subsciption logic. The commit also removes the merge-forcing
hack installed after the TTD removal.
* Consolidate all the tools used in the local simulation into a
single `ncli_testnet` binary.
Other changes:
Renamed the `EIP_4844_FORK_*` config constants to `DENEB_FORK_*` as
this matches the latest spec and it's already used in the official
Sepolia config.
* Working Makefile targets for Capella devnet2
make capella-devnet-2
make clean-capella-devnet-2
You'll need to have https://github.com/tmuxinator/tmuxinator installed.
It's available as a regular package in most Linux distributions or through
Nix or Brew on macOS.
This commit also fixes the initial hang in the Eth1 monitor in the "find
TTD block" procedure through a fix to the network metadata files which
hasn't been upstreamed yet.
Other changes:
* Disabled Geth snap sync in the simulation
When all Geth nodes are configured to run with snap sync enabled, they all
start snap sync after the first forkchoiceUpdated which causes the BNs to
skip validator duties because the EL is syncing. The snap sync never completes
due to poor connectivity between the Geth nodes in the simulation.
Introduce (optional) pruning of historical data - a pruned node will
continue to answer queries for historical data up to
`MIN_EPOCHS_FOR_BLOCK_REQUESTS` epochs, or roughly 5 months, capping
typical database usage at around 60-70gb.
To enable pruning, add `--history=prune` to the command line - on the
first start, old data will be cleared (which may take a while) - after
that, data is pruned continuously.
When pruning an existing database, the database will not shrink -
instead, the freed space is recycled as the node continues to run - to
free up space, perform a trusted node sync with a fresh database.
When switching on archive mode in a pruned node, history is retained
from that point onwards.
History pruning is scheduled to be enabled by default in a future
release.
In this PR, `minimal` mode from #4419 is not implemented meaning
retention periods for states and blocks are always the same - depending
on user demand, a future PR may implement `minimal` as well.
Since the sync committee duties are no longer updated on every slot
and previously the sync committee aggregators selection proofs were
generated during the duties update, this now resulted in the client
using stale selection proofs (they must be generated at each slot).
The fix consists of moving the selection proof generation logic in
a different function which is properly executed on each slot.
Other changes:
* The logtrace tool has been enhanced with a framework for adding
new simpler log aggregation and analysis algorithms.
The default CI testnet simulation will now ensure that the blocks
in the network have reasonable sync committee participation.
Jacek Sieka
Etan Kissling
tersec
Eugene Kabanov
andri lim
diegomrsantos
zah
4rgon4ut
Tanguy
henridf
Kim De Mey
Zahary Karadjov
virchau13
Philippe Schommers