Commit Graph

1003 Commits

Author SHA1 Message Date
tersec 68e08c6f15
use v1.3.0-rc.1 consensus spec test vectors (#4505) 2023-01-14 04:20:51 +00:00
zah 0f758c5f02
Working Makefile targets for Capella devnet2 (#4494)
* Working Makefile targets for Capella devnet2

make capella-devnet-2
make clean-capella-devnet-2

You'll need to have https://github.com/tmuxinator/tmuxinator installed.
It's available as a regular package in most Linux distributions or through
Nix or Brew on macOS.

This commit also fixes the initial hang in the Eth1 monitor in the "find
TTD block" procedure through a fix to the network metadata files which
hasn't been upstreamed yet.

Other changes:

* Disabled Geth snap sync in the simulation

When all Geth nodes are configured to run with snap sync enabled, they all
start snap sync after the first forkchoiceUpdated which causes the BNs to
skip validator duties because the EL is syncing. The snap sync never completes
due to poor connectivity between the Geth nodes in the simulation.
2023-01-13 12:21:58 +02:00
Zahary Karadjov b06502bf65
Gnosis const preset 2023-01-13 04:28:29 +02:00
henridf 309f8690de
Wire up engine_newPayloadV3 (#4482)
* Wire up eip4844's newPayloadV3

* Add eip4844 test

* Update AllTests-mainnet.md and fix typo
2023-01-11 18:21:19 +00:00
tersec 787703bd0c
use v1.3.0-rc.0 consensus spec test vectors (#4472) 2023-01-09 15:15:43 +01:00
Jacek Sieka 0ba9fc4ede
History pruning (fixes #4419) (#4445)
Introduce (optional) pruning of historical data - a pruned node will
continue to answer queries for historical data up to
`MIN_EPOCHS_FOR_BLOCK_REQUESTS` epochs, or roughly 5 months, capping
typical database usage at around 60-70gb.

To enable pruning, add `--history=prune` to the command line - on the
first start, old data will be cleared (which may take a while) - after
that, data is pruned continuously.

When pruning an existing database, the database will not shrink -
instead, the freed space is recycled as the node continues to run - to
free up space, perform a trusted node sync with a fresh database.

When switching on archive mode in a pruned node, history is retained
from that point onwards.

History pruning is scheduled to be enabled by default in a future
release.

In this PR, `minimal` mode from #4419 is not implemented meaning
retention periods for states and blocks are always the same - depending
on user demand, a future PR may implement `minimal` as well.
2023-01-07 10:02:15 +00:00
tersec d1b799eb64
bump nim-web3 for correct getPayloadV2 response signature (#4471) 2023-01-06 21:06:45 +00:00
Tanguy 75d0e0d4c8
Bump libp2p (#4470) 2023-01-06 19:33:14 +00:00
Etan Kissling 2184fd2322
bump `nim-eth`, extend empty block fallback for EIP4844 (#4425)
Implements `emptyPayloadToBlockHeader` for EIP-4844.
https://github.com/status-im/nim-eth/pull/570
2022-12-20 20:00:56 +01:00
zah 07d4160e00
Migrating the deposit contract snapshot can no longer fail on start-up (#4438)
The missing piece of data that had to be obtained previously from
the configured EL client is now part of the network metadata baked
into the binary.
2022-12-19 18:19:48 +01:00
tersec d269d82b71
use v1.3.0-alpha.2 EF consensus spec test vectors (#4430) 2022-12-15 01:51:06 +00:00
Jacek Sieka f3305189f1
bump & ci fixes (#4409) 2022-12-09 17:06:29 +01:00
Jacek Sieka 55a95bca3c
bump secp256k1 (#4388)
this time taking into account arm vs x86_64
2022-12-05 16:58:05 +00:00
tersec f1584eef2f
bump nim-eth to reduce undefined behavior from Nim (#4384) 2022-12-02 08:36:42 +01:00
zah 32039e95f0
Switch to Nim 1.6.11 (#4380) 2022-12-01 08:05:47 +02:00
Zahary Karadjov 7dd4718735
Version 22.11.0 2022-12-01 07:38:54 +02:00
Jacek Sieka cd160b5650
more strict read-only database mode (#4362)
* avoid creating pre-altair backwards compatibility tables
* allow running ncli_db era export without above tables present
* drop unused pre-altair backwards compatibility tables
* run benchmark on read-ronly database
* fix running benchmark from genesis
2022-11-28 23:21:58 +00:00
Etan Kissling c941dc801f
bump `nim-eth`, extend empty block fallback for Capella (#4357)
Implements `emptyPayloadToBlockHeader` for Capella.
https://github.com/status-im/nim-eth/pull/562
2022-11-28 14:41:25 +01:00
tersec b7cd0fe4a1
Revert "bump secp256k1 (#4356)" (#4359)
This reverts commit 60e01dcf78.
2022-11-25 13:07:16 +00:00
Jacek Sieka 60e01dcf78
bump secp256k1 (#4356)
See
eb5868e069
for changes
2022-11-25 10:51:20 +00:00
Jacek Sieka 04392c236b
bumps (#4352)
* build&style fixes
* use multithreading for LTO outside of make too
* blst 0.3.10 with no significant changes
2022-11-24 20:56:02 +00:00
tersec e965d5e0e9
bump nim-web3 to get withdrawal support in Engine API call sigs/types (#4348) 2022-11-24 09:52:45 +02:00
zah d07113767d
Bugfix: The VC was producing invalid sync committee contributions (#4343)
Since the sync committee duties are no longer updated on every slot
and previously the sync committee aggregators selection proofs were
generated during the duties update, this now resulted in the client
using stale selection proofs (they must be generated at each slot).

The fix consists of moving the selection proof generation logic in
a different function which is properly executed on each slot.

Other changes:

* The logtrace tool has been enhanced with a framework for adding
  new simpler log aggregation and analysis algorithms.
  The default CI testnet simulation will now ensure that the blocks
  in the network have reasonable sync committee participation.
2022-11-24 09:46:35 +02:00
tersec 1146470f7d
use v1.3.0-alpha.1 consensus spec test vectors (#4338) 2022-11-21 08:44:49 +01:00
Jacek Sieka cc1464a935
readd bearssl/cacert 2022-11-20 08:20:23 +01:00
Jacek Sieka 64322a3d45
prater: update CONFIG_NAME (#4322) 2022-11-15 17:35:48 +00:00
Jacek Sieka 367a13df34
stew: bump (#4323)
fix Opt constants, cleanups
2022-11-11 18:07:34 +00:00
Kim De Mey 2256a22594
Bump nim-eth for prettier logs in discv5 (#4312) 2022-11-10 16:37:37 +01:00
Jacek Sieka b170a09c47
remove `news` leftovers (#4299) 2022-11-08 20:06:54 +00:00
Jacek Sieka 38651d1bfe
bump nim-eth (#4300)
* removal of eth1 code
* various discovery fixes
* nim-devel fixes
2022-11-08 16:45:56 +00:00
Jacek Sieka 02b48fafad
remove unused / obsolete / unmaintained modules (#4298)
mostly this is chronicles-tail and its (extensive) dependencies along
with the simulation monitoring dashboard that is not maintained
2022-11-08 14:37:23 +01:00
Jacek Sieka e98cfa88cb
bumps (#4297)
* fix toml overflow handling
* random nim devel fixes
* unicode db update
2022-11-08 10:56:06 +00:00
Tanguy 865a930f65
Bump libp2p (#4282) 2022-11-03 20:22:53 +01:00
Jacek Sieka fc724b21e8
Tighten libp2p request quotas (#4254)
To further tighten Nimbus against spam, this PR introduces a global
quota for block requests (shared between peers) as well as a general
per-peer request limit that applies to all libp2p requests.

* apply request quota before decoding message
* for high-bandwidth requests (blocks), apply a shared global quota
which helps manage bandwidth for high-peer setups
* add metrics
2022-11-02 10:46:53 +00:00
tersec 7dd5c49c4e
use v1.3.0-alpha.0 test vectors (#4263) 2022-10-27 11:54:39 +00:00
Jacek Sieka 36e2518d79
fakeee: Increase incoming POST size (#4252)
Needed to handle payloads
2022-10-25 20:01:45 +00:00
Jacek Sieka 593b3cee20
blscurve: bump (fixes #4237) 2022-10-21 02:24:02 +02:00
zah a47cf5aa2c
Flake-based build environment for Nix users (#3534) 2022-10-15 03:46:30 +02:00
Zahary Karadjov 89582c10db
Merge branch 'stable' into unstable 2022-10-14 22:24:55 +03:00
cheatfate b54a03ae07
Fix MEV builder file descriptor leaks. 2022-10-14 16:09:39 +03:00
Zahary Karadjov fb983f867f
Merge branch 'stable' into unstable 2022-10-13 14:48:03 +03:00
Zahary Karadjov 5ff99b9bf0
Bump Chronos to fix a resource leak when using an external builder 2022-10-13 13:46:11 +03:00
Etan Kissling 2b531b6653
fix `nimbus-build-system` checkout (#4233)
Currently, `unstable` has issues checking out from Git, as
https://github.com/status-im/nimbus-build-system/pull/52 has merged and
https://github.com/status-im/nimbus-eth2/pull/4215
was pointing to the PR commit instead of the merged commit.
Selecting the corresponding merged commit to fix the issue.
2022-10-12 16:41:25 -05:00
Jakub Sokołowski 20c9c9b381
nimbus-build-system: allow for overriding CC (#4215)
Depends on: https://github.com/status-im/nimbus-build-system/pull/51

Signed-off-by: Jakub Sokołowski <jakub@status.im>

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2022-10-04 19:17:43 +03:00
Jacek Sieka 3c1caab107
web3: bump (#4183)
* point to master branch again
2022-09-27 14:25:26 +02:00
tersec 72e6b2021a
use v1.2.0 consensus spec test vectors (#4163) 2022-09-22 22:24:13 +00:00
Jacek Sieka c146ea823c
bump web3 (#4156)
* fixes `BlockObject` definition
* fixes JSON-RPC Option support
2022-09-22 11:15:57 +00:00
Etan Kissling 5a32dacfc5
correct `nim-presto` bump (#4145)
#4139 linked to the commit on the feature branch itself instead of the
commit merged to `master`. Fix dependency reference.
2022-09-19 16:27:58 +00:00
Eugene Kabanov ca871a5435
Fix HTTP/REST clients HTTP Content-Type header parsers. (#4139)
* Fix client HTTP content-type parsers.

* Fix tests.

* Address review comment and apply wildcard checks for generic decodeBytes.
2022-09-19 12:17:29 +03:00
Etan Kissling 6ad9b3214d
fix random `IndexError` in CI (#4136)
Bump `nim-chronos` to improve reliability of tests with `waitFor` logic.
The bump also adds more `FutureBase` helpers.
2022-09-17 19:45:09 +00:00
Etan Kissling 24534090b4
send SNI info when using https/wss web3 API (#4133)
When connecting to hosts on shared IP/Port using TLS, SNI must be sent
to allow the remote server to provide the correct TLS certificate.
Bump the `nim-json-rpc` and `nim-websock` dependencies to send SNI ext.
2022-09-17 03:01:59 +00:00
Jacek Sieka 9df08576a1
bump json-rpc, for potential `nil` http crash (fixes 4118) (#4125)
* bump json-rpc, for potential `nil` http crash (fixes #4118)

* bump
2022-09-16 08:08:25 +00:00
Etan Kissling 634408ff2c
use `nim-websock` instead of `news` (#4061)
`news` has a few open issues that are not present in `nim-websock`:
1. There is a 1 second delay between each MB of sent data.
2. Cancelling an ongoing `send` makes the entire WebSocket unusable.
3. Control packets do not have priority over ongoing message frames.

Using `news`, there are quite a few of these messages in Geth:
```
Previously seen beacon client is offline. Please ensure it is
operational to follow the chain!
```
It may take quite some time to reconnect when this happens.

Using `nim-websock`, this message still occurs because `eth1_monitor`
reconnects the EL connection when no new blocks occurred for 5 minutes,
but reconnecting is quick and the message is rarer.
2022-09-06 23:41:33 +02:00
Jacek Sieka 1538dea45e
bumps (#4069)
* bearssl: use master branch again
* chronos: less Option, better contenttype handling
* eth, presto: less imports
* stew: typo
* web3: tighter `Quantity` parsing rules (see
https://github.com/status-im/nimbus-eth2/pull/3850 and subsequently
https://github.com/status-im/nim-web3/pull/55)
2022-09-06 13:54:30 +00:00
Tanguy 2da13c0b22
Bump libp2p (#4066) 2022-09-05 20:05:36 +02:00
Jacek Sieka d9ceb61dbd
eth: bump (#4062) 2022-09-04 19:44:43 +02:00
tersec f33a241ba1
retry EF spec test vector downloads (#4059) 2022-09-01 19:51:19 +00:00
Etan Kissling ebfb624557
bump `news` to fix concurrency bug with >1MB frames (#4028)
When calling `newPayload` on a >1MB payload (can happen post-merge),
`news` splits up that payload into 1MB chunks. The chunks are each sent
individually, though, with `await` in-between. This means that when we
send concurrent `forkChoiceUpdated` calls, that those may end up getting
in-between the `newPayload` chunks, leading to invalid data being sent.
The EL then returns an error message with a `null` `id` entry (as it
could not read the request `id` due to the mangling) and disconnects.
A PR has been submitted to fix this in `news`, and merged into `status`
branch early as this fix is critical for reliable post-merge operation:
https://github.com/Tormund/news/pull/22
2022-08-25 21:14:40 +00:00
Jacek Sieka 492a2ccfac
bump ssz-serialization (#4017)
* free 7% speedup of `hash_tree_root`
* harden `len` for uninitialized `BitSeq`
2022-08-23 18:32:34 +00:00
zah a7192f5d6c
Fix the block header computation when proposing an empty execution payload (#3991)
* Fix the block header computation when proposing an empty execution payload
* Spec compliant base fee calculation when producing empty payloads
2022-08-19 13:28:42 +03:00
tersec b5b93e90c0
use v1.2.0-rc.3 test vectors (#3995) 2022-08-19 04:32:53 +00:00
Zahary Karadjov 7f681af56f
Make it easier to test trustedNodeSync or different networks 2022-08-18 19:48:11 +03:00
zah 74819729a5
Bellatrix metadata for Mainnet (#3980) 2022-08-17 14:40:58 +00:00
tersec e732b080f7
use v1.2.0-rc.2 test vectors (#3975) 2022-08-16 21:24:37 +00:00
zah dc50abbc90
Implement a missing ingnore rule for sync committee contributions (#3941) 2022-08-09 12:52:11 +03:00
Eugene Kabanov 250f7b4bdf
Exclusive keystore locking (#3907) 2022-08-08 00:53:20 +03:00
Eugene Kabanov fe5435ed96
Address #3624 (#3934) 2022-08-06 11:55:40 +00:00
Zahary Karadjov 5951df4193
Switch back to Nim 1.2
We've discovered a critical `lent` issue affecting Nim 1.6:
https://github.com/nim-lang/Nim/issues/20107
2022-07-29 11:26:36 +03:00
tersec b7f907a64a
bump nim-zxcvbn for styleCheck compatibility (#3919) 2022-07-27 14:25:09 +03:00
tersec 9d41fbfded
bump nim-presto for styleCheck compatibility (#3918) 2022-07-27 14:24:45 +03:00
Zahary Karadjov 4637669b6b
Merge branch 'stable' into unstable 2022-07-26 23:54:46 +03:00
tersec b7ed0aa235
bump nim-websock for styleCheck compatibility (#3909) 2022-07-26 12:20:36 +03:00
tersec ee24beee30
bump nim-stint for styleCheck compatibility (#3908) 2022-07-26 12:19:55 +03:00
zah c5d89bb46e
Switch to Nim 1.6.7 (#3902) 2022-07-26 12:19:32 +03:00
tersec 6b2abd01fb
bump nim-eth for styleCheck compatibility (#3896) 2022-07-25 23:15:46 +03:00
Zahary Karadjov 1a1553fee8
Bump eth2-networks to include the Prater TTD 2022-07-21 22:30:17 +03:00
Jacek Sieka 24d8401054
bumps (#3892)
various ABI fixes
2022-07-20 20:20:34 +02:00
tersec df660292a4
bump nim-metrics for styleCheck compatibility (#3889) 2022-07-20 10:13:32 +02:00
tersec 3dc04bc48f
bump nimcrypto for styleCheck compatibility (#3885) 2022-07-18 17:13:25 +00:00
Jacek Sieka 7dff72c6bf
bumps (#3882)
* fix UB in leb128 shift
* bearssl spellings
2022-07-18 07:17:53 +02:00
zah e6988c1cc9
Bump nim-json-serialization (#3878) 2022-07-15 14:36:03 +00:00
zah 20d45e69b5
Re-enabled requireAllFields after a fix in nim-json-serialization (#3871)
* Re-enabled requireAllFields after a fix in nim-json-serialization

The problem was that `Option[T]` fields were not treated as optional
when requireAllFields is set to true. This is now fixed in NJS.

* Add makefile targets for recreating the Jenkins simulation runs

* Fix a discrepancy with the REST spec
2022-07-15 03:19:19 +03:00
Eugene Kabanov 263a2ffa14
Validator client various fixes. (#3840)
* Improve fallback_service.
* Fix nextAction negative time issue.
* Improve logging in fallback_service.
* Improve logging in sync_committee_service.
* Prepare all services for cancellation.
* Signals handlers for validator client
* Address #3800

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2022-07-13 17:43:57 +03:00
tersec 91b51adb12
Revert "bump nim-web3 to enforce JSON-RPC Quantity syntax (#3846)" (#3850)
This reverts commit 007dcc42f1.
2022-07-08 00:07:50 +00:00
tersec 007dcc42f1
bump nim-web3 to enforce JSON-RPC Quantity syntax (#3846) 2022-07-07 21:14:41 +02:00
Jacek Sieka e1830519a4
Introduce message router (#3829)
Whether new blocks/attestations/etc are produced internally or received
via REST, their journey through the node is the same - to ensure that
they get the same treatment (logging, metrics, processing), this PR
moves the routing to a dedicated module and fixes several small
differences that existed before.

* `xxxValidator` -> `processMessageName` - the processor also was adding
messages to pools, so we want the name to reflect that action
* add missing "sent" metrics for some messages
* document ignore policy better - already-seen messages are not actaully
rebroadcast by libp2p
* skip redundant signature checks for internal validators consistently
2022-07-06 16:11:44 +00:00
tersec 9eb1a3efb3
bump nim-json-serialization to fix Nim 1.6 build warning (#3835) 2022-07-01 22:27:40 +00:00
Tanguy 4fbbbfd462
Bump nim-websock: fix crash when sending >1mb (#3831) 2022-07-01 15:11:45 +03:00
zah f08f9e2bd7
Bump nim-json-rpc and nim-websock (use news by default) (#3820) 2022-06-29 17:35:09 +03:00
tersec 2d2cda735b
bump nim-eth to fix Nim 1.6 warnings (#3818) 2022-06-28 22:56:48 +02:00
zah 2c3e47d7e6
Avoid overflow in the initial 'nextExchangeTransitionConfTime' calculation (#3809) 2022-06-28 23:50:40 +03:00
Zahary Karadjov 45ad4abe30
TTD value for Sepolia 2022-06-28 19:25:11 +03:00
Mamy Ratsimbazafy c4e5cd9f06
update nim blscurve: use builtin EIP2333 for BLST https://github.com/status-im/nim-blscurve/pull/142 2022-06-28 15:11:20 +02:00
tersec 5145c04be2
Revert "Switch back to nim-websock (#3665)" (#3812)
This reverts commit 8d3afd8285.
2022-06-27 22:21:30 +00:00
Etan Kissling 07f9936b33
bump nim-ssz-serialization to `f1b14875792df7b1e76c98c9ee669026d7cfe6bb` (#3802)
Update nim-ssz-serialization to get rid of a style check warning.
2022-06-26 23:17:40 +00:00
tersec c02c383b91
bump nim-web3 so default payload status is safe (#3801) 2022-06-26 18:31:32 +00:00
Etan Kissling aa1b8e4a17
bump nim-ssz-serialization to `3db6cc0f282708aca6c290914488edd832971d61` (#3119)
This updates `nim-ssz-serialization` to
`3db6cc0f282708aca6c290914488edd832971d61`.

Notable changes:
- Use `uint64` for `GeneralizedIndex`
- Add support for building merkle multiproofs
2022-06-26 19:33:06 +02:00
Jacek Sieka b764f4f0d7
bump bearssl (#3797)
cross compile path fix
2022-06-24 22:45:26 +00:00
Tanguy c45c017349
Bump libp2p to fix write on closed channel (#3795) 2022-06-24 12:52:10 +00:00
Ivan Yonchovski d701887a9f
Bump libbacktrace (#3794) 2022-06-24 07:04:29 +02:00
Mamy Ratsimbazafy e25cfe56a6
bump blst fix 32-bit builds on ADX CPUs (#3789) 2022-06-23 16:45:43 +02:00
tersec 4bb42eea7d
bump nim-web3 for eth_chainId support (#3785) 2022-06-21 21:09:29 +00:00
Tanguy 8d3afd8285
Switch back to nim-websock (#3665) 2022-06-21 17:43:55 +02:00
Jacek Sieka f3852cc4cd
sqlite: bump (#3783)
new nimterop
2022-06-21 15:56:48 +02:00
Jacek Sieka 347a485b5b
bearssl: split abi (#3755) 2022-06-21 10:29:16 +02:00
Eugene Kabanov eb6b7affee
Add the `execution_optimistic` flag to REST API responses. (#3780)
* Initial commit

* Make `events` API spec compliant.

* Add `Eth-Consensus-Version` in responses.

* Bump chronos to get redirect with headers working.

* Add `is_optimistic` field and handling to syncing RestSyncInfo.
2022-06-20 08:53:39 +03:00
zah c24c737866
Fix #3650 (participation format in BeaconState result is out of spec) (#3776)
* Fix #3650 (participation format in BeaconState result is out of spec)
* Make EpochParticipationFlags a distinct type
2022-06-20 08:38:56 +03:00
Eugene Kabanov 5e7b28cd66
AsyncEventBus missing attestations fix. (#3664) 2022-06-17 18:27:28 +03:00
tersec b0e46686f1
bump merge-testnets to use capella fork versions consistently (#3767) 2022-06-17 17:17:09 +03:00
zah 69f505e2ba
Correct parsing of the `weak-subjectivity-checkpoint` parameter (#3765) 2022-06-17 14:55:03 +03:00
zah e8efc0f184
Add support for the Sepolia network (#3762) 2022-06-16 17:11:26 +03:00
tersec 9da9fc83ed
bump merge-testnets for final ropsten TTD (#3754) 2022-06-16 08:18:46 +03:00
Jacek Sieka 099709872e
Merge branch 'stable' into unstable 2022-06-14 13:35:38 +02:00
Tanguy 29297e9ce4
Bump libp2p (#3709) 2022-06-08 07:53:50 +02:00
tersec 3bd9622572
bump nim-json-serialization to fix Nim 1.6 build warnings (#3711) 2022-06-07 16:26:40 +02:00
tersec 3e7d3bbc31
bump eth2-networks with mainnet/prater including capella (#3707) 2022-06-06 17:45:58 +03:00
tersec 399afd89e9
bump nim-json-serialization to reduce TaintedString deprecation warnings (#3703) 2022-06-05 08:55:59 +02:00
tersec a3f8b4e9eb
bump nim-confutils to reduce TaintedString deprecation warnings (#3702) 2022-06-04 08:58:41 +02:00
Eugene Kabanov 85d0234524
Bump chronos to fix httpclient assert issue. (#3669) 2022-05-31 10:50:33 +02:00
Jacek Sieka 48f01186d6
fix unnecessary HashList/HashArray cache invalidation (#3660)
* SSZ `[]` -> `mitem`
* `[]` -> `item`

immutable access via mutable instance cannot rely on template
overloading, and `[]` cannot be a `func` because of special seq handling
in compiler.
2022-05-30 13:30:42 +00:00
tersec 6e27661440
bump merge-testnets to get new ropsten TTD (#3668) 2022-05-27 00:09:16 +03:00
tersec bf1763fdef
consensus specs v1.2.0-rc.1 (#3661) 2022-05-25 13:49:29 +00:00
tersec dfd8cd22b7
bump nim-web3 and use engine API v1.0.0.alpha.9 (#3663) 2022-05-25 10:30:37 +00:00
zah e7ce3cacd0
Add support for the Ropsten beacon chain (#3648) 2022-05-20 18:26:07 +03:00
Zahary Karadjov a359dbe23e
Bump blscurve in order to fix the Windows distribution build 2022-05-18 19:56:10 +03:00
zah 9080cb561e
Bump Chronos to fix HTTP client assertion failures (#3628) 2022-05-15 02:02:20 +03:00
zah 6d11ad6ce1
Support for distributed keystores with multiple remotes based on threshold signatures (#3616)
Other fixes:

* Fix bit rot in the `make prater-dev-deposit` target.
* Correct content-type in the responses of the Nimbus signing node
* Invalid JSON payload was being sent in the web3signer requests
2022-05-10 03:32:12 +03:00
Kim De Mey f016e1abbd
Bump nim-eth vendor module (#3612)
Changes relevant to nimbus-eth2:
- Update discv5 implementation explainer comment
- Add discv5 max packet size limit
2022-05-03 08:50:23 +00:00
Jacek Sieka d0dbc4a8f9
Snappy revamp (#3564)
This PR makes the necessary adjustments to deal with the revamped snappy
API.

In practical terms for nimbus-eth2, there are performance increases to
gossip processing, database reading and writing as well as era file
processing. Exporting `.era` files for example, a snappy-heavy
operation, almost halves in total processing time:

Pre:

```
     Average,       StdDev,          Min,          Max,      Samples,         Test
      39.088,        8.735,       23.619,       53.301,           50, tState
     237.079,       46.692,      165.620,      355.481,           49, tBlocks
```

Post:

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
      25.350,        5.303,       15.351,       41.856,           50, tState
     141.238,       24.164,       99.990,      199.329,           49, tBlocks
```
2022-04-15 09:44:06 +02:00
tersec b237afeb16
bump nim-web3 for stylecheck fixes (#3596) 2022-04-15 07:11:48 +02:00
tersec 287a9c5f51
bump nim-stew to remove pragma disabling checks (#3555)
* bump nim-stew to remove pragma disabling checks

* use V=1 to try to capture the build VM overflow stack trace
2022-04-12 12:07:06 +02:00
Jacek Sieka 2d3d819fd2
fix crash when using >1024 file descriptors
* fixes the crash part of #3521, which in turn is a result of the leaks
fixed in #3582
2022-04-11 12:10:16 +02:00
zah 0304f7b2e6
Fix #3521 (HTTP connection leaks leading to crashes) (#3582) 2022-04-11 09:39:59 +03:00
tersec 992ba8a44d
bump nim-ssz-serialization: treat List[byte, ...] as hex strings for JSON (#3581)
* bump nim-ssz-serialization: treat List[byte, ...] as hex strings for JSON

* use to0xHex commit
2022-04-09 17:38:08 +00:00
Jacek Sieka f70ff38b53
enable `styleCheck:usages` (#3573)
Some upstream repos still need fixes, but this gets us close enough that
style hints can be enabled by default.

In general, "canonical" spellings are preferred even if they violate
nep-1 - this applies in particular to spec-related stuff like
`genesis_validators_root` which appears throughout the codebase.
2022-04-08 16:22:49 +00:00
Jacek Sieka 7207185bc3
bump nim-ssz-serialization (#3576)
currently pointing to a branch that was merged and no longer exists
2022-04-07 20:08:20 +00:00
Jacek Sieka 7103b663ae
remove `nim-protobuf-serialization` (#3575)
tests don't compile in the nimbus-eth2 environment due to missing
dependencies, and the project itself is not used within nimbus-eth2
2022-04-07 18:06:33 +00:00
tersec 34b62181de
bump nim-web3 (#3572) 2022-04-07 13:16:16 +00:00
Jacek Sieka d2c8271d60
bumps (#3567)
style/tests/no significant changes
2022-04-04 16:53:16 +00:00
Jacek Sieka 0f023dc04d
drop rocksdb (#3566)
rocksdb was never actually used in nimbus-eth2 and existed only to satisfy nim-eth dependencies for test running - these have since moved to nimbus-eth1.

* bump nim-eth
2022-04-04 15:38:42 +00:00
zah f89c604fb0
Add Nimble lock file (#3560) 2022-03-30 22:00:03 +00:00
Kim De Mey e222405b86
Bump nim-eth vendor module (#3556)
Changes in nim-eth relevant to nimbus-eth2:

- Debug logs providing info for the initial discoveryv5 configuration
- net/nat.nim: "raises" annotations
2022-03-30 08:30:36 +02:00
Jacek Sieka 5092fc41c7
use snappy-framed format for compressing bellatrix+ database entries (#3551)
`.era` files and Req/Resp protocols use framed formats - aligning the
database with these makes for less recompression work overall as gossip
is sent only once while req/resp repeats (potentially) - this also
allows efficient pruning-to-era where snappy-recompression is the major
cycle thief.
2022-03-29 11:33:06 +00:00
Ștefan Talpalaru 46e5175267
Windows: latest Mingw can't handle '\' as path sep (#3553) 2022-03-29 08:39:24 +02:00
tersec 391d4ddd0d
bump faststreams to avoid triggering Nim checks (#3552) 2022-03-28 22:04:43 +00:00
Dustin Brody 57fc9d5433
revert 2022-03-28 12:08:42 +00:00
Dustin Brody 146a4c097d
bump faststreams to avoid triggering Nim checks 2022-03-28 12:05:57 +00:00
tersec a058b1d1ce
bump confutils to fix 64-bit cmdline args on 32-bit platforms (#3542) 2022-03-24 09:58:48 +01:00
Jacek Sieka 4a237cb908
enable chronosStrictException (#3533)
* bump nim-json-rpc
2022-03-22 09:42:28 +01:00
Ștefan Talpalaru a1f3adc3e2
bump vendor/nimbus-build-system (#3526)
* bump vendor/nimbus-build-system
2022-03-19 08:58:05 +01:00
tersec 3f0a5026a4
bump nim-web3 for request header callbacks for JWT (#3496) 2022-03-15 09:40:04 +00:00
tersec aace7086d3
bump nim-stew (#3492) 2022-03-14 15:08:02 +00:00
Ștefan Talpalaru 276762958e
Windows: disable status bar (#3484)
It can randomly lock inside Windows terminal emulators. Better play it
safe.
2022-03-14 10:19:50 +01:00
Zahary Karadjov aa3bdb1228
Helpful error message when the user fails to use an array type in TOML
This applies to fields such as `web3-url` which are mapped to array
in TOML in a way that may surprise the user.
2022-03-09 19:11:41 +02:00
Zahary Karadjov e6723ddb24
Allow running Nimbus as a Windows service (--run-as-service) 2022-03-05 15:53:47 +02:00
zah cdeae90806
Add support for TOML config files (--config-file) (#3442) 2022-03-05 04:33:15 +02:00
zah 8967f9cf01
Work-around the sizeof change in behavior introduced in Nim 1.6 (#3462) 2022-03-04 10:52:49 +02:00
tersec 7b3d9d4e14
use v1.1.10 CL spec test vectors (#3454) 2022-03-02 07:26:17 +00:00
Tanguy 1bfbcc48b6
Bump libp2p (#3438) 2022-02-25 13:22:48 +01:00
tersec fef71a78a0
bump nim-web3 for random -> prevRandao rename (#3435) 2022-02-24 18:01:48 +01:00
tersec 42b7a0aac4
bump nim-web3 for engine alpha.7 API (#3433) 2022-02-24 04:22:46 +00:00
tersec f6ae49595d
bump nim-json-rpc to request with application/json Content-Type (#3431) 2022-02-23 11:31:59 +01:00
Jacek Sieka 9d392711fa
bump nim-json-ser (#3426)
fixes handling of null cstrings / cleanups
2022-02-21 11:48:30 +01:00
Kim De Mey c34104adbd
Bump nim-eth vendor module (#3408)
Changes in nim-eth relevant to nimbus-eth2:

- Style fixes according to --styleCheck:usages (#452)
- Add discoveryv5 session metrics (#454)
- Don’t use exceptions for enr get call (#453)
- Add DiscoveryConfig to tune routing table ip limits and bitPerHops
- More --styleCheck fixes for discoveryv5 and eth/common (#473)
2022-02-17 19:07:01 +00:00
Jacek Sieka 87e98b9e54
Revert "bump submodules (#3366)" (#3406)
This reverts commit 6e1ad080e8.
2022-02-17 12:50:37 +01:00
Ștefan Talpalaru 6e1ad080e8
bump submodules (#3366)
and add Nim-1.6 compatibility
2022-02-16 13:41:50 +02:00
Ștefan Talpalaru 496d0266ec
bump nim-metrics (#3392) 2022-02-14 21:57:06 +01:00
Zahary Karadjov 922a0d264c Add CORS support for the REST services
The added options work in opt-in fashion. If they are not specified,
the server will respond to all requests as if the CORS specification
doesn't exist. This will result in errors in CORS-enabled clients.

Please note that future versions may support more than one allowed
origin. The option names will stay the same, but the user will be
able to repeat them on the command line (similar to other options
such as --web3-url).

To be documented in the guide in a separate PR.
2022-02-14 18:52:17 +02:00
tersec d02daf8cbd
bump nim-web3 to fix kiln interop (#3373) 2022-02-11 18:38:44 +00:00
Eugene Kabanov b4eb150b9a
Revert restAccept workaround. (#3369)
Bump fixed version of nim-presto.
2022-02-11 12:01:45 +01:00
Mamy Ratsimbazafy 97a1735e4a
Bump BLST (security fic on currently unused primitive) (#3364) 2022-02-09 03:08:47 +01:00
Ștefan Talpalaru 70b38e37e6
Nim GC metrics for the main thread (#3108)
* Nim GC metrics for the main thread
2022-02-08 20:19:21 +01:00
tersec 02349b4181
update to engine API alpha.6 (#3351) 2022-02-04 12:12:19 +00:00
Zahary Karadjov 215caa21ae Eth1 monitor fixes
* Fix a resource leak introduced in https://github.com/status-im/nimbus-eth2/pull/3279

* Don't restart the Eth1 syncing proggress from scratch in case of
  monitor failures during Eth2 syncing.

* Switch to the primary operator as soon as it is back online.

* Log the web3 credentials in fewer places

Other changes:

The 'web3 test' command has been enhanced to obtain and print more
data regarding the selected provider.
2022-02-03 14:01:55 +02:00
tersec 8e6a920bf4
rename MERGE_FORK_EPOCH to BELLATRIX_FORK_EPOCH (#3350)
* rename MERGE_FORK_EPOCH to BELLATRIX_FORK_EPOCH

* fix REST test rules
2022-02-02 14:06:55 +01:00
Jacek Sieka d583e8e4ac
Store finalized block roots in database (3s startup) (#3320)
* Store finalized block roots in database (3s startup)

When the chain has finalized a checkpoint, the history from that point
onwards becomes linear - this is exploited in `.era` files to allow
constant-time by-slot lookups.

In the database, we can do the same by storing finalized block roots in
a simple sparse table indexed by slot, bringing the two representations
closer to each other in terms of conceptual layout and performance.

Doing so has a number of interesting effects:

* mainnet startup time is improved 3-5x (3s on my laptop)
* the _first_ startup might take slightly longer as the new index is
being built - ~10s on the same laptop
* we no longer rely on the beacon block summaries to load the full dag -
this is a lot faster because we no longer have to look up each block by
parent root
* a collateral benefit is that we no longer need to load the full
summaries table into memory - we get the RSS benefits of #3164 without
the CPU hit.

Other random stuff:

* simplify forky block generics
* fix withManyWrites multiple evaluation
* fix validator key cache not being updated properly in chaindag
read-only mode
* drop pre-altair summaries from `kvstore`
* recreate missing summaries from altair+ blocks as well (in case
database has lost some to an involuntary restart)
* print database startup timings in chaindag load log
* avoid allocating superfluos state at startup
* use a recursive sql query to load the summaries of the unfinalized
blocks
2022-01-30 18:51:04 +02:00
Emil 0051af430b Put `application/json` as a higher preference than `application/octet-stream` 2022-01-30 18:50:14 +02:00
tersec 60bf5b8bf4
use v1.1.9 test vectors (#3337) 2022-01-28 22:47:48 +00:00
Ștefan Talpalaru fdb76b8e0b
bump nimbus-build-system (#3324) 2022-01-26 17:11:28 +01:00
Ștefan Talpalaru 47a6f9950f
bump nimbus-build-system (#3322) 2022-01-26 14:15:53 +01:00
Mamy Ratsimbazafy 9e9ccf4a1f
Slashing prot interchange tests v5.2.1 (#3277)
* initial support for minification and new interchange tests. Removal of v1 and v1 migration.

* Synthetic attestations: SQLite3 requires one statement/query per prepared statement

* Fix DB import interrupted if no attestation was found

* Skip test relying on undocumented test behavior (https://github.com/eth-clients/slashing-protection-interchange-tests/pull/12#issuecomment-1011158701)

* Skip test relying on unclear minification behavior:
creating an invalid minified attestation with source > target or setting target = max(source, target)

* remove DB v1 and update submodule

* Apply suggestions from code review

Co-authored-by: Jacek Sieka <jacek@status.im>

Co-authored-by: Jacek Sieka <jacek@status.im>
2022-01-20 17:14:06 +01:00
Jacek Sieka 836f6984bb
move `state_transition` to `Result` (#3284)
* better error messages in api
* avoid `BlockData` copies when replaying blocks
2022-01-17 12:19:58 +01:00
Zahary Karadjov af614b6a85 bump nim-json-rpc 2022-01-16 18:26:21 +02:00
tersec 142e0ea163
bump nim-web3 to use enums for engine API types (#3282) 2022-01-14 14:11:42 +01:00
Jacek Sieka 7e1cdce8fc
stew: bump (#3271)
result fixes
2022-01-12 07:09:23 +00:00
Jacek Sieka aab302aa9c
bumps (#3268)
result + serialization cleanups
2022-01-11 09:35:20 +00:00
Jacek Sieka 6f7e0e3393
REST cleanups (#3255)
* REST cleanups

* reject out-of-range committee requests
* print all hex values as lower-case
* allow requesting state information by head state root
* turn `DomainType` into array (follow spec)
* `uint_to_bytesXX` -> `uint_to_bytes` (follow spec)
* fix wrong dependent root in `/eth/v1/validator/duties/proposer/`
* update documentation - `--subscribe-all-subnets` is no longer needed
when using the REST interface with validator clients
* more fixes
* common helpers for dependent block
* remove test rules obsoleted by more strict epoch tests
* fix trailing commas

* Update docs/the_nimbus_book/src/rest-api.md
* Update docs/the_nimbus_book/src/rest-api.md

Co-authored-by: sacha <sacha@status.im>
2022-01-08 22:06:34 +02:00
Tanguy 7a0119ac32
Bump libp2p (#3246) 2022-01-05 17:33:23 +01:00
tersec cd77377375
add Bellatrix fork and transition tests; "Ethereum Foundation" -> EF (#3242) 2022-01-05 09:42:56 +01:00
Zahary Karadjov 54d0d588b1 Implementation of the Keymanager API (BETA)
https://github.com/ethereum/keymanager-APIs
2022-01-04 18:51:45 +02:00
tersec 3c63a78c01
use v1.1.8 test vectors (#3236) 2022-01-03 17:43:00 +00:00
Jacek Sieka 61b6fc1016
3x speedup in snappy compression (#3234)
* 3x speedup in snappy compression

oh, the wonders of `copyMem` in `endians2` - speeds up all kinds of
operations like database stores, sending gossip etc.

* endian usage fixes
2022-01-03 18:17:10 +01:00
tersec 8be1699014
use v1.1.7 test vectors (#3231)
* use v1.1.7 test vectors
2022-01-03 13:06:14 +00:00
Mamy Ratsimbazafy 873b0f1128
Bump BLST, improve pubkey deserialization speed by 20% to 2x faster. improve signature deserialization by 5% to 4x faster. (#3227)
Nice!
2022-01-02 16:48:14 +01:00
Jacek Sieka 6b60a774e0
Lazy aggregated batch verification (#3212)
A novel optimisation for attestation and sync committee message
validation: when batching, we look for signatures of the same message
and aggregate these before batch-validating: this results in up to 60%
fewer signature verifications on a busy server, leading to a significant
reduction in CPU usage.

* increase batch size slightly which helps finding more aggregates
* add metrics for batch verification efficiency
* use simple `blsVerify` when there is only one signature to verify in
the batch, avoiding the RNG
2021-12-29 15:28:40 +01:00
Tanguy 4a72def1d5
Bump libp2p (#3207) 2021-12-17 12:39:24 +01:00
Ștefan Talpalaru 9311d0a887
Nim-1.2.16 (#3203) 2021-12-17 00:52:49 +01:00
Etan Kissling dc1562665f
bump nim-ssz-serialization to `3cd8d2d6b80bde0ce7f25609cb5cb9fc37852fe2` (#3194)
This updates `nim-ssz-serialization` to
`3cd8d2d6b80bde0ce7f25609cb5cb9fc37852fe2`.

Notable changes:
- Serialization of object variant (case object) to/from SSZ Union.
- int -> int64 fix in hashTreeRootCached
2021-12-16 17:26:54 +00:00
Etan Kissling f68aa9218f
bump nim-confutils to `6a56d01381f434d5fbcc61b6e497b9409155bcbc` (#3196)
This updates `nim-confutils` to
`6a56d01381f434d5fbcc61b6e497b9409155bcbc`.

Notable changes:
- feature: separator text when displaying help
- feature: multiple lines long description
- feature: add ignore property in addition to hidden
- add compile time check to detect duplicate abbr and duplicate name
2021-12-15 09:27:04 +00:00
Etan Kissling 6f75262a03
bump nim-snappy to `16cce7d07ce8be2cfc760d86744b8b54cd0808a9` (#3195)
This updates `nim-snappy` to `16cce7d07ce8be2cfc760d86744b8b54cd0808a9`.

Notable changes:
- avoid unnecessary compression of short payloads
2021-12-14 18:50:24 +00:00