Commit Graph

1155 Commits

Author SHA1 Message Date
Eugene Kabanov ca871a5435
Fix HTTP/REST clients HTTP Content-Type header parsers. (#4139)
* Fix client HTTP content-type parsers.

* Fix tests.

* Address review comment and apply wildcard checks for generic decodeBytes.
2022-09-19 12:17:29 +03:00
Etan Kissling 6ad9b3214d
fix random `IndexError` in CI (#4136)
Bump `nim-chronos` to improve reliability of tests with `waitFor` logic.
The bump also adds more `FutureBase` helpers.
2022-09-17 19:45:09 +00:00
Etan Kissling 24534090b4
send SNI info when using https/wss web3 API (#4133)
When connecting to hosts on shared IP/Port using TLS, SNI must be sent
to allow the remote server to provide the correct TLS certificate.
Bump the `nim-json-rpc` and `nim-websock` dependencies to send SNI ext.
2022-09-17 03:01:59 +00:00
Jacek Sieka 9df08576a1
bump json-rpc, for potential `nil` http crash (fixes 4118) (#4125)
* bump json-rpc, for potential `nil` http crash (fixes #4118)

* bump
2022-09-16 08:08:25 +00:00
Etan Kissling 634408ff2c
use `nim-websock` instead of `news` (#4061)
`news` has a few open issues that are not present in `nim-websock`:
1. There is a 1 second delay between each MB of sent data.
2. Cancelling an ongoing `send` makes the entire WebSocket unusable.
3. Control packets do not have priority over ongoing message frames.

Using `news`, there are quite a few of these messages in Geth:
```
Previously seen beacon client is offline. Please ensure it is
operational to follow the chain!
```
It may take quite some time to reconnect when this happens.

Using `nim-websock`, this message still occurs because `eth1_monitor`
reconnects the EL connection when no new blocks occurred for 5 minutes,
but reconnecting is quick and the message is rarer.
2022-09-06 23:41:33 +02:00
Jacek Sieka 1538dea45e
bumps (#4069)
* bearssl: use master branch again
* chronos: less Option, better contenttype handling
* eth, presto: less imports
* stew: typo
* web3: tighter `Quantity` parsing rules (see
https://github.com/status-im/nimbus-eth2/pull/3850 and subsequently
https://github.com/status-im/nim-web3/pull/55)
2022-09-06 13:54:30 +00:00
Tanguy 2da13c0b22
Bump libp2p (#4066) 2022-09-05 20:05:36 +02:00
Jacek Sieka d9ceb61dbd
eth: bump (#4062) 2022-09-04 19:44:43 +02:00
tersec f33a241ba1
retry EF spec test vector downloads (#4059) 2022-09-01 19:51:19 +00:00
Etan Kissling ebfb624557
bump `news` to fix concurrency bug with >1MB frames (#4028)
When calling `newPayload` on a >1MB payload (can happen post-merge),
`news` splits up that payload into 1MB chunks. The chunks are each sent
individually, though, with `await` in-between. This means that when we
send concurrent `forkChoiceUpdated` calls, that those may end up getting
in-between the `newPayload` chunks, leading to invalid data being sent.
The EL then returns an error message with a `null` `id` entry (as it
could not read the request `id` due to the mangling) and disconnects.
A PR has been submitted to fix this in `news`, and merged into `status`
branch early as this fix is critical for reliable post-merge operation:
https://github.com/Tormund/news/pull/22
2022-08-25 21:14:40 +00:00
Jacek Sieka 492a2ccfac
bump ssz-serialization (#4017)
* free 7% speedup of `hash_tree_root`
* harden `len` for uninitialized `BitSeq`
2022-08-23 18:32:34 +00:00
zah a7192f5d6c
Fix the block header computation when proposing an empty execution payload (#3991)
* Fix the block header computation when proposing an empty execution payload
* Spec compliant base fee calculation when producing empty payloads
2022-08-19 13:28:42 +03:00
tersec b5b93e90c0
use v1.2.0-rc.3 test vectors (#3995) 2022-08-19 04:32:53 +00:00
Zahary Karadjov 7f681af56f
Make it easier to test trustedNodeSync or different networks 2022-08-18 19:48:11 +03:00
zah 74819729a5
Bellatrix metadata for Mainnet (#3980) 2022-08-17 14:40:58 +00:00
tersec e732b080f7
use v1.2.0-rc.2 test vectors (#3975) 2022-08-16 21:24:37 +00:00
zah dc50abbc90
Implement a missing ingnore rule for sync committee contributions (#3941) 2022-08-09 12:52:11 +03:00
Eugene Kabanov 250f7b4bdf
Exclusive keystore locking (#3907) 2022-08-08 00:53:20 +03:00
Eugene Kabanov fe5435ed96
Address #3624 (#3934) 2022-08-06 11:55:40 +00:00
Zahary Karadjov 5951df4193
Switch back to Nim 1.2
We've discovered a critical `lent` issue affecting Nim 1.6:
https://github.com/nim-lang/Nim/issues/20107
2022-07-29 11:26:36 +03:00
tersec b7f907a64a
bump nim-zxcvbn for styleCheck compatibility (#3919) 2022-07-27 14:25:09 +03:00
tersec 9d41fbfded
bump nim-presto for styleCheck compatibility (#3918) 2022-07-27 14:24:45 +03:00
Zahary Karadjov 4637669b6b
Merge branch 'stable' into unstable 2022-07-26 23:54:46 +03:00
tersec b7ed0aa235
bump nim-websock for styleCheck compatibility (#3909) 2022-07-26 12:20:36 +03:00
tersec ee24beee30
bump nim-stint for styleCheck compatibility (#3908) 2022-07-26 12:19:55 +03:00
zah c5d89bb46e
Switch to Nim 1.6.7 (#3902) 2022-07-26 12:19:32 +03:00
tersec 6b2abd01fb
bump nim-eth for styleCheck compatibility (#3896) 2022-07-25 23:15:46 +03:00
Zahary Karadjov 1a1553fee8
Bump eth2-networks to include the Prater TTD 2022-07-21 22:30:17 +03:00
Jacek Sieka 24d8401054
bumps (#3892)
various ABI fixes
2022-07-20 20:20:34 +02:00
tersec df660292a4
bump nim-metrics for styleCheck compatibility (#3889) 2022-07-20 10:13:32 +02:00
tersec 3dc04bc48f
bump nimcrypto for styleCheck compatibility (#3885) 2022-07-18 17:13:25 +00:00
Jacek Sieka 7dff72c6bf
bumps (#3882)
* fix UB in leb128 shift
* bearssl spellings
2022-07-18 07:17:53 +02:00
zah e6988c1cc9
Bump nim-json-serialization (#3878) 2022-07-15 14:36:03 +00:00
zah 20d45e69b5
Re-enabled requireAllFields after a fix in nim-json-serialization (#3871)
* Re-enabled requireAllFields after a fix in nim-json-serialization

The problem was that `Option[T]` fields were not treated as optional
when requireAllFields is set to true. This is now fixed in NJS.

* Add makefile targets for recreating the Jenkins simulation runs

* Fix a discrepancy with the REST spec
2022-07-15 03:19:19 +03:00
Eugene Kabanov 263a2ffa14
Validator client various fixes. (#3840)
* Improve fallback_service.
* Fix nextAction negative time issue.
* Improve logging in fallback_service.
* Improve logging in sync_committee_service.
* Prepare all services for cancellation.
* Signals handlers for validator client
* Address #3800

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2022-07-13 17:43:57 +03:00
tersec 91b51adb12
Revert "bump nim-web3 to enforce JSON-RPC Quantity syntax (#3846)" (#3850)
This reverts commit 007dcc42f1.
2022-07-08 00:07:50 +00:00
tersec 007dcc42f1
bump nim-web3 to enforce JSON-RPC Quantity syntax (#3846) 2022-07-07 21:14:41 +02:00
Jacek Sieka e1830519a4
Introduce message router (#3829)
Whether new blocks/attestations/etc are produced internally or received
via REST, their journey through the node is the same - to ensure that
they get the same treatment (logging, metrics, processing), this PR
moves the routing to a dedicated module and fixes several small
differences that existed before.

* `xxxValidator` -> `processMessageName` - the processor also was adding
messages to pools, so we want the name to reflect that action
* add missing "sent" metrics for some messages
* document ignore policy better - already-seen messages are not actaully
rebroadcast by libp2p
* skip redundant signature checks for internal validators consistently
2022-07-06 16:11:44 +00:00
tersec 9eb1a3efb3
bump nim-json-serialization to fix Nim 1.6 build warning (#3835) 2022-07-01 22:27:40 +00:00
Tanguy 4fbbbfd462
Bump nim-websock: fix crash when sending >1mb (#3831) 2022-07-01 15:11:45 +03:00
zah f08f9e2bd7
Bump nim-json-rpc and nim-websock (use news by default) (#3820) 2022-06-29 17:35:09 +03:00
tersec 2d2cda735b
bump nim-eth to fix Nim 1.6 warnings (#3818) 2022-06-28 22:56:48 +02:00
zah 2c3e47d7e6
Avoid overflow in the initial 'nextExchangeTransitionConfTime' calculation (#3809) 2022-06-28 23:50:40 +03:00
Zahary Karadjov 45ad4abe30
TTD value for Sepolia 2022-06-28 19:25:11 +03:00
Mamy Ratsimbazafy c4e5cd9f06
update nim blscurve: use builtin EIP2333 for BLST https://github.com/status-im/nim-blscurve/pull/142 2022-06-28 15:11:20 +02:00
tersec 5145c04be2
Revert "Switch back to nim-websock (#3665)" (#3812)
This reverts commit 8d3afd8285.
2022-06-27 22:21:30 +00:00
Etan Kissling 07f9936b33
bump nim-ssz-serialization to `f1b14875792df7b1e76c98c9ee669026d7cfe6bb` (#3802)
Update nim-ssz-serialization to get rid of a style check warning.
2022-06-26 23:17:40 +00:00
tersec c02c383b91
bump nim-web3 so default payload status is safe (#3801) 2022-06-26 18:31:32 +00:00
Etan Kissling aa1b8e4a17
bump nim-ssz-serialization to `3db6cc0f282708aca6c290914488edd832971d61` (#3119)
This updates `nim-ssz-serialization` to
`3db6cc0f282708aca6c290914488edd832971d61`.

Notable changes:
- Use `uint64` for `GeneralizedIndex`
- Add support for building merkle multiproofs
2022-06-26 19:33:06 +02:00
Jacek Sieka b764f4f0d7
bump bearssl (#3797)
cross compile path fix
2022-06-24 22:45:26 +00:00
Tanguy c45c017349
Bump libp2p to fix write on closed channel (#3795) 2022-06-24 12:52:10 +00:00
Ivan Yonchovski d701887a9f
Bump libbacktrace (#3794) 2022-06-24 07:04:29 +02:00
Mamy Ratsimbazafy e25cfe56a6
bump blst fix 32-bit builds on ADX CPUs (#3789) 2022-06-23 16:45:43 +02:00
tersec 4bb42eea7d
bump nim-web3 for eth_chainId support (#3785) 2022-06-21 21:09:29 +00:00
Tanguy 8d3afd8285
Switch back to nim-websock (#3665) 2022-06-21 17:43:55 +02:00
Jacek Sieka f3852cc4cd
sqlite: bump (#3783)
new nimterop
2022-06-21 15:56:48 +02:00
Jacek Sieka 347a485b5b
bearssl: split abi (#3755) 2022-06-21 10:29:16 +02:00
Eugene Kabanov eb6b7affee
Add the `execution_optimistic` flag to REST API responses. (#3780)
* Initial commit

* Make `events` API spec compliant.

* Add `Eth-Consensus-Version` in responses.

* Bump chronos to get redirect with headers working.

* Add `is_optimistic` field and handling to syncing RestSyncInfo.
2022-06-20 08:53:39 +03:00
zah c24c737866
Fix #3650 (participation format in BeaconState result is out of spec) (#3776)
* Fix #3650 (participation format in BeaconState result is out of spec)
* Make EpochParticipationFlags a distinct type
2022-06-20 08:38:56 +03:00
Eugene Kabanov 5e7b28cd66
AsyncEventBus missing attestations fix. (#3664) 2022-06-17 18:27:28 +03:00
tersec b0e46686f1
bump merge-testnets to use capella fork versions consistently (#3767) 2022-06-17 17:17:09 +03:00
zah 69f505e2ba
Correct parsing of the `weak-subjectivity-checkpoint` parameter (#3765) 2022-06-17 14:55:03 +03:00
zah e8efc0f184
Add support for the Sepolia network (#3762) 2022-06-16 17:11:26 +03:00
tersec 9da9fc83ed
bump merge-testnets for final ropsten TTD (#3754) 2022-06-16 08:18:46 +03:00
Jacek Sieka 099709872e
Merge branch 'stable' into unstable 2022-06-14 13:35:38 +02:00
Tanguy 29297e9ce4
Bump libp2p (#3709) 2022-06-08 07:53:50 +02:00
tersec 3bd9622572
bump nim-json-serialization to fix Nim 1.6 build warnings (#3711) 2022-06-07 16:26:40 +02:00
tersec 3e7d3bbc31
bump eth2-networks with mainnet/prater including capella (#3707) 2022-06-06 17:45:58 +03:00
tersec 399afd89e9
bump nim-json-serialization to reduce TaintedString deprecation warnings (#3703) 2022-06-05 08:55:59 +02:00
tersec a3f8b4e9eb
bump nim-confutils to reduce TaintedString deprecation warnings (#3702) 2022-06-04 08:58:41 +02:00
Eugene Kabanov 85d0234524
Bump chronos to fix httpclient assert issue. (#3669) 2022-05-31 10:50:33 +02:00
Jacek Sieka 48f01186d6
fix unnecessary HashList/HashArray cache invalidation (#3660)
* SSZ `[]` -> `mitem`
* `[]` -> `item`

immutable access via mutable instance cannot rely on template
overloading, and `[]` cannot be a `func` because of special seq handling
in compiler.
2022-05-30 13:30:42 +00:00
tersec 6e27661440
bump merge-testnets to get new ropsten TTD (#3668) 2022-05-27 00:09:16 +03:00
tersec bf1763fdef
consensus specs v1.2.0-rc.1 (#3661) 2022-05-25 13:49:29 +00:00
tersec dfd8cd22b7
bump nim-web3 and use engine API v1.0.0.alpha.9 (#3663) 2022-05-25 10:30:37 +00:00
zah e7ce3cacd0
Add support for the Ropsten beacon chain (#3648) 2022-05-20 18:26:07 +03:00
Zahary Karadjov a359dbe23e
Bump blscurve in order to fix the Windows distribution build 2022-05-18 19:56:10 +03:00
zah 9080cb561e
Bump Chronos to fix HTTP client assertion failures (#3628) 2022-05-15 02:02:20 +03:00
zah 6d11ad6ce1
Support for distributed keystores with multiple remotes based on threshold signatures (#3616)
Other fixes:

* Fix bit rot in the `make prater-dev-deposit` target.
* Correct content-type in the responses of the Nimbus signing node
* Invalid JSON payload was being sent in the web3signer requests
2022-05-10 03:32:12 +03:00
Kim De Mey f016e1abbd
Bump nim-eth vendor module (#3612)
Changes relevant to nimbus-eth2:
- Update discv5 implementation explainer comment
- Add discv5 max packet size limit
2022-05-03 08:50:23 +00:00
Jacek Sieka d0dbc4a8f9
Snappy revamp (#3564)
This PR makes the necessary adjustments to deal with the revamped snappy
API.

In practical terms for nimbus-eth2, there are performance increases to
gossip processing, database reading and writing as well as era file
processing. Exporting `.era` files for example, a snappy-heavy
operation, almost halves in total processing time:

Pre:

```
     Average,       StdDev,          Min,          Max,      Samples,         Test
      39.088,        8.735,       23.619,       53.301,           50, tState
     237.079,       46.692,      165.620,      355.481,           49, tBlocks
```

Post:

```
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
      25.350,        5.303,       15.351,       41.856,           50, tState
     141.238,       24.164,       99.990,      199.329,           49, tBlocks
```
2022-04-15 09:44:06 +02:00
tersec b237afeb16
bump nim-web3 for stylecheck fixes (#3596) 2022-04-15 07:11:48 +02:00
tersec 287a9c5f51
bump nim-stew to remove pragma disabling checks (#3555)
* bump nim-stew to remove pragma disabling checks

* use V=1 to try to capture the build VM overflow stack trace
2022-04-12 12:07:06 +02:00
Jacek Sieka 2d3d819fd2
fix crash when using >1024 file descriptors
* fixes the crash part of #3521, which in turn is a result of the leaks
fixed in #3582
2022-04-11 12:10:16 +02:00
zah 0304f7b2e6
Fix #3521 (HTTP connection leaks leading to crashes) (#3582) 2022-04-11 09:39:59 +03:00
tersec 992ba8a44d
bump nim-ssz-serialization: treat List[byte, ...] as hex strings for JSON (#3581)
* bump nim-ssz-serialization: treat List[byte, ...] as hex strings for JSON

* use to0xHex commit
2022-04-09 17:38:08 +00:00
Jacek Sieka f70ff38b53
enable `styleCheck:usages` (#3573)
Some upstream repos still need fixes, but this gets us close enough that
style hints can be enabled by default.

In general, "canonical" spellings are preferred even if they violate
nep-1 - this applies in particular to spec-related stuff like
`genesis_validators_root` which appears throughout the codebase.
2022-04-08 16:22:49 +00:00
Jacek Sieka 7207185bc3
bump nim-ssz-serialization (#3576)
currently pointing to a branch that was merged and no longer exists
2022-04-07 20:08:20 +00:00
Jacek Sieka 7103b663ae
remove `nim-protobuf-serialization` (#3575)
tests don't compile in the nimbus-eth2 environment due to missing
dependencies, and the project itself is not used within nimbus-eth2
2022-04-07 18:06:33 +00:00
tersec 34b62181de
bump nim-web3 (#3572) 2022-04-07 13:16:16 +00:00
Jacek Sieka d2c8271d60
bumps (#3567)
style/tests/no significant changes
2022-04-04 16:53:16 +00:00
Jacek Sieka 0f023dc04d
drop rocksdb (#3566)
rocksdb was never actually used in nimbus-eth2 and existed only to satisfy nim-eth dependencies for test running - these have since moved to nimbus-eth1.

* bump nim-eth
2022-04-04 15:38:42 +00:00
zah f89c604fb0
Add Nimble lock file (#3560) 2022-03-30 22:00:03 +00:00
Kim De Mey e222405b86
Bump nim-eth vendor module (#3556)
Changes in nim-eth relevant to nimbus-eth2:

- Debug logs providing info for the initial discoveryv5 configuration
- net/nat.nim: "raises" annotations
2022-03-30 08:30:36 +02:00
Jacek Sieka 5092fc41c7
use snappy-framed format for compressing bellatrix+ database entries (#3551)
`.era` files and Req/Resp protocols use framed formats - aligning the
database with these makes for less recompression work overall as gossip
is sent only once while req/resp repeats (potentially) - this also
allows efficient pruning-to-era where snappy-recompression is the major
cycle thief.
2022-03-29 11:33:06 +00:00
Ștefan Talpalaru 46e5175267
Windows: latest Mingw can't handle '\' as path sep (#3553) 2022-03-29 08:39:24 +02:00
tersec 391d4ddd0d
bump faststreams to avoid triggering Nim checks (#3552) 2022-03-28 22:04:43 +00:00
Dustin Brody 57fc9d5433
revert 2022-03-28 12:08:42 +00:00
Dustin Brody 146a4c097d
bump faststreams to avoid triggering Nim checks 2022-03-28 12:05:57 +00:00
tersec a058b1d1ce
bump confutils to fix 64-bit cmdline args on 32-bit platforms (#3542) 2022-03-24 09:58:48 +01:00
Jacek Sieka 4a237cb908
enable chronosStrictException (#3533)
* bump nim-json-rpc
2022-03-22 09:42:28 +01:00
Ștefan Talpalaru a1f3adc3e2
bump vendor/nimbus-build-system (#3526)
* bump vendor/nimbus-build-system
2022-03-19 08:58:05 +01:00
tersec 3f0a5026a4
bump nim-web3 for request header callbacks for JWT (#3496) 2022-03-15 09:40:04 +00:00
tersec aace7086d3
bump nim-stew (#3492) 2022-03-14 15:08:02 +00:00
Ștefan Talpalaru 276762958e
Windows: disable status bar (#3484)
It can randomly lock inside Windows terminal emulators. Better play it
safe.
2022-03-14 10:19:50 +01:00
Zahary Karadjov aa3bdb1228
Helpful error message when the user fails to use an array type in TOML
This applies to fields such as `web3-url` which are mapped to array
in TOML in a way that may surprise the user.
2022-03-09 19:11:41 +02:00
Zahary Karadjov e6723ddb24
Allow running Nimbus as a Windows service (--run-as-service) 2022-03-05 15:53:47 +02:00
zah cdeae90806
Add support for TOML config files (--config-file) (#3442) 2022-03-05 04:33:15 +02:00
zah 8967f9cf01
Work-around the sizeof change in behavior introduced in Nim 1.6 (#3462) 2022-03-04 10:52:49 +02:00
tersec 7b3d9d4e14
use v1.1.10 CL spec test vectors (#3454) 2022-03-02 07:26:17 +00:00
Tanguy 1bfbcc48b6
Bump libp2p (#3438) 2022-02-25 13:22:48 +01:00
tersec fef71a78a0
bump nim-web3 for random -> prevRandao rename (#3435) 2022-02-24 18:01:48 +01:00
tersec 42b7a0aac4
bump nim-web3 for engine alpha.7 API (#3433) 2022-02-24 04:22:46 +00:00
tersec f6ae49595d
bump nim-json-rpc to request with application/json Content-Type (#3431) 2022-02-23 11:31:59 +01:00
Jacek Sieka 9d392711fa
bump nim-json-ser (#3426)
fixes handling of null cstrings / cleanups
2022-02-21 11:48:30 +01:00
Kim De Mey c34104adbd
Bump nim-eth vendor module (#3408)
Changes in nim-eth relevant to nimbus-eth2:

- Style fixes according to --styleCheck:usages (#452)
- Add discoveryv5 session metrics (#454)
- Don’t use exceptions for enr get call (#453)
- Add DiscoveryConfig to tune routing table ip limits and bitPerHops
- More --styleCheck fixes for discoveryv5 and eth/common (#473)
2022-02-17 19:07:01 +00:00
Jacek Sieka 87e98b9e54
Revert "bump submodules (#3366)" (#3406)
This reverts commit 6e1ad080e8.
2022-02-17 12:50:37 +01:00
Ștefan Talpalaru 6e1ad080e8
bump submodules (#3366)
and add Nim-1.6 compatibility
2022-02-16 13:41:50 +02:00
Ștefan Talpalaru 496d0266ec
bump nim-metrics (#3392) 2022-02-14 21:57:06 +01:00
Zahary Karadjov 922a0d264c Add CORS support for the REST services
The added options work in opt-in fashion. If they are not specified,
the server will respond to all requests as if the CORS specification
doesn't exist. This will result in errors in CORS-enabled clients.

Please note that future versions may support more than one allowed
origin. The option names will stay the same, but the user will be
able to repeat them on the command line (similar to other options
such as --web3-url).

To be documented in the guide in a separate PR.
2022-02-14 18:52:17 +02:00
tersec d02daf8cbd
bump nim-web3 to fix kiln interop (#3373) 2022-02-11 18:38:44 +00:00
Eugene Kabanov b4eb150b9a
Revert restAccept workaround. (#3369)
Bump fixed version of nim-presto.
2022-02-11 12:01:45 +01:00
Mamy Ratsimbazafy 97a1735e4a
Bump BLST (security fic on currently unused primitive) (#3364) 2022-02-09 03:08:47 +01:00
Ștefan Talpalaru 70b38e37e6
Nim GC metrics for the main thread (#3108)
* Nim GC metrics for the main thread
2022-02-08 20:19:21 +01:00
tersec 02349b4181
update to engine API alpha.6 (#3351) 2022-02-04 12:12:19 +00:00
Zahary Karadjov 215caa21ae Eth1 monitor fixes
* Fix a resource leak introduced in https://github.com/status-im/nimbus-eth2/pull/3279

* Don't restart the Eth1 syncing proggress from scratch in case of
  monitor failures during Eth2 syncing.

* Switch to the primary operator as soon as it is back online.

* Log the web3 credentials in fewer places

Other changes:

The 'web3 test' command has been enhanced to obtain and print more
data regarding the selected provider.
2022-02-03 14:01:55 +02:00
tersec 8e6a920bf4
rename MERGE_FORK_EPOCH to BELLATRIX_FORK_EPOCH (#3350)
* rename MERGE_FORK_EPOCH to BELLATRIX_FORK_EPOCH

* fix REST test rules
2022-02-02 14:06:55 +01:00
Jacek Sieka d583e8e4ac
Store finalized block roots in database (3s startup) (#3320)
* Store finalized block roots in database (3s startup)

When the chain has finalized a checkpoint, the history from that point
onwards becomes linear - this is exploited in `.era` files to allow
constant-time by-slot lookups.

In the database, we can do the same by storing finalized block roots in
a simple sparse table indexed by slot, bringing the two representations
closer to each other in terms of conceptual layout and performance.

Doing so has a number of interesting effects:

* mainnet startup time is improved 3-5x (3s on my laptop)
* the _first_ startup might take slightly longer as the new index is
being built - ~10s on the same laptop
* we no longer rely on the beacon block summaries to load the full dag -
this is a lot faster because we no longer have to look up each block by
parent root
* a collateral benefit is that we no longer need to load the full
summaries table into memory - we get the RSS benefits of #3164 without
the CPU hit.

Other random stuff:

* simplify forky block generics
* fix withManyWrites multiple evaluation
* fix validator key cache not being updated properly in chaindag
read-only mode
* drop pre-altair summaries from `kvstore`
* recreate missing summaries from altair+ blocks as well (in case
database has lost some to an involuntary restart)
* print database startup timings in chaindag load log
* avoid allocating superfluos state at startup
* use a recursive sql query to load the summaries of the unfinalized
blocks
2022-01-30 18:51:04 +02:00
Emil 0051af430b Put `application/json` as a higher preference than `application/octet-stream` 2022-01-30 18:50:14 +02:00
tersec 60bf5b8bf4
use v1.1.9 test vectors (#3337) 2022-01-28 22:47:48 +00:00
Ștefan Talpalaru fdb76b8e0b
bump nimbus-build-system (#3324) 2022-01-26 17:11:28 +01:00
Ștefan Talpalaru 47a6f9950f
bump nimbus-build-system (#3322) 2022-01-26 14:15:53 +01:00
Mamy Ratsimbazafy 9e9ccf4a1f
Slashing prot interchange tests v5.2.1 (#3277)
* initial support for minification and new interchange tests. Removal of v1 and v1 migration.

* Synthetic attestations: SQLite3 requires one statement/query per prepared statement

* Fix DB import interrupted if no attestation was found

* Skip test relying on undocumented test behavior (https://github.com/eth-clients/slashing-protection-interchange-tests/pull/12#issuecomment-1011158701)

* Skip test relying on unclear minification behavior:
creating an invalid minified attestation with source > target or setting target = max(source, target)

* remove DB v1 and update submodule

* Apply suggestions from code review

Co-authored-by: Jacek Sieka <jacek@status.im>

Co-authored-by: Jacek Sieka <jacek@status.im>
2022-01-20 17:14:06 +01:00
Jacek Sieka 836f6984bb
move `state_transition` to `Result` (#3284)
* better error messages in api
* avoid `BlockData` copies when replaying blocks
2022-01-17 12:19:58 +01:00
Zahary Karadjov af614b6a85 bump nim-json-rpc 2022-01-16 18:26:21 +02:00
tersec 142e0ea163
bump nim-web3 to use enums for engine API types (#3282) 2022-01-14 14:11:42 +01:00
Jacek Sieka 7e1cdce8fc
stew: bump (#3271)
result fixes
2022-01-12 07:09:23 +00:00
Jacek Sieka aab302aa9c
bumps (#3268)
result + serialization cleanups
2022-01-11 09:35:20 +00:00
Jacek Sieka 6f7e0e3393
REST cleanups (#3255)
* REST cleanups

* reject out-of-range committee requests
* print all hex values as lower-case
* allow requesting state information by head state root
* turn `DomainType` into array (follow spec)
* `uint_to_bytesXX` -> `uint_to_bytes` (follow spec)
* fix wrong dependent root in `/eth/v1/validator/duties/proposer/`
* update documentation - `--subscribe-all-subnets` is no longer needed
when using the REST interface with validator clients
* more fixes
* common helpers for dependent block
* remove test rules obsoleted by more strict epoch tests
* fix trailing commas

* Update docs/the_nimbus_book/src/rest-api.md
* Update docs/the_nimbus_book/src/rest-api.md

Co-authored-by: sacha <sacha@status.im>
2022-01-08 22:06:34 +02:00
Tanguy 7a0119ac32
Bump libp2p (#3246) 2022-01-05 17:33:23 +01:00
tersec cd77377375
add Bellatrix fork and transition tests; "Ethereum Foundation" -> EF (#3242) 2022-01-05 09:42:56 +01:00
Zahary Karadjov 54d0d588b1 Implementation of the Keymanager API (BETA)
https://github.com/ethereum/keymanager-APIs
2022-01-04 18:51:45 +02:00
tersec 3c63a78c01
use v1.1.8 test vectors (#3236) 2022-01-03 17:43:00 +00:00
Jacek Sieka 61b6fc1016
3x speedup in snappy compression (#3234)
* 3x speedup in snappy compression

oh, the wonders of `copyMem` in `endians2` - speeds up all kinds of
operations like database stores, sending gossip etc.

* endian usage fixes
2022-01-03 18:17:10 +01:00
tersec 8be1699014
use v1.1.7 test vectors (#3231)
* use v1.1.7 test vectors
2022-01-03 13:06:14 +00:00
Mamy Ratsimbazafy 873b0f1128
Bump BLST, improve pubkey deserialization speed by 20% to 2x faster. improve signature deserialization by 5% to 4x faster. (#3227)
Nice!
2022-01-02 16:48:14 +01:00
Jacek Sieka 6b60a774e0
Lazy aggregated batch verification (#3212)
A novel optimisation for attestation and sync committee message
validation: when batching, we look for signatures of the same message
and aggregate these before batch-validating: this results in up to 60%
fewer signature verifications on a busy server, leading to a significant
reduction in CPU usage.

* increase batch size slightly which helps finding more aggregates
* add metrics for batch verification efficiency
* use simple `blsVerify` when there is only one signature to verify in
the batch, avoiding the RNG
2021-12-29 15:28:40 +01:00
Tanguy 4a72def1d5
Bump libp2p (#3207) 2021-12-17 12:39:24 +01:00
Ștefan Talpalaru 9311d0a887
Nim-1.2.16 (#3203) 2021-12-17 00:52:49 +01:00
Etan Kissling dc1562665f
bump nim-ssz-serialization to `3cd8d2d6b80bde0ce7f25609cb5cb9fc37852fe2` (#3194)
This updates `nim-ssz-serialization` to
`3cd8d2d6b80bde0ce7f25609cb5cb9fc37852fe2`.

Notable changes:
- Serialization of object variant (case object) to/from SSZ Union.
- int -> int64 fix in hashTreeRootCached
2021-12-16 17:26:54 +00:00
Etan Kissling f68aa9218f
bump nim-confutils to `6a56d01381f434d5fbcc61b6e497b9409155bcbc` (#3196)
This updates `nim-confutils` to
`6a56d01381f434d5fbcc61b6e497b9409155bcbc`.

Notable changes:
- feature: separator text when displaying help
- feature: multiple lines long description
- feature: add ignore property in addition to hidden
- add compile time check to detect duplicate abbr and duplicate name
2021-12-15 09:27:04 +00:00
Etan Kissling 6f75262a03
bump nim-snappy to `16cce7d07ce8be2cfc760d86744b8b54cd0808a9` (#3195)
This updates `nim-snappy` to `16cce7d07ce8be2cfc760d86744b8b54cd0808a9`.

Notable changes:
- avoid unnecessary compression of short payloads
2021-12-14 18:50:24 +00:00
Etan Kissling d023613045
bump nim-eth to `5655bd035cfd7319c6f2e60b7fdefef65a057939` (#3193)
This updates `nim-eth` to `5655bd035cfd7319c6f2e60b7fdefef65a057939`.

Notable changes:
- db: Allow Sqlite keystores to be used in read-only mode
- net: avoid allocation in hash(ValidIpAddress)
- net: Remove hashData usage on objects
- p2p: reject WHOAREYOU packets with non-empty message
- p2p: Adjust logging when node is not reachable but enrAutoUpdate is on
- p2p: Allow a node to self resolve
- p2p: Fix logDistance for BE arch and remove toBytes for NodeId
- p2p: Export discovery routing table and its buckets nodes
- ssz: remove outdated and incorrect SSZ code
- utp: Various updates and fixes
2021-12-14 17:27:46 +00:00
tersec f88bdc5d14
bump nim-web3 to properly decode TypedTransactions (#3187) 2021-12-13 15:50:18 +00:00
Tanguy 4be7e800f2
Bump chronos (Improve ram usage) (#3180) 2021-12-10 13:25:43 +01:00
Jacek Sieka 6f077ce82c
chronos: fast path for writes (and a bunch of other changes) (#3172) 2021-12-08 14:47:04 +01:00
tersec aeb90a1585
bump nim-web3 to adjust engine API field names for alpha.5 (#3166) 2021-12-07 14:30:16 +00:00
tersec 8d7df05f2e
Revert "Bump chronos and presto. (#3159)" (#3160)
This reverts commit 4c90b82d9f.
2021-12-04 15:52:28 +00:00
Eugene Kabanov 4c90b82d9f
Bump chronos and presto. (#3159)
* Add some indicators to help fixing issue.

* Bump presto to help debugging.

* Fix compilation problems in presto.

* Fix SIGSEGV.

* Bump latest changes in chronos and presto.
Fix rare cases in validator_client.

* Use proper commits for chronos and presto.
2021-12-04 14:26:16 +00:00
zah 74c63ed740
More efficient implementation of the 'POST beacon_committee_subscriptions' API (#3153) 2021-12-03 17:04:58 +02:00
nbc-bump-bot[bot] fa2cf028e2
auto-bump nim-libp2p (#3114)
Co-authored-by: = <Menduist@users.noreply.github.com>
2021-12-03 15:03:34 +01:00
tersec 61fb458f89
use v1.1.6 test vectors (#3146) 2021-12-01 12:55:42 +00:00
Ștefan Talpalaru 5e50297f95
bump nimbus-build-system (#3140) 2021-12-01 05:31:22 +01:00
Zahary Karadjov 872b88db50
Hotfix: non-crashing version of nim-json-rpc/news 2021-11-30 02:51:35 +02:00
zah bc6a6a8b8f
Switch back to the news package in nim-json-rpc (#3136) 2021-11-29 21:47:31 +01:00
Zahary Karadjov 88c623e250 Add support for HTTPS Web3 providers 2021-11-23 15:56:18 +02:00
Ștefan Talpalaru 2d986c5346
bump nimbus-build-system (#3117) 2021-11-23 02:43:04 +01:00
Ștefan Talpalaru 4639b04b2a
bump nimbus-build-system (#3116) 2021-11-22 21:21:13 +01:00
Tanguy 4bcdccab9a
Bump libp2p (#3112) 2021-11-18 10:57:27 +01:00
Eugene Kabanov eda42f4a89
Bump nim-chronos. (#3101) 2021-11-15 18:34:07 +00:00
tersec fde73b5a70
bump nim-web3 to switch payload ID from uint64 to 8-byte buffer (#3103) 2021-11-15 14:55:03 +00:00
Zahary Karadjov 5e62df5ad5 /bin/bash -> /usr/bin/env bash (needed on NixOS) 2021-11-15 15:45:01 +02:00
tersec 97ad5d49b0
kintsugi merge vector tests (#3072) 2021-11-10 13:41:02 +02:00
kdeme 03a70fbf36 Use nim-ssz-serialization module and rm local ssz code 2021-11-10 13:37:24 +02:00
Ștefan Talpalaru 782318d3fc
Nim-1.2.14 (#3074)
* Nim-1.2.14

* bump submodules
2021-11-10 11:39:43 +01:00
Zahary Karadjov 66d60e47ae Bump Chronicles 2021-11-09 13:33:53 +02:00
tersec b76326f4b8
use v1.1.5 consensus spec test vectors (#3066) 2021-11-09 06:46:24 +00:00
tersec a0f6a19267
bump nim-web3 for Kintsugi engine API changes (#3065) 2021-11-09 02:43:51 +00:00
Tanguy 8092418746
Bump libp2p (Short agent fix) (#3064)
* cherry-pick libp2p #645

* switch to master
2021-11-08 21:40:23 +01:00
Dustin Brody 023a6a9387 use v1.1.4 test vectors 2021-11-07 21:42:11 +02:00
Ștefan Talpalaru 74f03ddadb
CI: statically check stack size on Linux (#3055) 2021-11-05 18:11:25 +01:00
Ștefan Talpalaru b09cd71348 build: show stderr by default
Also added: `-d:cwarnings` and `-d:limitStackUsage`
2021-11-05 16:06:48 +02:00
Jacek Sieka 233d756518
Logging and startup improvements (#3038)
* Logging and startup improvements

Color support for released binaries!

* startup scripts no longer log to file by default - this only affects
source builds - released binaries don't support file logging
* add --log-stdout option to control logging to stdout (colors, json)
* detect tty:s vs redirected logs and log accordingly
* add option to disable log colors at runtime
* simplify several "common" logs, showing the most important information
earlier and more clearly
* remove line numbers / file information / tid - these take up space and
are of little use to end users
  * still enabled in debug builds and tools
* remove `testnet_servers_image` compile-time option
* server images, released binaries and compile-from-source now offer
the same behaviour and features
* fixes https://github.com/status-im/nimbus-eth2/issues/2326
* fixes https://github.com/status-im/nimbus-eth2/issues/1794
* remove instanteneous block speed from sync message, keeping only
average

before:

```
INF 2021-10-28 16:45:59.000+02:00 Slot start                                 topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:884 lastSlot=2384027 wallSlot=2384028 delay=461us84ns peers=0 head=75a10ee5:3348 headEpoch=104 finalized=cd6804ba:3264 finalizedEpoch=102 sync="wwwwwwwwww:0:0.0000:0.0000:00h00m (3348)"
INF 2021-10-28 16:45:59.046+02:00 Slot end                                   topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:821 slot=2384028 nextSlot=2384029 head=75a10ee5:3348 headEpoch=104 finalizedHead=cd6804ba:3264 finalizedEpoch=102 nextAttestationSlot=-1 nextProposalSlot=-1 nextActionWait=n/a
```

after:

```
INF 2021-10-28 22:43:23.033+02:00 Slot start                                 topics="beacnde" slot=2385815 epoch=74556 sync="DDPDDPUDDD:10:5.2258:01h19m (2361088)" peers=37 head=eacd2dae:2361096 finalized=73782:a4751487 delay=33ms687us715ns
INF 2021-10-28 22:43:23.291+02:00 Slot end                                   topics="beacnde" slot=2385815 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 head=eacd2dae:2361096
```

* fix comment

* documentation updates

* mention `--log-file` may be deprecated in the future
* update various docs
2021-11-02 18:06:36 +01:00
Etan Kissling 74f2350a2c
update nim-presto to bfcbece (#3039)
Upstream changes:
- allow reserved Nim keywords as REST params
2021-10-31 19:51:05 +01:00
Eugene Kabanov 7c9a6b7170
Improve chronos.Future tracking. (#2988)
* Add `child_id` field.

* Fix json-rpc api call and bump chronos.

* Bump chronos master and fix compilation warnings.

* One more bump of `chronos`.

* add random import

* export rest_utils a bit more

Co-authored-by: Jacek Sieka <jacek@status.im>
2021-10-27 14:01:11 +02:00
tersec c10f504a03
bump nim-web3 (#3024) 2021-10-22 13:55:48 +00:00
nbc-bump-bot[bot] 29b26f3f6b
nim-libp2p auto bump (#2840)
* auto-bump nim-libp2p

* Remove peer info for other peers

Not definitive, just to test the libp2p's unstable branch

* finish up Remove peer info for other peers

* getKey -> getPublicKey

* bump libp2p

* libp2p bump

Co-authored-by: = <Menduist@users.noreply.github.com>
Co-authored-by: Tanguy <tanguy@status.im>
2021-10-21 13:01:29 +02:00
Kim De Mey 724a9083ec
Bump nim-eth for allowing decoding ENRs with RLP lists (#3000) 2021-10-19 18:52:49 +02:00
tersec b66e7c8573
v1.1.3 tests; re-enable minimal-preset altair transition tests (#2993) 2021-10-16 07:17:07 +00:00
Zahary Karadjov 9381ad352f
The deposit contract uses dynamic bytes for most of its event properties
https://github.com/ethereum/consensus-specs/blob/dev/solidity_deposit_contract/deposit_contract.sol#L19-L24
2021-10-14 15:33:55 +03:00
Zahary Karadjov dae7b6aeea
Bump nim-web3 2021-10-14 14:38:07 +03:00
Etan Kissling 25a22fae5f
bump `nim-stint` (#2969)
Upstream changes:
- Parsing should raise CatchableError for user inputs
- RangeError are actually defect and not catchable.
2021-10-14 14:13:51 +03:00
Zahary Karadjov fe8bbb2388
Merge stable into unstable 2021-10-14 13:32:42 +03:00
Zahary Karadjov dbbdbea7da Hotfix: Revert a nim-web3 change that interferes with Eth1 deposit syncing 2021-10-10 13:35:06 +02:00
tersec 3c14e5728d
use v1.1.2 test vectors (#2968) 2021-10-08 11:36:35 +00:00
tersec 5cc2acb44e
use v1.1.1 test vectors (#2954) 2021-10-05 02:26:56 +00:00
Kim De Mey 0b1e0772de
Bump nim-eth to latest on master (#2927) 2021-09-30 09:00:47 +02:00
Zahary Karadjov 66b9f33edd Definitions related to the new execution engine JSON-RPC API 2021-09-30 00:20:21 +03:00
kdeme 933879e298 Bump nim-eth to allow for tcp/udp ports to always be configured in ENR 2021-09-29 20:15:24 +03:00
Mamy Ratsimbazafy 4aed54200d
fix multithreading memory leak (#2920)
* fix multithreading memory leak

* remove destructors for Nim 1.2 taskpool
2021-09-29 18:09:53 +02:00
Zahary Karadjov cc5d85331b Altair config for mainnet
This still doesn't work properly because it leads to a change in
the "eth" ENR field of the client. Since the client uses the value
of this field to look for other nodes on the network, the change
effectively prevents us from finding peers.
2021-09-29 14:19:20 +03:00
tersec f2df006ee9
update to v1.1.0 (#2909) 2021-09-28 04:28:33 +00:00
tersec edad6e5e83
use v1.1.0-beta.5 test vectors (#2900) 2021-09-25 03:07:07 +00:00
Etan Kissling aaaae89445
bump nimbus-build-system (#2898)
Changes:
- QUICK_AND_DIRTY_NIMBLE
- fix build output color on macOS
2021-09-24 15:34:23 +02:00
Eugene Kabanov b566d4657f
REST /eth/v1/events API call implementation. (#2878)
* Placing callbacks into strategic places.

* Initial events call implementation.

* Post rebase fixes.

* Change addSyncContribution() implementation.

* Add `attestation-sent` event.
Remove gcsafe, raises from callbacks implementations.
Move `attestation-received` fire at the end of attestation processing.

* Address review comments.
2021-09-22 14:17:15 +02:00
Jacek Sieka 855d0257ff
update lighthouse bootnodes (#2880)
https://github.com/eth2-clients/eth2-networks/pull/60
2021-09-21 13:34:25 +02:00
Mamy Ratsimbazafy 0dad1ec091
bump taskpools: remove default CPU pinning 2eb22c61b3 2021-09-21 11:09:48 +02:00
Mamy Ratsimbazafy 77f4169ffe
bump BLST v0.3.5 (#2876) 2021-09-17 10:20:21 +02:00
Mamy Ratsimbazafy d1cb5b7220
Parallel attestation verification (#2718)
* Add parallel attestation verification

* Update tests, batchVerify doesn't use the threadpool with only single core (nim-blscurve update)

* bump nim-blscurve

* Debug info for failing eth2 test vectors

* remove submodule eth2-testnets

* verbose debugging of make failure on Windows (libbacktrace?)

* Remove CI debug mode

* initialization convention

* Fix new altair tests
2021-09-17 03:13:52 +03:00
Eugene Kabanov 7ab1856c04
SSZ encoded responses for REST API calls. (#2851) 2021-09-16 16:32:32 +03:00
Zahary Karadjov a71de3feda Address review comments 2021-09-14 12:12:49 +03:00
tersec 5ea9e0baf3
use v1.1.0-beta.4 test vectors (#2861) 2021-09-10 17:35:14 +00:00
Kim De Mey ec736bc578
Bump nim-eth vendor module (#2848) 2021-09-06 21:59:20 +02:00
Jacek Sieka 6d47d96c84
altair upgrade for prater (#2828)
and a few import fixes for free
2021-08-27 16:54:51 +00:00
Jacek Sieka 01596c45dd
cleanups and fixes (#2827)
* import cleanup
* fix json-rpc exception handlers
* avoid unnecessary presto client import
* introduce ForkedBeaconBlock, some altair logging
* url fixes
2021-08-27 11:00:06 +02:00
Ștefan Talpalaru 5f148d9a26
bump vendor/news - this time the "status" branch (#2824) 2021-08-27 08:32:51 +02:00
tersec 4492775c6b
add random tests and rename "Official" to "Ethereum Foundation" (#2826)
* add random tests and rename "Official" to "Ethereum Foundation"

* checkDir = true covers dirExists(...)

* invalidate CI EF fixtures cache

* more correct cache invalidation
2021-08-27 06:30:30 +00:00
tersec 8de2342f60
use v1.1.0-beta.3 test vectors (#2825) 2021-08-26 16:01:21 +00:00
Ștefan Talpalaru b113d69f48
bump vendor/news (#2816) 2021-08-25 14:03:42 +00:00
tersec 79b2539ef0
eth2.0-spec-tests -> consensus-spec-tests repo rename (#2807) 2021-08-23 07:13:30 +00:00
Jacek Sieka e6e69f5f54
module bumps (#2800) 2021-08-20 18:39:18 +00:00
Ștefan Talpalaru bc37a6eb63
Windows: fix build when '.' not in PATH (#2787) 2021-08-20 14:00:17 +02:00
Jacek Sieka a7a65bce42
disentangle eth2 types from the ssz library (#2785)
* reorganize ssz dependencies

This PR continues the work in
https://github.com/status-im/nimbus-eth2/pull/2646,
https://github.com/status-im/nimbus-eth2/pull/2779 as well as past
issues with serialization and type, to disentangle SSZ from eth2 and at
the same time simplify imports and exports with a structured approach.

The principal idea here is that when a library wants to introduce SSZ
support, they do so via 3 files:

* `ssz_codecs` which imports and reexports `codecs` - this covers the
basic byte conversions and ensures no overloads get lost
* `xxx_merkleization` imports and exports `merkleization` to specialize
and get access to `hash_tree_root` and friends
* `xxx_ssz_serialization` imports and exports `ssz_serialization` to
specialize ssz for a specific library

Those that need to interact with SSZ always import the `xxx_` versions
of the modules and never `ssz` itself so as to keep imports simple and
safe.

This is similar to how the REST / JSON-RPC serializers are structured in
that someone wanting to serialize spec types to REST-JSON will import
`eth2_rest_serialization` and nothing else.

* split up ssz into a core library that is independendent of eth2 types
* rename `bytes_reader` to `codec` to highlight that it contains coding
and decoding of bytes and native ssz types
* remove tricky List init overload that causes compile issues
* get rid of top-level ssz import
* reenable merkleization tests
* move some "standard" json serializers to spec
* remove `ValidatorIndex` serialization for now
* remove test_ssz_merkleization
* add tests for over/underlong byte sequences
* fix broken seq[byte] test - seq[byte] is not an SSZ type

There are a few things this PR doesn't solve:

* like #2646 this PR is weak on how to handle root and other
dontSerialize fields that "sometimes" should be computed - the same
problem appears in REST / JSON-RPC etc

* Fix a build problem on macOS

* Another way to fix the macOS builds

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2021-08-18 20:57:58 +02:00
Ștefan Talpalaru 9caf852dd4
cron CI: various fixes (#2594) 2021-08-18 14:40:03 +02:00
Jacek Sieka 63717531dc remove remaining traces of nim-prompt
upstream dead, as is feature in eth2
2021-08-16 21:56:50 +03:00
Jacek Sieka 584e70e1cb
bump faststreams (#2783) 2021-08-16 07:50:38 +02:00
Jacek Sieka 07728e4f7c
altair devnet 3 (#2782) 2021-08-15 19:50:52 +02:00
Jacek Sieka 92aba71604
Pyrmont altair fork (#2771)
Prater upgrading to altair at epoch 61650 -
https://github.com/eth2-clients/eth2-networks/pull/56

also includes altair-devnet-2 support in binary
2021-08-10 10:16:43 +02:00
Eugene Kabanov b5dd6ca11c
Bump `chronos` and `http-utils` to fix number of issues. (#2765) 2021-08-06 18:37:51 +00:00
Ștefan Talpalaru e576064788
bump nimbus-build-system (#2749) 2021-07-30 10:27:56 +02:00
tersec 941cc125a3
update to v1.1.0-beta.2 tests, with exception for altair sync committees (#2736)
* update to v1.1.0-beta.2 tests, with exception for altair sync committees

* bootstrap node can't drop any attestations
2021-07-26 17:34:13 +00:00
nbc-bump-bot[bot] f68061b2d8
nim-libp2p auto bump (#2647)
* auto-bump nim-libp2p

* update peerhook for libp2p

Co-authored-by: = <Menduist@users.noreply.github.com>
Co-authored-by: Tanguy Cizain <tanguycizain@gmail.com>
2021-07-26 12:36:17 +00:00
Kim De Mey 3e3e17fec3
Remove raises Exception and bump nim-eth (#2729) 2021-07-19 14:25:11 +02:00
Jacek Sieka 2d6a661ac6
Syncv2 (#2723)
* bump libp2p

* altair sync v2

Use V2 sync requests after the altair fork has happened, according to
the wall clock

* Fix the behavior of the v1 req/resp calls after Altair

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2021-07-15 21:01:07 +02:00
tersec 11541ff61b
bump nim-eth2-scenarios to use v1.1.0-beta.1 test vectors (#2724) 2021-07-15 09:04:51 +00:00
Jacek Sieka 3f9c1fdf4e
More RuntimeConfig cleanup (#2716)
* remove from BeaconChainDB (doesn't depend on runtime config)
* eth2-testnets -> eth2-networks
* use `cfg` name throughout
2021-07-13 16:27:10 +02:00
Eugene Kabanov 3b6f4fab4a
New validator client using REST API. (#2651)
* Initial commit.

* Exporting getConfig().

* Add beacon node checking procedures.

* Post rebase fixes.

* Use runSlotLoop() from nimbus_beacon_node.
Fallback implementation.
Fixes for ETH2 REST serialization.

* Add beacon_clock.durationToNextSlot().
Move type declarations from beacon_rest_api to json_rest_serialization.
Fix seq[ValidatorIndex] serialization.
Refactor ValidatorPool and add some utility procedures.
Create separate version of validator_client.

* Post-rebase fixes.
Remove CookedPubKey from validator_pool.nim.

* Now we should be able to produce attestations and aggregate and proofs.
But its not working yet.

* Debugging attestation sending.

* Add durationToNextAttestation.
Optimize some debug logs.
Fix aggregation_bits encoding.
Bump chronos/presto.

* Its alive.

* Fixes for launch_local_testnet script.
Bump chronos.

* Switch client API to not use `/api` prefix.

* Post-rebase adjustments.

* Fix endpoint for publishBlock().

* Add CONFIG_NAME.
Add more checks to ensure that beacon_node is compatible.

* Add beacon committee subscription support to validator_client.

* Fix stacktrace should be an array of strings.
Fix committee subscriptions should not be `data` keyed.

* Log duration to next block proposal.

* Fix beacon_node_status import.

* Use jsonMsgResponse() instead of jsonError().

* Fix graffityBytes usage.
Remove unnecessary `await`.
Adjust creation of SignedBlock instance.
Remove legacy files.

* Rework durationToNextSlot() and durationToNextEpoch() to use `fromNow`.

* Fix race condition for block proposal and attestations for same slot.
Fix local_testnet script to properly kill tasks on Windows.
Bump chronos and nim-http-tools, to allow connections to infura.io (basic auth).

* Catch services errors.
Improve performance of local_testnet.sh script on Windows.
Fix race condition when attestation producing.

* Post-rebase fixes.

* Bump chronos and presto.

* Calculate block publishing delay.
Fix pkill in one more place.

* Add error handling and timeouts to firstSuccess() template.
Add onceToAll() template.
Add checkNodes() procedure.
Refactor firstSuccess() template.
Add error checking to api.nim calls.

* Deprecated usage onceToAll() for better stability.
Address comment and send attestations asap.

* Avoid unnecessary loop when calculating minimal duration.
2021-07-13 13:15:07 +02:00
tersec 667a01ef42
bump nim-eth2-scenarios to stop downloading alpha.5 test vectors (#2714) 2021-07-12 21:56:48 +00:00
Jacek Sieka 23eea197f6
Implement split preset/config support (#2710)
* Implement split preset/config support

This is the initial bulk refactor to introduce runtime config values in
a number of places, somewhat replacing the existing mechanism of loading
network metadata.

It still needs more work, this is the initial refactor that introduces
runtime configuration in some of the places that need it.

The PR changes the way presets and constants work, to match the spec. In
particular, a "preset" now refers to the compile-time configuration
while a "cfg" or "RuntimeConfig" is the dynamic part.

A single binary can support either mainnet or minimal, but not both.
Support for other presets has been removed completely (can be readded,
in case there's need).

There's a number of outstanding tasks:

* `SECONDS_PER_SLOT` still needs fixing
* loading custom runtime configs needs redoing
* checking constants against YAML file

* yeerongpilly support

`build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG`

* load fork epoch from config

* fix fork digest sent in status
* nicer error string for request failures
* fix tools

* one more

* fixup

* fixup

* fixup

* use "standard" network definition folder in local testnet

Files are loaded from their standard locations, including genesis etc,
to conform to the format used in the `eth2-networks` repo.

* fix launch scripts, allow unknown config values

* fix base config of rest test

* cleanups

* bundle mainnet config using common loader
* fix spec links and names
* only include supported preset in binary

* drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 15:01:38 +02:00
Ștefan Talpalaru 17da9aaa09
fix metrics on Windows (#2707) 2021-07-07 10:19:41 +02:00
Mamy Ratsimbazafy 38a5bc8de3
bump blscurve closes #2693 and #2695 (#2700) 2021-07-06 14:31:54 +02:00
tersec df381d0cf0
update spec test fixtures to v1.1.0-alpha.8 (#2687) 2021-06-29 16:47:09 +00:00
Ștefan Talpalaru 39591ac858
bump two submodules (#2657) 2021-06-17 17:07:57 +00:00
tersec 1c3314f08b
update to Altair as of v1.1.0-alpha.7 (#2649)
* update to Altair as of v1.1.0-alpha.7

* introduce Altair types into attestation pool

* avoid allocating/copying pubkeys excessively in get_next_sync_committee()
2021-06-14 17:42:46 +00:00
Jacek Sieka d859bc12f0
write uncompressed validator keys to database (#2639)
* write uncompressed validator keys to database

Loading 150k+ validator keys on startup in compressed format takes a lot
of time - better store them in uncompressed format which makes behaviour
just after startup faster / more predictable.

* refactor cached validator key access
* fix isomorphic cast to work with non-var instances
* remove cooked pubkey cache - directly use database cache in chaindag
as well (one less cache to keep in sync)
* bump blscurve, introduce loadValid for known-to-be-valid keys
2021-06-10 10:37:02 +03:00
nbc-bump-bot[bot] 1a76007858
auto-bump nim-libp2p (#2591)
Co-authored-by: = <arnetheduck@users.noreply.github.com>
2021-06-04 12:36:03 +02:00
tersec 820a6f65d5
use v1.1.0-alpha.6 test vectors (#2618)
* bump nim-eth2-vendors and use v1.1.0-alpha6 test vectors

* continue to download alpha 5 test vectors for const sanity checks
2021-05-30 09:51:01 +00:00
Jacek Sieka d16da06c92 ncli_db: validator performance database tool
Record attestation performance per epoch in sqlite database
2021-05-27 19:14:26 +03:00
Ștefan Talpalaru 9ddf7fea23 dist: use Ubuntu-18.04 to build ARM64 binaries 2021-05-26 16:51:06 +03:00
Jacek Sieka 80ed54378c
nim-eth: fix commit 2021-05-26 09:40:08 +02:00
Jacek Sieka eebc828778
create new database in separate file (#2596)
The V1 table structure shows great improvements in performance, but if
there's an old `kvstore` without rowid:s, these benefits are nullified:
reorgs during writes and deletes remain expensive (even if the
degradation is reduced somewhat).

This PR creates the tables in a new file instead, and uses the old file
as a read-only store - this has several interesting properties:

* the old database is left completely untouched - this guarantees that
downgrades work smooth (they'll only need to resync their missing
portions)
* starting sync after this PR means only a v1 database is created
* v0 databases stick around - no migration is performed (for now)

Future PR:s can introduce migration of the data from one database to
another - a simply copy will take hours which is downtime we want to
avoid - at that point, it might make sense to migrate straight to era
files instead.
2021-05-26 09:07:18 +02:00
tersec d69e06e519
bump nim-eth2-scenarios to stop downloading v1.0.1 test vectors (#2598) 2021-05-24 19:31:48 +00:00
tersec 056ff59595
bump eth2-testnets (#2589) 2021-05-21 13:09:35 +00:00
tersec 1c05865656
bump nim-eth2-scenarios to download v1.1.0-alpha.5 test vectors (#2588) 2021-05-21 10:54:59 +00:00
Zahary Karadjov dc49a51654
Merge stable into unstable (take 2) 2021-05-20 13:52:09 +03:00
Zahary Karadjov b7aa30adfd
Merge stable into unstable 2021-05-20 13:50:40 +03:00
Eugene Kabanov cf06c4e87e
Make REST server more compatible with Lighthouse and Teku validator clients. (#2575)
* Allow REST server to parse arrays with comma delimiter.

* Fix compilation issues because of new presto framework.
2021-05-18 12:24:57 +02:00
tersec fb88309b34
bump nim-eth2-scenarios to download v1.1.0-alpha.4 test vectors (#2574) 2021-05-18 11:59:46 +02:00
Zahary Karadjov 6b14d33ad6
Bump Confutils
This fixes an issue with the --bootstrap-file parameter which was
accidentally considered mandatory due to an upstream issue.
2021-05-18 11:02:01 +03:00
Ștefan Talpalaru 1f1367d09a
macOS binary distribution (both AMD64 and ARM64) 2021-05-17 21:42:28 +03:00
Zahary Karadjov f6a1f602b4
Handle changes in the latest version of Confutils 2021-05-17 21:42:28 +03:00
Zahary Karadjov b9924214ab
Better error-handling for the slashingdb import/export feature
* Error when specifying an invalid --data-dir (or --validator-dir)
* Error when entering an invalid validator public key (e.g. invalid hex value)
* Warning when attempting to export a validator not present in the local database

Some unnecessary remains of the v1 mode has been removed as well
2021-05-17 21:42:23 +03:00
Jacek Sieka 97f4e1fffe
Db1 cont (#2573)
* Revert "Revert "Upgrade database schema" (#2570)"

This reverts commit 6057c2ffb4.

* ssz: fix loading empty lists into existing instances

Not a problem earlier because we didn't reuse instances

* bump nim-eth

* bump nim-web3
2021-05-17 18:37:26 +02:00
Zahary Karadjov 5c313b958e
Simplify the slashing db import/export CLI 2021-05-17 17:12:03 +03:00
tersec 6057c2ffb4
Revert "Upgrade database schema" (#2570)
This reverts commit 22ddf74752.
2021-05-17 06:34:44 +00:00
Jacek Sieka 2c257438a0 Upgrade database schema
This is a minimal performance hotfix for the kvstore table, such that
new databases get a kvstore table with rowid, fixing the most urgent
performance problem we have with pruning.
2021-05-14 20:08:07 +03:00
Jacek Sieka 22ddf74752 Upgrade database schema
The `kvstore` design we're using now turns out to not be the best way to
use `sqlite` - in particular, there are some significant benefits to
using rowid in certain situations and to keep data in separate tables.

With this branch, there are massive improvements in startup time
(seconds instead of minutes) and state/block storage and pruning times
(milliseconds instead of seconds) - these improvements can in particular
be seen on slow drives and translate directly into better attestation
performance.

* update kvstore to new keyspace design
* remove `DirStoreRef` and the hidden `--state-db-kind` option - this
was an experiment to store large blobs in files, but with the new
kvstore, there's no compelling reason to do so
* remove `DbMap` - unused and would need updating for new keyspace
design
* introduce separate tables for each data type (blocks, states etc)
* remove "WITHOUT ROWID" pessimization for tables with large blobs
* close DbSeq statements explicitly (and earlier)
* store beacon block summaries in separate table, without SSZ
compression and load them all with single query on startup
* stop storing backwards compat full states
* mark genesis beacon block as trusted
* avoid faststreams when loading SSZ data
* remove `DisagreementBehavior` (unused)
2021-05-14 20:05:23 +03:00
tersec f9b964ca5d
bump nim-eth2-scenarios for merge test vectors (#2553) 2021-05-10 13:56:31 +00:00
Jacek Sieka 5cd5da74c4
Gossipsub unsubscribe fixes (fixes #2540) (#2545) 2021-05-07 06:31:28 +02:00
Jacek Sieka 01b404f43d
bump chronos (#2544) 2021-05-06 18:20:37 +02:00
tersec dd43a2c3b0
bump nim-eth2-scenarios to get merge SSZ test vectors (#2541) 2021-05-05 15:35:36 +00:00
Ștefan Talpalaru 2018a55c51
bump a few submodules (#2529) 2021-05-05 08:55:39 +02:00
Jacek Sieka ce49da6c0a
Introduce unittest2 and junit reports (#2522)
* Introduce unittest2 and junit reports

* fix XML path

* don't combine multiple CI runs

* fixup

* public combined report also

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-28 18:41:02 +02:00
nbc-bump-bot[bot] d4cda02c97
auto-bump nim-libp2p (#2523)
Co-authored-by: = <sinkingsugar@users.noreply.github.com>
2021-04-28 09:57:25 +02:00
cheatfate 18da0f8928 Bump nim-chronos to fix issues in httpserver.
Bump nim-http-utils to fix issues in httpserver.
2021-04-28 00:00:00 +03:00
nbc-bump-bot[bot] 8a3c070337
auto-bump nim-libp2p (#2517)
Co-authored-by: = <sinkingsugar@users.noreply.github.com>
2021-04-23 12:54:06 +02:00
Ștefan Talpalaru 5b32f7ca17 Nim-1.2.12 2021-04-19 10:53:58 +03:00
Jacek Sieka 4959da78a8
mem fixes / libp2p (#2510)
* mem fixes / libp2p

* bump
2021-04-18 12:30:25 +02:00
Ștefan Talpalaru 9e439f19c8
bump nim-metrics (#2511) 2021-04-18 10:22:59 +02:00
cheatfate 14742890d6 Fix browsers unable to access HTTP endpoints (HTTP 400 error). 2021-04-14 22:05:00 +03:00
Jacek Sieka 4ed2e34a9e Revamp attestation pool
This is a revamp of the attestation pool that cleans up several aspects
of attestation processing as the network grows larger and block space
becomes more precious.

The aim is to better exploit the divide between attestation subnets and
aggregations by keeping the two kinds separate until it's time to either
produce a block or aggregate. This means we're no longer eagerly
combining single-vote attestations, but rather wait until the last
moment, and then try to add singles to all aggregates, including those
coming from the network.

Importantly, the branch improves on poor aggregate quality and poor
attestation packing in cases where block space is running out.

A basic greed scoring mechanism is used to select attestations for
blocks - attestations are added based on how much many new votes they
bring to the table.

* Collect single-vote attestations separately and store these until it's
time to make aggregates
* Create aggregates based on single-vote attestations
* Select _best_ aggregate rather than _first_ aggregate when on
aggregation duty
* Top up all aggregates with singles when it's time make the attestation
cut, thus improving the chances of grabbing the best aggregates out
there
* Improve aggregation test coverage
* Improve bitseq operations
* Simplify aggregate signature creation
* Make attestation cache temporary instead of storing it in attestation
pool - most of the time, blocks are not being produced, no need to keep
the data around
* Remove redundant aggregate storage that was used only for RPC
* Use tables to avoid some linear seeks when looking up attestation data
* Fix long cleanup on large slot jumps
* Avoid some pointers
* Speed up iterating all attestations for a slot (fixes #2490)
2021-04-13 20:24:02 +03:00
cheatfate 79d4735355 Address #2491. 2021-04-13 17:07:41 +03:00
Dmitriy Ryajov 03f478748f bump libp2p to lastest master 2021-04-12 20:28:39 +03:00
Dmitriy Ryajov 5eebf4b5f5 bump libp2p 2021-04-12 20:28:39 +03:00
Dmitriy Ryajov 4f4901b10a use master builders 2021-04-12 20:28:39 +03:00
Dmitriy Ryajov 920de91c37 use libp2p builder 2021-04-12 20:28:39 +03:00
cheatfate cc3851ec02 Bump nim-presto. 2021-04-09 21:42:13 +03:00
cheatfate 72695dd62a Rest API initial implementation. 2021-04-09 21:42:13 +03:00
cheatfate 3d701d8973 Add nim-presto submodule. 2021-04-09 21:42:13 +03:00
cheatfate c4d891f583 Fix sync_manager.nim to return proper status.
Bump REST API dependencies.
2021-04-09 21:42:13 +03:00
Zahary Karadjov 9776fbfe17
Merge branch 'version-1.1.0' into unstable 2021-04-08 20:50:06 +03:00
Zahary Karadjov ba59dd85cd Support for fallback web3 providers; Fix resource leaks on Eth1 monitor restarts 2021-04-08 12:46:42 +03:00
tersec 5f5e3e43c3
bump num-eth2-scenarios from v1.1.0-alpha.2 to v1.1.0-alpha3 (#2481) 2021-04-07 13:52:17 +00:00
Jacek Sieka adec9d878e
bump chronos (#2476) 2021-04-06 13:58:36 +02:00
tersec 49a5667288
update some v1.1.0 alpha1 to alpha2 (#2457)
* update some v1.1.0 alpha1 to alpha2

* remove unused getDepositMessage overload and move other out of datatypes/base

* bump nim-eth2-scenarios to download v1.1.0-alpha.2 test vectors

* construct object rather than result
2021-03-29 19:17:48 +00:00
Jacek Sieka 74732a23fe
json cleanups (#2456)
* move json-rpc specific marshalling to rpc
* serialize Epoch/Slot with cast to avoid Defect
* avoid a few eth1 deps
* simplify imports
2021-03-26 15:11:06 +01:00
Jacek Sieka 2695cfa864
EH cleanup (#2455)
almost 100% raises in nimbus-eth2 now!

* fix some rare exception-related crashes in json-rpc
2021-03-26 07:52:01 +01:00
Zahary Karadjov 2eacfc4685 Bump modules to take advantage of the new Json format flavors support
Since quite a lot of additional procs were now compiled as generics, this lead to compiler bugs that had
to be worked-around:

* The `Domain` type was renamed to `Eth2Domain` to avoid compilation errors
  due to conflicts with `nativesockets.Domain`.
  Similarly, `eth2_network.KeyPair` was renamed to `NetKeyPair`.

* A new more robust version of `hexToByteArray` was added to stew
2021-03-25 09:37:35 +02:00
Jacek Sieka 8b76ceed52
Fix minor exception effect issues (#2448)
Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.
2021-03-24 17:20:55 +01:00
Zahary Karadjov a727fc8e52
Merge stable into unstable 2021-03-24 03:24:20 +02:00
Zahary Karadjov 8391d3f83e
v1.0.12 2021-03-24 03:23:14 +02:00
Zahary Karadjov 5fd82ddaf5
Merge stable into unstable 2021-03-22 23:13:05 +02:00
Jacek Sieka d1357801e7
bumps 2021-03-22 20:53:27 +01:00
Zahary Karadjov af0d302398
Use the latest bootstrap nodes for Prater 2021-03-22 18:37:32 +02:00
Jacek Sieka 3743df6434
introduce unittest2 (#2437)
This commit introduces `unittest2`, driven by the need to fix several
long-standing issues with `unittest`, in particular exception handling,
test isolation, timing and others.

Upgrading to `unittest2` generally requires updating the unittest import
and potentially fixing the code to support the new `one-proc-per-test`
setup.

* nim-unittest2 has been moved from Stefans repository to status-im -
https://github.com/status-im/nim-unittest2/pull/2 fixes some exception
handling issues and brings it in line with latest nim devel - a notable
difference to unittest is that each test is run inside a proc which
gives it a fresh stack and removes a lot of globals, meaning tests
should run under conditions more similar to "normal" code and interfere
with each other less - this is a crucial first step to improving the
testing experience in general, that stefan implemented for the
multithreading support
* the multithreading in unittest2 is optional, and disabled by default
* nim-testutils contains utilities for fuzzing, as well as other
"large:ish" features that have dependencies -
https://github.com/status-im/nim-testutils/pull/27 pulls in unittest2 so
unittest2 can be kept "low-dependency"
* nim-chronos needs the above unittest changes to test its own new,
stricter exception handling -
https://github.com/status-im/nim-chronos/pull/166 - the strictess
introduced is expected to increase with time
2021-03-22 08:21:22 +01:00
Jacek Sieka 01fe564e46
chronos-based metrics (#2432)
This opens up the road for removing `-d:insecure` for metrics.
2021-03-19 10:36:38 +01:00
Zahary Karadjov c519c6012e
Merge branch 'stable' into unstable 2021-03-15 21:52:12 +02:00
Zahary Karadjov f3fc551ea7
Support for the prater testnet 2021-03-15 21:51:24 +02:00
nbc-bump-bot[bot] 2201480bd9
auto-bump nim-libp2p (#2409)
Co-authored-by: = <sinkingsugar@users.noreply.github.com>
2021-03-15 17:39:10 +00:00
Jacek Sieka eb830c7968
Merge remote-tracking branch 'origin/stable' into unstable 2021-03-15 17:29:43 +01:00
Jacek Sieka f5180e0840
Revert "Nim-1.2.10" (#2411)
This reverts commit 12e15a0b8d.
2021-03-15 16:26:41 +01:00
Ștefan Talpalaru c5035c6eca dist: enable libbacktrace on ARM and ARM64
(and make sure it will work on Windows, when enabled)
2021-03-15 14:38:39 +02:00
Ștefan Talpalaru 12e15a0b8d Nim-1.2.10 2021-03-15 14:29:48 +02:00
nbc-bump-bot[bot] 4fcff4cd27
auto-bump nim-libp2p (#2406)
Co-authored-by: = <sinkingsugar@users.noreply.github.com>
2021-03-15 07:36:36 +00:00
Zahary Karadjov 77ee210762
v1.0.10 2021-03-10 23:33:37 +02:00
zah 33c7f264bc
Bump news to fix resource leaks (#2388) 2021-03-10 12:22:41 +00:00
nbc-bump-bot[bot] a88d17dc04
auto-bump nim-libp2p (#2389)
Co-authored-by: = <arnetheduck@users.noreply.github.com>
2021-03-09 14:28:28 +01:00
Mamy André-Ratsimbazafy d425aabce2 BLST v0.3.3 2021-03-08 14:43:48 +02:00
kdeme 5f750f84b4 Use setupAddress for better IP and ports configuration 2021-03-06 00:09:09 +02:00
kdeme 769225b591 Bump nimbus-build-system to use nim-1-2-6-RC1-hotfixed
This should practically solve the segfaults we have been seeing in
issue #2134
2021-03-04 23:00:44 +02:00
nbc-bump-bot[bot] 9408a9af1c
auto-bump nim-libp2p (#2373)
Co-authored-by: = <arnetheduck@users.noreply.github.com>
2021-03-04 10:14:00 +01:00
nbc-bump-bot[bot] 965972dd0b
auto-bump nim-libp2p (#2360)
Co-authored-by: = <sinkingsugar@users.noreply.github.com>
2021-03-01 10:26:16 +00:00
nbc-bump-bot[bot] 037a3e8946
auto-bump nim-libp2p (#2359)
Co-authored-by: = <sinkingsugar@users.noreply.github.com>
2021-02-26 09:43:35 +00:00
Dustin Brody f14e7babb6 update eth2 specs to version v1.0.1 2021-02-25 14:21:59 +02:00
Ștefan Talpalaru 16abf2989b bump NimYAML 2021-02-25 14:20:26 +02:00
Ștefan Talpalaru 22620afa8f bump NimYAML for Nim-1.4 compatibility 2021-02-25 14:20:26 +02:00
nbc-bump-bot[bot] 45087f36bd
auto-bump nim-libp2p (#2350)
Co-authored-by: = <sinkingsugar@users.noreply.github.com>
2021-02-25 11:25:00 +00:00
Eugene Kabanov 8478345619
Bump nim-bearssl to include latest fixes in submodule repository and cacert.pem. (#2354) 2021-02-24 19:22:40 +01:00
Zahary Karadjov 3f6834cce7
Fix a crash occurring with --slashing-db-kind=both 2021-02-22 21:05:08 +02:00
Zahary Karadjov 7e9648e52c
Use the latest Teku bootstrap nodes 2021-02-22 17:16:53 +02:00
Zahary Karadjov 5f924c0038 Bump modules; Compile with DEBUG logs by default 2021-02-19 08:44:13 +02:00
Kim De Mey aa29be288e
Bump eth2-testnets for updated LH bootstrap nodes (#2322) 2021-02-15 17:56:34 +01:00
Jacek Sieka 946608b572
bump json-rpc (#2321) 2021-02-15 13:47:57 +01:00
nbc-bump-bot[bot] ad66ba507e
auto-bump nim-libp2p (#2317)
Co-authored-by: = <sinkingsugar@users.noreply.github.com>
2021-02-13 06:16:39 +00:00
nbc-bump-bot[bot] 8192015b86
auto-bump nim-libp2p (#2315)
Co-authored-by: = <sinkingsugar@users.noreply.github.com>
2021-02-12 05:53:38 +00:00
nbc-bump-bot[bot] 0419a4122e
auto-bump nim-libp2p (#2309)
Co-authored-by: = <dryajov@users.noreply.github.com>
2021-02-11 02:20:39 +00:00
Jacek Sieka f012d7060b
better error message on disk / database issues (#2307)
bumps stew for better result defects as well
2021-02-10 13:21:06 +01:00
nbc-bump-bot[bot] c13205035e
auto-bump nim-libp2p (#2305)
Co-authored-by: = <dryajov@users.noreply.github.com>
2021-02-10 05:32:19 +00:00
Mamy Ratsimbazafy 03f47c8f2f
Slashing protection refactor - EIP 3076 (#2094)
* Create CLI tool for slashing export

* Use SQLite as a DB instead of a KV-store

* Keeps v1 and v2 DBs around

* Uses the same schema as Lighthouse v1.1.0

* Passes all interchange tests + skeleton of finalization pruning

* Removes tests that would violate v5 / minimal slashing DB and MinSlot rules

* Migration tool added using low-watermark scheme for faster migration of large number of validators
2021-02-09 17:23:06 +02:00
nbc-bump-bot[bot] 93cfc2d446
auto-bump nim-libp2p (#2300)
Co-authored-by: = <sinkingsugar@users.noreply.github.com>
2021-02-08 08:32:47 +01:00
Jacek Sieka ece50c4706
bump nim-json-rpc (fixes https://github.com/status-im/nim-json-rpc/issues/93) (#2299) 2021-02-07 22:30:45 +01:00
nbc-bump-bot[bot] 0b560cc6fb
auto-bump nim-libp2p (#2296)
Co-authored-by: = <sinkingsugar@users.noreply.github.com>
2021-02-06 09:50:08 +01:00
Zahary Karadjov 1394ef37fb
Tentative fix for the freezing Eth1 monitor activity 2021-02-04 17:01:47 +02:00
Ștefan Talpalaru 80c11546ff Windows binary release
CI: use both cores on GitHub Actions and set timeouts for the local testnet tests
2021-02-04 10:25:44 +02:00
Eugene Kabanov c4c04197c5
Bump nim-http-utils. (#2290) 2021-02-03 19:05:30 +01:00
Ștefan Talpalaru cadb1aae15
bump submodules (#2279)
* bump submodules

* undo a couple of bumps (32-bit failure)

* launch_local_testnet.sh: timeout and various fixes

* undo nim-http-utils bump (breaks RPC for VC)
2021-02-02 19:02:52 +01:00
Kim De Mey 73d9c2aa53
Add enr auto update cli option and bump nim-eth (#2278) 2021-02-02 09:07:21 +01:00
Zahary Karadjov fa99c3b417 Fix #2261
Also bumps Confutils to allow setting the hidden --web3-mode param
(to allow testing the eth1 syncing without validators)
2021-01-30 01:32:20 +02:00
Kim De Mey a3678d6bf0
Bump eth2-testnets for mainnet nimbus bootstrap nodes (#2276) 2021-01-29 14:07:38 +01:00
nbc-bump-bot[bot] 3e89aa0f4f
auto-bump nim-libp2p (#2275)
Co-authored-by: = <sinkingsugar@users.noreply.github.com>
2021-01-29 09:48:27 +01:00
Dmitriy Ryajov 1117993a24
Libp2p connection limits (#2272)
* max peers gets set as connection limits

* pass max connections
2021-01-28 07:31:29 +01:00
tersec 4de9875cdf
Revert "pass max peers to libp2p (#2265)" (#2270)
This reverts commit 64e4d74f87.
2021-01-26 17:53:52 +01:00
Dmitriy Ryajov 64e4d74f87
pass max peers to libp2p (#2265) 2021-01-26 10:35:22 +01:00
Mamy Ratsimbazafy 70a03658e3
Block validation flow v2 + Batch (serial) sig verification (#2250)
* bump nim-blscurve

* Outline the block validation flow

* introduce the SigVerified types, pass the tests

* Split clearance/quarantine to prepare for batch crypto verif

* Add a batch signature collector

* Make clearance use SigVerified block and split verification between crypto and state transition

* Always use signedBeaconBlock for the onBlockAdded callback

* RANDAO signing_root is the epoch instead of the full block

* Support skipping BLS for testing

* Fix compilation of the validator client

* Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c)

* address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561819858

* address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828025

* onBlockAdded callback should use TrustedSignedBeaconBlock https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561837261

* address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828946

* Use the application RNG: https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561815336

* Improve codegen of conversion zero-cost)

* Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch)

* Graceful handle rogue pubkeys and signatures now that those are lazy-loaded
2021-01-25 20:45:48 +02:00
nbc-bump-bot[bot] f0f292f509
auto-bump nim-libp2p (#2245)
Co-authored-by: = <dryajov@users.noreply.github.com>
2021-01-22 08:43:22 +01:00
Ștefan Talpalaru 9a0cac61c8 only ignore untracked files in submodules 2021-01-21 16:09:28 +02:00
Jacek Sieka 76f15302a7 better timesharing in eth2 processor
* use `idleAsync` to more evenly divide cpu attention when syncing in
particular - this gives networking better latency
* more strict exception handling in eth2_processor
2021-01-18 22:27:34 +02:00
nbc-bump-bot[bot] a0689e4237
auto-bump nim-libp2p (#2239)
Co-authored-by: = <sinkingsugar@users.noreply.github.com>
2021-01-15 21:15:58 +09:00
nbc-bump-bot[bot] bb0a022968
auto-bump nim-libp2p (#2238)
Co-authored-by: = <sinkingsugar@users.noreply.github.com>
2021-01-15 15:50:14 +09:00
Giovanni Petrantoni 295e3c9c73
Topics validation and direct peers (#2237)
* pick the right libp2p branch

* add topics validation
2021-01-15 04:17:06 +00:00
nbc-bump-bot[bot] ce64da1fcd
auto-bump nim-libp2p (#2235)
Co-authored-by: = <arnetheduck@users.noreply.github.com>
2021-01-15 01:39:09 +00:00
Kim De Mey 66d8f317cd
Use async queryRandom instead of inefficient randomNodes to discover nodes (#2211) 2021-01-14 08:58:13 +01:00
nbc-bump-bot[bot] f5efcc8669
auto-bump nim-libp2p (#2228)
Co-authored-by: = <sinkingsugar@users.noreply.github.com>
2021-01-12 04:27:32 +01:00
nbc-bump-bot[bot] be3fac8495
auto-bump nim-libp2p (#2206)
Co-authored-by: = <sinkingsugar@users.noreply.github.com>
2021-01-11 09:53:20 +01:00
Jacek Sieka 93b434d0ac
bumps (#2214)
* bumps

several modules pinned to merged branches

* faststreams master broken, go back
2021-01-07 13:12:37 +01:00
Ștefan Talpalaru e58a355a1c
CI release: ARM and ARM64 builds (#2213) 2021-01-07 10:19:29 +01:00
Jacek Sieka 52ca12c79f
bump nat, avoid one more finalizer (#2209) 2020-12-31 15:58:31 +01:00
Giovanni Petrantoni ed24f60f70
remove async from sub/unsub (#2197)
* remove await/async from sub/unsub

* fix unsubscribe wrong key (missed _snappy)

* use the right libp2p commit hash

* remove unused async

* fix inspector

* fix subnet calculation in RPC and insert broadcast attestations into node's pool

* unify codepaths to ensure only mostly-checked-to-be-valid attestations enter the pool, even from node's own broadcasts

* update attestation pool tests for new validateAttestation param

Co-authored-by: Dustin Brody <tersec@users.noreply.github.com>
2020-12-24 09:48:52 +01:00
Jacek Sieka 0f8a3a5ae8
checkpoint database at end of each slot (#2195)
* checkpoint database at end of each slot

To avoid spending time on synchronizing with the file system while doing
processing, the manual checkpointing mode turns off fsync during
processing and instead checkpoints the database when the slot has ended.

From an sqlite perspecitve, in WAL mode this guaranees database
consistency but may lead to data loss which is fine - anything missing
from the beacon chain database can be recovered on the next startup.

* log sync status and delay in slot start message

* bump
2020-12-18 22:01:24 +01:00
Jacek Sieka 452042b17b
msgid formatting, libp2p bump (#2198)
* msgid formatting, libp2p bump

* fix libp2p commit
2020-12-18 18:45:27 +00:00
Jacek Sieka ca2a09d424 bump secp 2020-12-16 19:41:12 +02:00
Ștefan Talpalaru ff23ca27c2
bump vendor/nimbus-benchmarking 2020-12-16 14:11:11 +01:00
Jacek Sieka b0086c5d99
bump libp2p (#2187)
* bump libp2p

* also stew
2020-12-16 09:36:51 +01:00
Jacek Sieka 4e191a06ac use stew/leb128
* 3 varint parsers -> 1
* 1/3 memory usage / allocations when writing stuff to libp2p
2020-12-15 17:06:15 +02:00
Ștefan Talpalaru e52c88386b
build warnings (#2179) 2020-12-14 16:45:31 +00:00
Kim De Mey f49fcb9df3
Bump nim-eth for discovery v5 ip limits (#2085) 2020-12-11 14:35:08 +01:00
Ștefan Talpalaru e59437c5e2
new Make target: "benchmarks" (#2174)
It runs some benchmarks, collects the output and generates HTML charts.
2020-12-11 12:12:43 +01:00
Mamy Ratsimbazafy 2b3b219c77
Bump blst: perf + 32-bit platforms support + pure C fallback + drop Miracl fallback tests (#2152)
* bump BLST

* Cached subgroup checks

* Bump BLST - BLST now has pure C fallback for all arch (Miracl unneeded)

* Drop testing Miracl fallback in Nimbus

* Charting uncharted waters: 32-bit should imply -mno-adx
2020-12-10 10:18:07 +01:00
Jacek Sieka c4d9d5f64e
bump nim-stew (no change) 2020-12-09 17:24:25 +01:00
Jacek Sieka 0b33183341
bumps (#2154)
* avoid dangling cstring returns in Result, fixing some low-risk crash
potential
* bump libp2p for connection race fix and a few cleanups
2020-12-07 17:09:05 +01:00
Zahary Karadjov 4feb0a308e Fix #2125 (ETH status bar display); Bump LibP2P 2020-12-02 00:03:59 +02:00
Zahary Karadjov 4eaf29875a Less risk for Eth1-induced delays in block proposal 2020-12-01 15:38:50 +02:00
Zahary Karadjov 38f7558e50 Work around a strange codegen issue to fix local sim in CI; Bump LibP2P 2020-12-01 15:38:00 +02:00
Jacek Sieka 7e4fd325b5
bump nim-libp2p 2020-11-29 14:47:19 +01:00
= e1fb2d98fb auto-bump nim-libp2p 2020-11-28 09:20:02 +00:00
Jacek Sieka d16e127daf
Peer stuff (#2084)
* Revert "Revert "Full "node" RPC calls implementation and fixes to peer lifetime states. (#2065)" (#2082)"

This reverts commit 7cc3dc8027.

* fix nil disconnectedFut crash

* fixes

don't resetPeer, it causes peer miscounts

* disconnect disconnecting peers

...when there's a race.

* avoid connection spamming

* never decrease SeenTable timeout
* only recover ENR for known peers

* seen only when really disconnected
2020-11-26 20:23:45 +01:00
Giovanni Petrantoni 9b9aa93cd4
Seenttl by spec (#2083)
* add rpc call to get chronos futures at runtime

* add seenTTL gossip parameter

Co-authored-by: Jacek Sieka <jacek@status.im>
2020-11-26 13:08:22 +01:00
Jacek Sieka 7cc3dc8027
Revert "Full "node" RPC calls implementation and fixes to peer lifetime states. (#2065)" (#2082)
This reverts commit d041287a4f.
2020-11-26 09:05:23 +01:00
Eugene Kabanov d041287a4f
Full "node" RPC calls implementation and fixes to peer lifetime states. (#2065)
* Initial commit.

* Fix log lines and compilation error.

* Add get_v1_node_peers() implementation.

* Fix peer's lifetime states.

* Use the most recent multiaddress.

* Fix assign NewPeerScore again.
Fix compilation error with last seen address.
Fix Peer upgraded log line place.

* syncing, health, peer_count, peer_id and fixes for identity.

* Fix compilation problems.

* Move object declaration to callsigs.
Fix identity addresses fields.

* Finish node RPC calls.

* Avoid leak of lifetime future.

* Bump chronos.

* Fix json generator problem.
2020-11-26 08:09:59 +01:00
Zahary Karadjov 7fdafa65ee
New bootstrap nodes for Mainnet 2020-11-25 21:27:40 +02:00
zah 372c9b798c
Fix the corrupted database state on Pyrmont nodes; Add mainnet genesis (#2056)
* Handle some web3 timeouts better

* Add support for developer .env files

* Eth1 improvements; Mainnet genesis state

Notable changes:

* The deposits table have been removed from the database. The client
  will no longer process all deposits on start-up.

* The network metadata now includes a "state snapshot" of the deposit
  contract. This allows the client to skip syncing deposits made prior
  to the snapshot (i.e. genesis). Suitable metadata added for Pyrmont
  and Mainnet.

* The Eth1 monitor won't be started unless there are validators attached
  to the node.

* The genesis detection code is now optional and disabled by default

* Bugfix: The client should not produce blocks that will fail validation
  when it hasn't downloaded the latest deposits yet

* Bugfix: Work around the database corruption affecting Pyrmont nodes

* Remove metadata for Toledo and Medalla
2020-11-24 22:21:47 +01:00
Jacek Sieka f2722ed866
auto-bump nim-libp2p (#2064)
Co-authored-by: = <dryajov@users.noreply.github.com>
2020-11-24 12:38:18 +01:00
nbc-bump-bot[bot] 0856f0539a
auto-bump nim-libp2p (#2036)
Co-authored-by: = <dryajov@users.noreply.github.com>
2020-11-23 10:38:34 +01:00
Zahary Karadjov d232f16b40 Add more eth1-related logging 2020-11-20 17:13:51 +02:00
Zahary Karadjov e370e80aee
Add more Pyrmont bootstrap nodes 2020-11-18 14:40:45 +02:00
Eugene Kabanov 28369574b7
Bump `nim-chronos` to fix write() deadlocks. (#2037) 2020-11-18 11:39:15 +01:00
Zahary Karadjov ebfacf597c
Review and fix some usages of Future.cancel; Bump Chronos 2020-11-18 00:27:21 +02:00
Jacek Sieka 4479c0a9f1
fix some libp2p channel hangs (#2032)
* fix some libp2p channel hangs

* make sure no concurrent reads are issues by awaiting cancellations
* bumps with fixes

* merge bump
2020-11-17 16:07:05 +01:00
Jacek Sieka a6b188bfd4
misc fixes (#2027)
* log when database is loading (to avoid confusion)
* generate network keys later during startup
* fix quarantine not scheduling chain of parents for download and
increase size to one epoch
* log validator count, enr and peerid more clearly on startup
2020-11-16 20:15:43 +01:00
Jacek Sieka 5b011b65ae add rpc call to get chronos futures at runtime 2020-11-16 12:31:39 +02:00