Commit Graph

680 Commits

Author SHA1 Message Date
Zahary Karadjov 215caa21ae Eth1 monitor fixes
* Fix a resource leak introduced in https://github.com/status-im/nimbus-eth2/pull/3279

* Don't restart the Eth1 syncing proggress from scratch in case of
  monitor failures during Eth2 syncing.

* Switch to the primary operator as soon as it is back online.

* Log the web3 credentials in fewer places

Other changes:

The 'web3 test' command has been enhanced to obtain and print more
data regarding the selected provider.
2022-02-03 14:01:55 +02:00
tersec 8e6a920bf4
rename MERGE_FORK_EPOCH to BELLATRIX_FORK_EPOCH (#3350)
* rename MERGE_FORK_EPOCH to BELLATRIX_FORK_EPOCH

* fix REST test rules
2022-02-02 14:06:55 +01:00
Jacek Sieka d583e8e4ac
Store finalized block roots in database (3s startup) (#3320)
* Store finalized block roots in database (3s startup)

When the chain has finalized a checkpoint, the history from that point
onwards becomes linear - this is exploited in `.era` files to allow
constant-time by-slot lookups.

In the database, we can do the same by storing finalized block roots in
a simple sparse table indexed by slot, bringing the two representations
closer to each other in terms of conceptual layout and performance.

Doing so has a number of interesting effects:

* mainnet startup time is improved 3-5x (3s on my laptop)
* the _first_ startup might take slightly longer as the new index is
being built - ~10s on the same laptop
* we no longer rely on the beacon block summaries to load the full dag -
this is a lot faster because we no longer have to look up each block by
parent root
* a collateral benefit is that we no longer need to load the full
summaries table into memory - we get the RSS benefits of #3164 without
the CPU hit.

Other random stuff:

* simplify forky block generics
* fix withManyWrites multiple evaluation
* fix validator key cache not being updated properly in chaindag
read-only mode
* drop pre-altair summaries from `kvstore`
* recreate missing summaries from altair+ blocks as well (in case
database has lost some to an involuntary restart)
* print database startup timings in chaindag load log
* avoid allocating superfluos state at startup
* use a recursive sql query to load the summaries of the unfinalized
blocks
2022-01-30 18:51:04 +02:00
Emil 0051af430b Put `application/json` as a higher preference than `application/octet-stream` 2022-01-30 18:50:14 +02:00
tersec 60bf5b8bf4
use v1.1.9 test vectors (#3337) 2022-01-28 22:47:48 +00:00
Ștefan Talpalaru fdb76b8e0b
bump nimbus-build-system (#3324) 2022-01-26 17:11:28 +01:00
Ștefan Talpalaru 47a6f9950f
bump nimbus-build-system (#3322) 2022-01-26 14:15:53 +01:00
Mamy Ratsimbazafy 9e9ccf4a1f
Slashing prot interchange tests v5.2.1 (#3277)
* initial support for minification and new interchange tests. Removal of v1 and v1 migration.

* Synthetic attestations: SQLite3 requires one statement/query per prepared statement

* Fix DB import interrupted if no attestation was found

* Skip test relying on undocumented test behavior (https://github.com/eth-clients/slashing-protection-interchange-tests/pull/12#issuecomment-1011158701)

* Skip test relying on unclear minification behavior:
creating an invalid minified attestation with source > target or setting target = max(source, target)

* remove DB v1 and update submodule

* Apply suggestions from code review

Co-authored-by: Jacek Sieka <jacek@status.im>

Co-authored-by: Jacek Sieka <jacek@status.im>
2022-01-20 17:14:06 +01:00
Jacek Sieka 836f6984bb
move `state_transition` to `Result` (#3284)
* better error messages in api
* avoid `BlockData` copies when replaying blocks
2022-01-17 12:19:58 +01:00
Zahary Karadjov af614b6a85 bump nim-json-rpc 2022-01-16 18:26:21 +02:00
tersec 142e0ea163
bump nim-web3 to use enums for engine API types (#3282) 2022-01-14 14:11:42 +01:00
Jacek Sieka 7e1cdce8fc
stew: bump (#3271)
result fixes
2022-01-12 07:09:23 +00:00
Jacek Sieka aab302aa9c
bumps (#3268)
result + serialization cleanups
2022-01-11 09:35:20 +00:00
Jacek Sieka 6f7e0e3393
REST cleanups (#3255)
* REST cleanups

* reject out-of-range committee requests
* print all hex values as lower-case
* allow requesting state information by head state root
* turn `DomainType` into array (follow spec)
* `uint_to_bytesXX` -> `uint_to_bytes` (follow spec)
* fix wrong dependent root in `/eth/v1/validator/duties/proposer/`
* update documentation - `--subscribe-all-subnets` is no longer needed
when using the REST interface with validator clients
* more fixes
* common helpers for dependent block
* remove test rules obsoleted by more strict epoch tests
* fix trailing commas

* Update docs/the_nimbus_book/src/rest-api.md
* Update docs/the_nimbus_book/src/rest-api.md

Co-authored-by: sacha <sacha@status.im>
2022-01-08 22:06:34 +02:00
Tanguy 7a0119ac32
Bump libp2p (#3246) 2022-01-05 17:33:23 +01:00
tersec cd77377375
add Bellatrix fork and transition tests; "Ethereum Foundation" -> EF (#3242) 2022-01-05 09:42:56 +01:00
Zahary Karadjov 54d0d588b1 Implementation of the Keymanager API (BETA)
https://github.com/ethereum/keymanager-APIs
2022-01-04 18:51:45 +02:00
tersec 3c63a78c01
use v1.1.8 test vectors (#3236) 2022-01-03 17:43:00 +00:00
Jacek Sieka 61b6fc1016
3x speedup in snappy compression (#3234)
* 3x speedup in snappy compression

oh, the wonders of `copyMem` in `endians2` - speeds up all kinds of
operations like database stores, sending gossip etc.

* endian usage fixes
2022-01-03 18:17:10 +01:00
tersec 8be1699014
use v1.1.7 test vectors (#3231)
* use v1.1.7 test vectors
2022-01-03 13:06:14 +00:00
Mamy Ratsimbazafy 873b0f1128
Bump BLST, improve pubkey deserialization speed by 20% to 2x faster. improve signature deserialization by 5% to 4x faster. (#3227)
Nice!
2022-01-02 16:48:14 +01:00
Jacek Sieka 6b60a774e0
Lazy aggregated batch verification (#3212)
A novel optimisation for attestation and sync committee message
validation: when batching, we look for signatures of the same message
and aggregate these before batch-validating: this results in up to 60%
fewer signature verifications on a busy server, leading to a significant
reduction in CPU usage.

* increase batch size slightly which helps finding more aggregates
* add metrics for batch verification efficiency
* use simple `blsVerify` when there is only one signature to verify in
the batch, avoiding the RNG
2021-12-29 15:28:40 +01:00
Tanguy 4a72def1d5
Bump libp2p (#3207) 2021-12-17 12:39:24 +01:00
Ștefan Talpalaru 9311d0a887
Nim-1.2.16 (#3203) 2021-12-17 00:52:49 +01:00
Etan Kissling dc1562665f
bump nim-ssz-serialization to `3cd8d2d6b80bde0ce7f25609cb5cb9fc37852fe2` (#3194)
This updates `nim-ssz-serialization` to
`3cd8d2d6b80bde0ce7f25609cb5cb9fc37852fe2`.

Notable changes:
- Serialization of object variant (case object) to/from SSZ Union.
- int -> int64 fix in hashTreeRootCached
2021-12-16 17:26:54 +00:00
Etan Kissling f68aa9218f
bump nim-confutils to `6a56d01381f434d5fbcc61b6e497b9409155bcbc` (#3196)
This updates `nim-confutils` to
`6a56d01381f434d5fbcc61b6e497b9409155bcbc`.

Notable changes:
- feature: separator text when displaying help
- feature: multiple lines long description
- feature: add ignore property in addition to hidden
- add compile time check to detect duplicate abbr and duplicate name
2021-12-15 09:27:04 +00:00
Etan Kissling 6f75262a03
bump nim-snappy to `16cce7d07ce8be2cfc760d86744b8b54cd0808a9` (#3195)
This updates `nim-snappy` to `16cce7d07ce8be2cfc760d86744b8b54cd0808a9`.

Notable changes:
- avoid unnecessary compression of short payloads
2021-12-14 18:50:24 +00:00
Etan Kissling d023613045
bump nim-eth to `5655bd035cfd7319c6f2e60b7fdefef65a057939` (#3193)
This updates `nim-eth` to `5655bd035cfd7319c6f2e60b7fdefef65a057939`.

Notable changes:
- db: Allow Sqlite keystores to be used in read-only mode
- net: avoid allocation in hash(ValidIpAddress)
- net: Remove hashData usage on objects
- p2p: reject WHOAREYOU packets with non-empty message
- p2p: Adjust logging when node is not reachable but enrAutoUpdate is on
- p2p: Allow a node to self resolve
- p2p: Fix logDistance for BE arch and remove toBytes for NodeId
- p2p: Export discovery routing table and its buckets nodes
- ssz: remove outdated and incorrect SSZ code
- utp: Various updates and fixes
2021-12-14 17:27:46 +00:00
tersec f88bdc5d14
bump nim-web3 to properly decode TypedTransactions (#3187) 2021-12-13 15:50:18 +00:00
Tanguy 4be7e800f2
Bump chronos (Improve ram usage) (#3180) 2021-12-10 13:25:43 +01:00
Jacek Sieka 6f077ce82c
chronos: fast path for writes (and a bunch of other changes) (#3172) 2021-12-08 14:47:04 +01:00
tersec aeb90a1585
bump nim-web3 to adjust engine API field names for alpha.5 (#3166) 2021-12-07 14:30:16 +00:00
tersec 8d7df05f2e
Revert "Bump chronos and presto. (#3159)" (#3160)
This reverts commit 4c90b82d9f.
2021-12-04 15:52:28 +00:00
Eugene Kabanov 4c90b82d9f
Bump chronos and presto. (#3159)
* Add some indicators to help fixing issue.

* Bump presto to help debugging.

* Fix compilation problems in presto.

* Fix SIGSEGV.

* Bump latest changes in chronos and presto.
Fix rare cases in validator_client.

* Use proper commits for chronos and presto.
2021-12-04 14:26:16 +00:00
zah 74c63ed740
More efficient implementation of the 'POST beacon_committee_subscriptions' API (#3153) 2021-12-03 17:04:58 +02:00
nbc-bump-bot[bot] fa2cf028e2
auto-bump nim-libp2p (#3114)
Co-authored-by: = <Menduist@users.noreply.github.com>
2021-12-03 15:03:34 +01:00
tersec 61fb458f89
use v1.1.6 test vectors (#3146) 2021-12-01 12:55:42 +00:00
Ștefan Talpalaru 5e50297f95
bump nimbus-build-system (#3140) 2021-12-01 05:31:22 +01:00
Zahary Karadjov 872b88db50
Hotfix: non-crashing version of nim-json-rpc/news 2021-11-30 02:51:35 +02:00
zah bc6a6a8b8f
Switch back to the news package in nim-json-rpc (#3136) 2021-11-29 21:47:31 +01:00
Zahary Karadjov 88c623e250 Add support for HTTPS Web3 providers 2021-11-23 15:56:18 +02:00
Ștefan Talpalaru 2d986c5346
bump nimbus-build-system (#3117) 2021-11-23 02:43:04 +01:00
Ștefan Talpalaru 4639b04b2a
bump nimbus-build-system (#3116) 2021-11-22 21:21:13 +01:00
Tanguy 4bcdccab9a
Bump libp2p (#3112) 2021-11-18 10:57:27 +01:00
Eugene Kabanov eda42f4a89
Bump nim-chronos. (#3101) 2021-11-15 18:34:07 +00:00
tersec fde73b5a70
bump nim-web3 to switch payload ID from uint64 to 8-byte buffer (#3103) 2021-11-15 14:55:03 +00:00
Zahary Karadjov 5e62df5ad5 /bin/bash -> /usr/bin/env bash (needed on NixOS) 2021-11-15 15:45:01 +02:00
tersec 97ad5d49b0
kintsugi merge vector tests (#3072) 2021-11-10 13:41:02 +02:00
kdeme 03a70fbf36 Use nim-ssz-serialization module and rm local ssz code 2021-11-10 13:37:24 +02:00
Ștefan Talpalaru 782318d3fc
Nim-1.2.14 (#3074)
* Nim-1.2.14

* bump submodules
2021-11-10 11:39:43 +01:00