Commit Graph

3132 Commits

Author SHA1 Message Date
Zahary Karadjov 2eacfc4685 Bump modules to take advantage of the new Json format flavors support
Since quite a lot of additional procs were now compiled as generics, this lead to compiler bugs that had
to be worked-around:

* The `Domain` type was renamed to `Eth2Domain` to avoid compilation errors
  due to conflicts with `nativesockets.Domain`.
  Similarly, `eth2_network.KeyPair` was renamed to `NetKeyPair`.

* A new more robust version of `hexToByteArray` was added to stew
2021-03-25 09:37:35 +02:00
Jacek Sieka 8b76ceed52
Fix minor exception effect issues (#2448)
Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.
2021-03-24 17:20:55 +01:00
Kim De Mey a4a2c1c0e1
Add discovery query with forkId and attnets filter (#2420) 2021-03-24 11:48:53 +01:00
tersec 36311bfc05
incorporate proposals into nextActionWait; switch some proc to func (#2438) 2021-03-24 10:05:04 +00:00
Zahary Karadjov a727fc8e52
Merge stable into unstable 2021-03-24 03:24:20 +02:00
Zahary Karadjov 8391d3f83e
v1.0.12 2021-03-24 03:23:14 +02:00
kdeme adcae8d416 Avoid dialing peers that will be connection limited by libp2p 2021-03-23 14:46:52 +02:00
tersec 3076f5c3b6
rm std/random from beacon_chain and rm attestation timing randomness (#2442)
* remove added attestation timing randomness

* remove os/random from rest of beacon_chain, primarily deposit_contract

* remove scaffolding

* randomize std/random seed in beacon node and validator client

* use CSPRNG to more securely seed std/random
2021-03-23 06:57:10 +00:00
Giovanni Petrantoni 9558946df4
Expose gossip direct peers (#2246)
* Expose gossip direct peers

* direct-peer fixes

* add explicit info about direct peer

* build fixes wip
2021-03-23 15:10:17 +09:00
Zahary Karadjov 5fd82ddaf5
Merge stable into unstable 2021-03-22 23:13:05 +02:00
Jacek Sieka d1357801e7
bumps 2021-03-22 20:53:27 +01:00
Zahary Karadjov 25695d1e4f
v1.0.11 2021-03-22 19:25:34 +02:00
tersec 85289269d1
increase block proposal speed with many validators (#2423)
* increase block proposal speed with many validators

* document CookedSig rationale
2021-03-22 19:02:27 +02:00
Sacha Saint-Leger 5362d966ad
Nimbus guide: Rocket Pool edits (#2446)
* update nimbus book readme

* Revert "update nimbus book readme"

This reverts commit d568f0f7a3.

* update joe's guide

* incorporate pineapple's feedback
2021-03-22 17:57:22 +01:00
Zahary Karadjov af0d302398
Use the latest bootstrap nodes for Prater 2021-03-22 18:37:32 +02:00
Ștefan Talpalaru b2c4441834
book: document "logrotate" (#2390)
* book: document "logrotate"

* Update docs/the_nimbus_book/src/log-rotate.md

Co-authored-by: Sacha Saint-Leger <sacha@status.im>

* Update docs/the_nimbus_book/src/log-rotate.md

Co-authored-by: Sacha Saint-Leger <sacha@status.im>

* Update docs/the_nimbus_book/src/log-rotate.md

Co-authored-by: Sacha Saint-Leger <sacha@status.im>

* Update docs/the_nimbus_book/src/log-rotate.md

Co-authored-by: Sacha Saint-Leger <sacha@status.im>

* Update docs/the_nimbus_book/src/log-rotate.md

Co-authored-by: Sacha Saint-Leger <sacha@status.im>

* rewording

* Update docs/the_nimbus_book/src/log-rotate.md

Co-authored-by: Sacha Saint-Leger <sacha@status.im>

Co-authored-by: Sacha Saint-Leger <sacha@status.im>
2021-03-22 16:40:26 +01:00
tersec dfd99ec943
Altair (HF1/v1.1.0) minimal and mainnet presets/constants (#2444)
* Altair mainnet & minimal presets

* std/math not used
2021-03-22 14:44:45 +00:00
Mentor Palokaj 895fe4baf1
Make it clear that the default port for ETH2 is 9000 udp/tcp (#2421)
* Add default port info

* Add default port info

* Edit CLI and docs to reflect 9000 port

* Apply suggestions from code review

Co-authored-by: Jacek Sieka <arnetheduck@gmail.com>
2021-03-22 08:22:56 +01:00
Jacek Sieka 3743df6434
introduce unittest2 (#2437)
This commit introduces `unittest2`, driven by the need to fix several
long-standing issues with `unittest`, in particular exception handling,
test isolation, timing and others.

Upgrading to `unittest2` generally requires updating the unittest import
and potentially fixing the code to support the new `one-proc-per-test`
setup.

* nim-unittest2 has been moved from Stefans repository to status-im -
https://github.com/status-im/nim-unittest2/pull/2 fixes some exception
handling issues and brings it in line with latest nim devel - a notable
difference to unittest is that each test is run inside a proc which
gives it a fresh stack and removes a lot of globals, meaning tests
should run under conditions more similar to "normal" code and interfere
with each other less - this is a crucial first step to improving the
testing experience in general, that stefan implemented for the
multithreading support
* the multithreading in unittest2 is optional, and disabled by default
* nim-testutils contains utilities for fuzzing, as well as other
"large:ish" features that have dependencies -
https://github.com/status-im/nim-testutils/pull/27 pulls in unittest2 so
unittest2 can be kept "low-dependency"
* nim-chronos needs the above unittest changes to test its own new,
stricter exception handling -
https://github.com/status-im/nim-chronos/pull/166 - the strictess
introduced is expected to increase with time
2021-03-22 08:21:22 +01:00
tersec baa43ae7f7
add stricter ABI-compatibility checks for punned types (#2434)
* add stricter ABI-compatibility checks for punned types

* enforce type isomorphism for each pair of cast types to eliminate potential safety hole
2021-03-19 12:30:20 +00:00
Jacek Sieka 01fe564e46
chronos-based metrics (#2432)
This opens up the road for removing `-d:insecure` for metrics.
2021-03-19 10:36:38 +01:00
tersec 97850741a0
remove unused imports in slashing prtection (#2436) 2021-03-19 08:26:02 +00:00
tersec 21b2367bf1
Remove Travis and Azure build statuses from README.md (#2433) 2021-03-18 21:50:59 +00:00
Dustin Brody a3a6aa68e8
check if Nim-caching version also works 2021-03-17 19:58:33 +01:00
Dustin Brody 732a437fa8
maybe fix Jenkins syntax 2021-03-17 19:45:27 +01:00
Dustin Brody 51079770b9
disable Nim compiler cache 2021-03-17 19:41:02 +01:00
tersec 9ff6118c28
clear nimcache on make update (#2426)
* test clearing nimcache

* don't be verbose
2021-03-17 17:29:34 +00:00
Sacha Saint-Leger 756c82671d
Update rp-quick-start.md 2021-03-17 17:12:32 +01:00
Sacha Saint-Leger b9f3fc4ceb
Rocket Pool guide (#2425)
* update nimbus book readme

* Revert "update nimbus book readme"

This reverts commit d568f0f7a3.

* Rocket Pool guide
2021-03-17 15:33:28 +01:00
tersec b059cb42c5
increase block proposal speed with many validators (#2423)
* increase block proposal speed with many validators

* document CookedSig rationale
2021-03-17 13:35:59 +00:00
Jacek Sieka 3cb31e66b4
set upper bound on EpochRef cache (#2403)
* set upper bound on EpochRef cache

* max 32 EpochRef instances
* less memory waste in BlockRef by removing EpochRef seq that is mostly
unused (~20mb)
* less memory waste in dag block lookup by not keeping an extra copy of
digest (~70mb)
* fix `==` and `$` for Eth2Digest
* remove `ChainDAG.tmpState` (~50mb?)

all in all, this branch cuts mainnet memory usage by ~160-180mb and puts
limits on EpochRef cache usage - where normally it hovered around 950mb
before, it's now sitting at 600-700mb on my machine.

* docs
2021-03-17 11:17:15 +01:00
Kim De Mey 5e3770e994
Remove unused test (#2424) 2021-03-17 06:36:48 +00:00
Mamy Ratsimbazafy 6e38d474cc
Add pruning timings (#2422) 2021-03-17 07:30:16 +01:00
Ștefan Talpalaru 683edbff7a
restore terminal echoing after pressing Ctrl+C at a password prompt (#2412) 2021-03-16 08:06:45 +00:00
Giovanni Petrantoni 26c56c2800
more debug rpcs (#2385)
* more debug rpcs

* cleanups

* oops fixing

* add wip change log notes about rpc renaming

* fix wrong breaking changes changelog location
2021-03-16 08:39:29 +01:00
Zahary Karadjov c519c6012e
Merge branch 'stable' into unstable 2021-03-15 21:52:12 +02:00
Zahary Karadjov f3fc551ea7
Support for the prater testnet 2021-03-15 21:51:24 +02:00
nbc-bump-bot[bot] 2201480bd9
auto-bump nim-libp2p (#2409)
Co-authored-by: = <sinkingsugar@users.noreply.github.com>
2021-03-15 17:39:10 +00:00
Jacek Sieka eb830c7968
Merge remote-tracking branch 'origin/stable' into unstable 2021-03-15 17:29:43 +01:00
Jacek Sieka f5180e0840
Revert "Nim-1.2.10" (#2411)
This reverts commit 12e15a0b8d.
2021-03-15 16:26:41 +01:00
tersec 8def2486b0
immutable validator database factoring (#2297)
* initial immutable validator database factoring

* remove changes from chain_dag: this abstraction properly belongs in beacon_chain_db

* add merging mutable/immutable validator portions; individually test database roundtripping of immutable validators and states-sans-immutable-validators

* update test summaries

* use stew/assign2 instead of Nim assignment

* add reading/writing of immutable validators in chaindag

* remove unused import

* replace chunked k/v store of immutable validators with per-row SQL table storage

* use List instead of HashList

* un-stub some ncli_db code so that it uses

* switch HashArray to array; move BeaconStateNoImmutableValidators from datatypes to beacon_chain_db

* begin only-mutable-part state storage

* uncomment some assigns

* work around https://github.com/nim-lang/Nim/issues/17253

* fix most of the issues/oversights; local sim runs again

* fix test suite by adding missing beaconstate field to copy function

* have ncli bench also store immutable validators

* extract some immutable-validator-specific code from the beacon chain db module

* add more rigorous database state roundtripping, with changing validator sets

* adjust ncli_db to use new schema

* simplify putState/getState by moving all immutable validator accounting into beacon state DB

* remove redundant test case and move code to immutable-beacon-chain module

* more efficient, but still brute-force, mutable+immutable validator merging

* reuse BeaconState in getState

* ensure HashList/HashArray caches are cleared when reusing getState buffers; add ncli_db and a unit test to verify this

* HashList.clear() -> HashList.clearCache()

* only copy incrementally necessary immutable validators

* increase strictness of test cases and fix/work around resulting HashList cache invalidation issues

* remove explanatory scaffolding

* allow for storage of full (with all validators) states for backwards/forwards-compatibility

* adjust DbSeq type usage

* store full, with-validators, state every 64 epochs to enable reverting versions

* reduce memory allocation and intermediate objects in state storage codepath

* eliminate allocation/copying through intermediate BeaconStateNoImmutableValidators objects

* skip benchmarking initial genesis-validator-heavy state store

* always store new-style state and sometimes old-style state

* document intent behind BeaconState/Validator type-punnery

* more accurate failure message on SQLite in-memory database initialization failure
2021-03-15 14:11:51 +00:00
Ștefan Talpalaru c5035c6eca dist: enable libbacktrace on ARM and ARM64
(and make sure it will work on Windows, when enabled)
2021-03-15 14:38:39 +02:00
Ștefan Talpalaru eafe694536 dist: faster builds for ARM and ARM64
We're moving from very slow QEMU-virtualised builds to very fast ones
based on cross toolchains.

- refactor jobs; add checksums to release note
- build and publish Docker images with ARM/ARM64 binaries
2021-03-15 14:38:39 +02:00
Ștefan Talpalaru 12e15a0b8d Nim-1.2.10 2021-03-15 14:29:48 +02:00
Jacek Sieka aabdd34704
e2store: add era format (#2382)
Era files contain 8192 blocks and a state corresponding to the length of
the array holding block roots in the state, meaning that each block is
verifiable using the pubkeys and block roots from the state. Of course,
one would need to know the root of the state as well, which is available
in the first block of the _next_ file - or known from outside.

This PR also adds an implementation to write e2s, e2i and era files, as
well as a python script to inspect them.

All in all, the format is very similar to what goes on in the network
requests meaning it can trivially serve as a backing format for serving
said requests.

Mainnet, up to the first 671k slots, take up 3.5gb - in each era file,
the BeaconState contributes about 9mb at current validator set sizes, up
from ~3mb in the early blocks, for a grand total of ~558mb for the 82 eras
tested - this overhead could potentially be calculated but one would lose
the ability to verify individual blocks (eras could still be verified using
historical roots).

```
-rw-rw-r--. 1 arnetheduck arnetheduck   16  5 mar 11.47 ethereum2-mainnet-00000000-00000001.e2i
-rw-rw-r--. 1 arnetheduck arnetheduck 1,8M  5 mar 11.47 ethereum2-mainnet-00000000-00000001.e2s
-rw-rw-r--. 1 arnetheduck arnetheduck  65K  5 mar 11.47 ethereum2-mainnet-00000001-00000001.e2i
-rw-rw-r--. 1 arnetheduck arnetheduck  18M  5 mar 11.47 ethereum2-mainnet-00000001-00000001.e2s
...
-rw-rw-r--. 1 arnetheduck arnetheduck  65K  5 mar 11.52 ethereum2-mainnet-00000051-00000001.e2i
-rw-rw-r--. 1 arnetheduck arnetheduck  68M  5 mar 11.52 ethereum2-mainnet-00000051-00000001.e2s
-rw-rw-r--. 1 arnetheduck arnetheduck  61K  5 mar 11.11 ethereum2-mainnet-00000052-00000001.e2i
-rw-rw-r--. 1 arnetheduck arnetheduck  62M  5 mar 11.11 ethereum2-mainnet-00000052-00000001.e2s
```
2021-03-15 11:31:39 +01:00
nbc-bump-bot[bot] 4fcff4cd27
auto-bump nim-libp2p (#2406)
Co-authored-by: = <sinkingsugar@users.noreply.github.com>
2021-03-15 07:36:36 +00:00
Joe Clapis 35d9291cfd
Validator APIs now return actual balance instead of effective balance (#2408) 2021-03-15 07:19:04 +00:00
tersec dfc3322fe2
last few v1.0.0 spec refs to v1.0.1 (#2404) 2021-03-13 20:51:39 +00:00
Dustin Brody 97504fdb9d ncli_db pruneDatabase checkpointing; remove onSlotEnd lookaheadTime 2021-03-12 23:15:46 +02:00
tersec 6533999c82
benchmark state loading in ncli_db (#2400) 2021-03-12 10:02:09 +00:00