Commit Graph

3102 Commits

Author SHA1 Message Date
Jacek Sieka 3cb31e66b4
set upper bound on EpochRef cache (#2403)
* set upper bound on EpochRef cache

* max 32 EpochRef instances
* less memory waste in BlockRef by removing EpochRef seq that is mostly
unused (~20mb)
* less memory waste in dag block lookup by not keeping an extra copy of
digest (~70mb)
* fix `==` and `$` for Eth2Digest
* remove `ChainDAG.tmpState` (~50mb?)

all in all, this branch cuts mainnet memory usage by ~160-180mb and puts
limits on EpochRef cache usage - where normally it hovered around 950mb
before, it's now sitting at 600-700mb on my machine.

* docs
2021-03-17 11:17:15 +01:00
Kim De Mey 5e3770e994
Remove unused test (#2424) 2021-03-17 06:36:48 +00:00
Mamy Ratsimbazafy 6e38d474cc
Add pruning timings (#2422) 2021-03-17 07:30:16 +01:00
Ștefan Talpalaru 683edbff7a
restore terminal echoing after pressing Ctrl+C at a password prompt (#2412) 2021-03-16 08:06:45 +00:00
Giovanni Petrantoni 26c56c2800
more debug rpcs (#2385)
* more debug rpcs

* cleanups

* oops fixing

* add wip change log notes about rpc renaming

* fix wrong breaking changes changelog location
2021-03-16 08:39:29 +01:00
Zahary Karadjov c519c6012e
Merge branch 'stable' into unstable 2021-03-15 21:52:12 +02:00
Zahary Karadjov f3fc551ea7
Support for the prater testnet 2021-03-15 21:51:24 +02:00
nbc-bump-bot[bot] 2201480bd9
auto-bump nim-libp2p (#2409)
Co-authored-by: = <sinkingsugar@users.noreply.github.com>
2021-03-15 17:39:10 +00:00
Jacek Sieka eb830c7968
Merge remote-tracking branch 'origin/stable' into unstable 2021-03-15 17:29:43 +01:00
Jacek Sieka f5180e0840
Revert "Nim-1.2.10" (#2411)
This reverts commit 12e15a0b8d.
2021-03-15 16:26:41 +01:00
tersec 8def2486b0
immutable validator database factoring (#2297)
* initial immutable validator database factoring

* remove changes from chain_dag: this abstraction properly belongs in beacon_chain_db

* add merging mutable/immutable validator portions; individually test database roundtripping of immutable validators and states-sans-immutable-validators

* update test summaries

* use stew/assign2 instead of Nim assignment

* add reading/writing of immutable validators in chaindag

* remove unused import

* replace chunked k/v store of immutable validators with per-row SQL table storage

* use List instead of HashList

* un-stub some ncli_db code so that it uses

* switch HashArray to array; move BeaconStateNoImmutableValidators from datatypes to beacon_chain_db

* begin only-mutable-part state storage

* uncomment some assigns

* work around https://github.com/nim-lang/Nim/issues/17253

* fix most of the issues/oversights; local sim runs again

* fix test suite by adding missing beaconstate field to copy function

* have ncli bench also store immutable validators

* extract some immutable-validator-specific code from the beacon chain db module

* add more rigorous database state roundtripping, with changing validator sets

* adjust ncli_db to use new schema

* simplify putState/getState by moving all immutable validator accounting into beacon state DB

* remove redundant test case and move code to immutable-beacon-chain module

* more efficient, but still brute-force, mutable+immutable validator merging

* reuse BeaconState in getState

* ensure HashList/HashArray caches are cleared when reusing getState buffers; add ncli_db and a unit test to verify this

* HashList.clear() -> HashList.clearCache()

* only copy incrementally necessary immutable validators

* increase strictness of test cases and fix/work around resulting HashList cache invalidation issues

* remove explanatory scaffolding

* allow for storage of full (with all validators) states for backwards/forwards-compatibility

* adjust DbSeq type usage

* store full, with-validators, state every 64 epochs to enable reverting versions

* reduce memory allocation and intermediate objects in state storage codepath

* eliminate allocation/copying through intermediate BeaconStateNoImmutableValidators objects

* skip benchmarking initial genesis-validator-heavy state store

* always store new-style state and sometimes old-style state

* document intent behind BeaconState/Validator type-punnery

* more accurate failure message on SQLite in-memory database initialization failure
2021-03-15 14:11:51 +00:00
Ștefan Talpalaru c5035c6eca dist: enable libbacktrace on ARM and ARM64
(and make sure it will work on Windows, when enabled)
2021-03-15 14:38:39 +02:00
Ștefan Talpalaru eafe694536 dist: faster builds for ARM and ARM64
We're moving from very slow QEMU-virtualised builds to very fast ones
based on cross toolchains.

- refactor jobs; add checksums to release note
- build and publish Docker images with ARM/ARM64 binaries
2021-03-15 14:38:39 +02:00
Ștefan Talpalaru 12e15a0b8d Nim-1.2.10 2021-03-15 14:29:48 +02:00
Jacek Sieka aabdd34704
e2store: add era format (#2382)
Era files contain 8192 blocks and a state corresponding to the length of
the array holding block roots in the state, meaning that each block is
verifiable using the pubkeys and block roots from the state. Of course,
one would need to know the root of the state as well, which is available
in the first block of the _next_ file - or known from outside.

This PR also adds an implementation to write e2s, e2i and era files, as
well as a python script to inspect them.

All in all, the format is very similar to what goes on in the network
requests meaning it can trivially serve as a backing format for serving
said requests.

Mainnet, up to the first 671k slots, take up 3.5gb - in each era file,
the BeaconState contributes about 9mb at current validator set sizes, up
from ~3mb in the early blocks, for a grand total of ~558mb for the 82 eras
tested - this overhead could potentially be calculated but one would lose
the ability to verify individual blocks (eras could still be verified using
historical roots).

```
-rw-rw-r--. 1 arnetheduck arnetheduck   16  5 mar 11.47 ethereum2-mainnet-00000000-00000001.e2i
-rw-rw-r--. 1 arnetheduck arnetheduck 1,8M  5 mar 11.47 ethereum2-mainnet-00000000-00000001.e2s
-rw-rw-r--. 1 arnetheduck arnetheduck  65K  5 mar 11.47 ethereum2-mainnet-00000001-00000001.e2i
-rw-rw-r--. 1 arnetheduck arnetheduck  18M  5 mar 11.47 ethereum2-mainnet-00000001-00000001.e2s
...
-rw-rw-r--. 1 arnetheduck arnetheduck  65K  5 mar 11.52 ethereum2-mainnet-00000051-00000001.e2i
-rw-rw-r--. 1 arnetheduck arnetheduck  68M  5 mar 11.52 ethereum2-mainnet-00000051-00000001.e2s
-rw-rw-r--. 1 arnetheduck arnetheduck  61K  5 mar 11.11 ethereum2-mainnet-00000052-00000001.e2i
-rw-rw-r--. 1 arnetheduck arnetheduck  62M  5 mar 11.11 ethereum2-mainnet-00000052-00000001.e2s
```
2021-03-15 11:31:39 +01:00
nbc-bump-bot[bot] 4fcff4cd27
auto-bump nim-libp2p (#2406)
Co-authored-by: = <sinkingsugar@users.noreply.github.com>
2021-03-15 07:36:36 +00:00
Joe Clapis 35d9291cfd
Validator APIs now return actual balance instead of effective balance (#2408) 2021-03-15 07:19:04 +00:00
tersec dfc3322fe2
last few v1.0.0 spec refs to v1.0.1 (#2404) 2021-03-13 20:51:39 +00:00
Dustin Brody 97504fdb9d ncli_db pruneDatabase checkpointing; remove onSlotEnd lookaheadTime 2021-03-12 23:15:46 +02:00
tersec 6533999c82
benchmark state loading in ncli_db (#2400) 2021-03-12 10:02:09 +00:00
tersec 5ebf36f54d
add metric for nextActionWait (#2399)
* add metric for nextActionWait

* use toFloatSeconds
2021-03-12 09:46:26 +00:00
Mamy Ratsimbazafy c47d636cb3
Split Eth2Processor in prep for batching (#2396)
* Split Eth2Processor in gossip and consensus part and materialize the shared block queue

* Update initialization in test_sync_manager
2021-03-11 11:10:57 +01:00
tersec ef4a5b0cc3
remove delta-encoding from state diff balances (#2397)
* remove delta-encoding from state diff balances

* switch HashList to List
2021-03-11 05:39:04 +00:00
Zahary Karadjov 77ee210762
v1.0.10 2021-03-10 23:33:37 +02:00
tersec f0eb45af44
avoid int64 -> uint64 -> int64 conversions in DbSeq (#2398) 2021-03-10 18:01:43 +00:00
Mamy Ratsimbazafy f7cddcc8ab
Fix #2393 (#2395)
* Fix #2393

* check both

* Fix shortLog(int64)
2021-03-10 16:53:42 +01:00
zah 33c7f264bc
Bump news to fix resource leaks (#2388) 2021-03-10 12:22:41 +00:00
tersec 236a9cdd1d
ensure HashList/HashArray maxChunks power of 2 in int64 (#2394) 2021-03-10 09:13:32 +00:00
Jacek Sieka 74a75d05a0
update doc links 2021-03-10 09:47:42 +01:00
tersec 82c300186b
annotate slashing protection v2 with uint64 -> int64 overflow conditions (#2392)
* annotate slashing protection v2 with uint64 -> int64 overflow conditions

* fix variables

* remove assertion which gets tripped by interchange tests
2021-03-10 08:35:04 +00:00
Mamy Ratsimbazafy 8e28a05cea
Move pruning out of latency critical path (#2384)
* Deferred DAG and fork choice pruning

* fixup

* Address https://github.com/status-im/nimbus-eth2/pull/2384/files#r589448448, rely only on onSLotEnd for state pruning

* no need to store needPruning in the data structure

* lastPrunePoint is updated in pruning proc

* Split eager and LazyPruning

* enforce pruning in updateHead
2021-03-09 15:36:17 +01:00
nbc-bump-bot[bot] a88d17dc04
auto-bump nim-libp2p (#2389)
Co-authored-by: = <arnetheduck@users.noreply.github.com>
2021-03-09 14:28:28 +01:00
Zahary Karadjov a5713c5d84
v1.0.9 2021-03-09 01:14:50 +02:00
Zahary Karadjov 196e3ea6b6
Switch to slashing DB v2 by default 2021-03-09 00:47:29 +02:00
Mamy André-Ratsimbazafy d425aabce2 BLST v0.3.3 2021-03-08 14:43:48 +02:00
Mamy Ratsimbazafy de1060e7f3
centralize p2p validation in a single file and address https://github.com/status-im/nimbus-eth2/pull/2377#issuecomment-791313118 (#2383) 2021-03-06 08:32:55 +01:00
kdeme 5f750f84b4 Use setupAddress for better IP and ports configuration 2021-03-06 00:09:09 +02:00
Mamy Ratsimbazafy d47f53cd9d
Reorg (5/5) (#2377)
* Reorg things left into networking and gossip_processing

* time -> beacon_clock

* fix builds
2021-03-05 14:12:00 +01:00
Afr Schoe c6f94ce165
docs: add link to keep-updated in build, fix #2379 (#2380) 2021-03-05 09:53:55 +01:00
kdeme 769225b591 Bump nimbus-build-system to use nim-1-2-6-RC1-hotfixed
This should practically solve the segfaults we have been seeing in
issue #2134
2021-03-04 23:00:44 +02:00
nbc-bump-bot[bot] 9408a9af1c
auto-bump nim-libp2p (#2373)
Co-authored-by: = <arnetheduck@users.noreply.github.com>
2021-03-04 10:14:00 +01:00
Mamy Ratsimbazafy 5d7f9c3a04
Consensus object pools [reorg 4/5] (#2374)
* Add documentation

* make test doesn't try to build the beacon node :/
2021-03-04 10:13:44 +01:00
tersec 4278e80657
document two uint64 -> int64 conversions (#2375)
* document two uint64 -> int64 conversions

* fix minimal preset slot time & calculation
2021-03-04 10:13:23 +01:00
Jacek Sieka c4edff504b
remove obsolete todo
issue has been fixed
2021-03-03 09:28:26 +01:00
Mamy Ratsimbazafy 2f17ac7b64
Move SSZ, deposit_contracts & eth1_monitor [reorg files 3/5] (#2371)
* move deposit_contract

* Move SSZ

* fix ssz import in tests

* move also eth1_monitor

* forgot to delete the original

* fix comma [skip ci]

* Fix "make" & tools imports

* Fix import

* Fix import again

* rename deposit_contract -> eth1

* Revert ssz move to subfolder

* path fixes [skip ci]
2021-03-03 07:23:05 +01:00
tersec 451cc03d76
datatypes spec ref url updates (#2372) 2021-03-02 17:31:34 +01:00
Mamy Ratsimbazafy 19775c1b22
Move tools (#2370)
* Delete tracing and styled stacktraces

* Move NBC tooling to ncli folder

* fix inspector import [skip ci]

* fix comma
2021-03-02 15:02:10 +01:00
tersec 2b5a3a6810
remove more int64 usage (#2369)
* remove more int64 usage

* explain loop bounds
2021-03-02 13:40:28 +00:00
Mamy Ratsimbazafy 3276dfc683
Consolidate modules by areas [part 1] (#2365)
* Move sync in subfolder

* move validator related thingies in validators

* fix binary builds

* update bounds comment [skip ci]
2021-03-02 11:27:45 +01:00
tersec de643d9926
allow multiple hard fork datatypes to coexist (#2328)
* allow multiple hard fork datatypes to coexist

* update to 1.0.1

* merge recent datatypes.nim updates

* trigger rebuild now the out-of-disk-space machine offline
2021-03-02 10:13:39 +00:00