status-im/nimbus-eth2
2
0
Fork 0
mirror of https://github.com/status-im/nimbus-eth2.git synced 2025-01-10 22:36:01 +00:00
Code Issues Projects Releases Wiki Activity
2,398 Commits 110 Branches 116 Tags 244 MiB
Commit Graph

40 Commits

Author SHA1 Message Date
Jacek Sieka
dcf8a6b05d
improve slot processing speeds (#1670) 
about 40% better slot processing times (with LTO enabled) - these don't
do BLS but are used
heavily during replay (state transition = slot + block transition)

tests using a recent medalla state and advancing it 1000 slots:

```
./ncli slots --preState2:state-302271-3c1dbf19-c1f944bf.ssz --slot:1000
--postState2:xx.ssz
```
pre:

```

All time are ms
Average,       StdDev,          Min,          Max,      Samples,
Test
Validation is turned off meaning that no BLS operations are performed
39.236,        0.000,       39.236,       39.236,            1,
Load state from file
0.049,        0.002,        0.046,        0.063,          968,
Apply slot
256.504,       81.008,      213.471,      591.902,           32,
Apply epoch slot
28.597,        0.000,       28.597,       28.597,            1,
Save state to file
```

cast:
```
All time are ms
Average,       StdDev,          Min,          Max,      Samples,
Test
Validation is turned off meaning that no BLS operations are performed
37.079,        0.000,       37.079,       37.079,            1,
Load state from file
0.042,        0.002,        0.040,        0.090,          968,
Apply slot
215.552,       68.763,      180.155,      500.103,           32,
Apply epoch slot
25.106,        0.000,       25.106,       25.106,            1,
Save state to file
```

cast+rewards:
```
All time are ms
Average,       StdDev,          Min,          Max,      Samples,
Test
Validation is turned off meaning that no BLS operations are performed
40.049,        0.000,       40.049,       40.049,            1,
Load state from file
0.048,        0.001,        0.045,        0.060,          968,
Apply slot
164.981,       76.273,      142.099,      477.868,           32,
Apply epoch slot
28.498,        0.000,       28.498,       28.498,            1,
Save state to file
```

cast+rewards+shr
```
All time are ms
Average,       StdDev,          Min,          Max,      Samples,
Test
Validation is turned off meaning that no BLS operations are performed
12.898,        0.000,       12.898,       12.898,            1,
Load state from file
0.039,        0.002,        0.038,        0.054,          968,
Apply slot
139.971,       68.797,      120.088,      428.844,           32,
Apply epoch slot
24.761,        0.000,       24.761,       24.761,            1,
Save state to file

```
 2020-09-16 20:59:33 +00:00
tersec
aca1a318f2
cleanly close kvstore databases and bump nim-eth (#1630) 
* cleanly close kvstore databases

* close databases for all subcommands and during error conditions
 2020-09-12 05:35:58 +00:00
tersec
48893f1c2e
add ncli_db subcommand to prune database of unnecessary blocks and states (#1593) 
* add ncli_db subcommand to prune database of unnecessary blocks, states, and state roots

* tweak comments

* reduce default aggressiveness in pruning old states

* move copyPrunedDatabase() to ncli_db, as it's not generally useful as part of beacon_chain_db and doesn't use any internal interfaces
 2020-09-11 15:20:34 +02:00
Viktor Kirilov
67d73c4c60 added the --network=<x> option to the tools for which it matters 2020-09-01 12:02:22 +03:00
tersec
b8da265f89
add setting for benchmarking and profiling of sqlite block storage times (#1575) 2020-08-27 14:52:22 +02:00
Viktor Kirilov
0a96e5f564
renamed CandidateChains to ChainDagRef and made the Quarantine type a ref type so there is a single instance in the beacon node (#1407) 2020-07-31 14:49:06 +00:00
Viktor Kirilov
c032366547
removed the BlockPool type and all of the proxy functions around it (#1401) 
* removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type

* fixed the build after the rebase
 2020-07-30 21:18:17 +02:00
tersec
af012ed4dc
combine ncli_hash_tree_root and ncli_pretty into ncli_ssz (#1393) 
* combine ncli_hash_tree_root and ncli_pretty into ncli_ssz

* disable ncli_query building by default

* fold ncli_transition into ncli_ssz and rename to ncli
 2020-07-29 21:19:36 +00:00
Jacek Sieka
157ddd2ac4
Fork choice fixes 5 (#1381) 
* limit attestations kept in attestation pool

With fork choice updated, the attestation pool only needs to keep track
of attestations that will eventually end up in blocks - we can thus
limit the horizon of attestations that we keep more aggressively.

To get here, we expose getEpochRef which gets metadata about a
particular epochref, and make sure to populate it when a block is added
- this ensures that state rewinds during block addition are minimized.

In addition, we'll use the target root/epoch when validating
attestations - this helps minimize the number of different states that
we need to rewind to, in general.

* remove CandidateChains.justifiedState

unused

* remove BlockPools.Head object

* avoid quadratic quarantine loop

* fix
 2020-07-28 13:54:32 +00:00
Jacek Sieka
fb2f742972
Fork choice fixes 2 (#1356) 
* fork choice cleanup

* enable v2 pruning
* prefer `get_current_epoch`
* fix finalization check to use correct epoch

* small cleanups

* add `count_active_validators`
* remove misleading logs
* fix justified checkpoint slot calculation in rpc
 2020-07-22 23:01:44 +02:00
tersec
6b77f3dda5
update compute_subnet_for_attestation() to use https://github.com/ethereum/eth2.0-specs/pull/1876 signature, which isn't in v0.12.1, which works with lookahead (#1346) 2020-07-22 08:04:21 +00:00
Jacek Sieka
8b01284b0e
cache block hash (#1329) 
hash_tree_root was turning up when running beacon_node, turns out to be
repeated hash_tree_root invocations - this pr brings them back down to
normal.

this PR caches the root of a block in the SignedBeaconBlock object -
this has the potential downside that even invalid blocks will be hashed
(as part of deserialization) - later, one could imagine delaying this
until checks have passed

there's also some cleanup of the `cat=` logs which were applied randomly
and haphazardly, and to a large degree are duplicated by other
information in the log statements - in particular, topics fulfill the
same role
 2020-07-16 15:16:51 +02:00
Zahary Karadjov
c4af4e2f35
Working test suite with run-time presets 2020-07-08 02:02:14 +03:00
Jacek Sieka
1301600341
Trusted blocks (#1227) 
* cleanups

* fix ncli state root check flag
* add block dump to ncli_db
* limit ncli_db benchmark length
* tone down finalization logs

* introduce trusted blocks

We only store blocks whose signature we've verified in the database - as
such, there's no need to check it again, and most importantly, no need
to deserialize the signature when loading from database.

50x startup time improvement, 200x block load time improvement.

* fix rewinding when deposits have invalid signature
* speed up ancestor iteration by avoiding copy
* avoid deserializing signatures for trusted data
* load blocks lazily when rewinding (less memory used)

* chronicles workarounds

* document trustedbeaconblock
 2020-06-25 12:23:10 +02:00
tersec
807b920c19
state_transition implements the spec fairly directly (#1220) 2020-06-23 13:54:24 +00:00
Jacek Sieka
49e9167b28 clean up dump feature 
* don't write blocks that get added to database
* don't write states
* write to folders
* add state dumping feature to `ncli_db` to get any known state from the
database
 2020-06-16 13:44:37 +00:00
Jacek Sieka
42832cefa8
Small fixes (#1165) 
* random fixes

* create dump dir on startup
* don't crash on failure to write dump
* fix a few `uint64` instances being used when indexing arrays - this
should be a compile error but isn't due to compiler bugs
* fix standalone test_block_pool compilation
* add signed block processing in ncli

* reuse cache entry instead of allocating a new one

* allow for small clock disparities when validating blocks
 2020-06-12 18:43:20 +02:00
Jacek Sieka
ea8f96d284
ncli_db: allow saving states by root (#1136) 
also dump state+block when validation fails
 2020-06-06 13:26:19 +02:00
Dustin Brody
3cb7896bab 12x speedup on state sim with 100k validators sans BLS by caching get_beacon_proposer_index(...) 2020-06-04 17:07:51 +00:00
Jacek Sieka
56ffb696be
reorder ssz (#1099) 
* reorder ssz

* split into hash_trees and ssz_serialization, roughly, for hashing and
IO
* move bitseqs into ssz (from stew)
* clean up imports

* docs, imports
 2020-06-03 15:52:02 +02:00
Jacek Sieka
e692aa426f
ncli_db: add validation flag, better ux 2020-06-01 16:48:24 +02:00
Jacek Sieka
f1dcee2749
wip: cache 
* cache lists and arrays of complex objects (5x block processing speed
on ncli_db)

trivial baseline cache that stores tree in flat memory structure
 2020-06-01 14:52:50 +02:00
Jacek Sieka
061be037d1 ncli_db: database tool 
includes a benchmark tool for now
 2020-05-28 17:43:02 +00:00
Jacek Sieka
f53b55cbe0 SSZ cleanup 
* be stricter about SSZ length prefix
* compute zeroHash list at compile time
* remove SSZ schema stuff
* move SSZ navigation to ncli
* cleanup a few leftover openArray uses
 2020-05-28 19:15:16 +03:00
Dustin Brody
57519bebac remove some unused imports, add tests for pubsub topics, and subscribe to interop attestations 2020-05-14 16:41:46 +03:00
tersec
f0d02f5f29
return nonzero exit code on state transition failure (#996) 2020-05-08 16:16:44 +02:00
Jacek Sieka
c74ba5c0c6
ssz: move ref support outside (#943) 
* ssz: move ref support outside

Instead of allocating ref's inside SSZ, move it to separate helper:

* makes `ref` allocations explicit
* less magic inside SSZ
* `ref` in nim generally means reference whereas SSZ was loading as
value - if a type indeed used references it would get copies instead of
references to a single value on roundtrip which is unexpected

TODO: EF tests would benefit from some refactoring since they all do the
same thing practically..

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
 2020-04-29 22:12:07 +02:00
Jacek Sieka
28d6cd2524
avoid memory allocations and copies when loading states (#942) 
* rolls back some of the ref changes
* adds utility to calculate stack sizes
* works around bugs in nim exception handling and rvo
 2020-04-28 10:08:32 +02:00
Jacek Sieka
03a147ab8d
avoid state copy in state transition (#930) 
In BlockPool, we keep the head state around, so it's trivial to restore
the temporary state there and keep going as if nothing happened.

This solves 3 problems:
* stack space - the state copy on mainnet is huge
* GC scanning - using stack space for state slows down the GC
significantly
* reckless copying - the copy itself takes a long time

In state_sim, we'll do the same and allocate on heap - this helps a
little with GC - without it, the collection of the temporary strings
created with `toHex` while printing the json dominates the trace.
 2020-04-26 21:13:33 +02:00
Zahary Karadjov
4e9fa51ae9 Introduce BeaconNodeRef and use it in all the right places 2020-04-26 13:04:53 +03:00
Jacek Sieka
9a3db7a81f
[WIP] Fake bls at runtime (#735) 
* Initial implementation of runtime bls skipping.

Add libnfuzz skipBLSValidation handling, check that it propagates.

* Rename skipBLSValidation -> skipBlsValidation, start skipStateRootValidation
* Replace skipValidation flags with more granular flags.

Also added skipBlockParentRootValidation flag
Mainly replaced with skipBlsValidation but also StateRoot or
BlockParentRootValidation flags where appropriate.

* Adjust interop test to pass when skipping merkle validation.
* Stop skipping validation for mainchain_monitor.
* Remove comment.
* Also skipMerkleValidation for test_beacon_chain_db.
 2020-03-05 13:52:10 +01:00
Joao Gabriel Carvalho
91f87b55b4
SignedBeaconBlock in state_transition (#773) 
* using SignedBeaconBlock in state_transition
 2020-02-29 16:15:44 +01:00
Jacek Sieka
521b0ed6ba
Clean 20200205 (#729) 
* beacon node code cleanup
* rudimentary error checking on mainnet monitor
* start client even when sending deposit
* work around missing block number exception
* connect to testnet with web3 url
* pretty-print digests in json
 2020-02-07 08:13:38 +01:00
Dustin Brody
79893cd241 rm ncli_signing_root 2019-12-17 00:06:03 +02:00
Jacek Sieka
fd4de5de0f
start untangling beaconnode (#589) 
* Move BeaconNode type to its own file (fewer imports)
* disentangle sync protocol/request manager
* fix some old nimisms
* de-fear some logs
* simplify eth1 data production
* add stack tracing to release builds
* drop release compile flag for testnet
 2019-11-25 15:36:25 +01:00
Dustin Brody
6bb63f3563 root out remaining Transfer references in ncli 2019-11-08 13:23:08 +00:00
Dustin Brody
6d860e1829 fix ncli build after removing Transfer 2019-11-08 13:23:08 +00:00
Jacek Sieka
ee5d1c1467
minimize imports, specially for spec - cuts 2/3 of ncli build time (#500) 
* minimize imports, specially for spec - cuts 2/3 of ncli build time
* ptr_arith->ptrops
 2019-10-25 12:59:56 +02:00
Dustin Brody
9fa6e3db0c
minor tweak to avoid build error (#465) 2019-09-30 19:39:14 +00:00
Jacek Sieka
2b20bf9bed
ncli: move to separate folder 2019-09-28 18:23:02 +02:00