status-im
/
nimbus-eth2
mirror of https://github.com/status-im/nimbus-eth2.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
3,679 Commits 112 Branches 114 Tags 212 MiB
Commit Graph

113 Commits

Author SHA1 Message Date
zah 7c688bffda
Address #1586 (#1596) 2020-09-18 18:34:13 +02:00
Jacek Sieka dcf8a6b05d
improve slot processing speeds (#1670) 
about 40% better slot processing times (with LTO enabled) - these don't
do BLS but are used
heavily during replay (state transition = slot + block transition)

tests using a recent medalla state and advancing it 1000 slots:

```
./ncli slots --preState2:state-302271-3c1dbf19-c1f944bf.ssz --slot:1000
--postState2:xx.ssz
```
pre:

```

All time are ms
Average,       StdDev,          Min,          Max,      Samples,
Test
Validation is turned off meaning that no BLS operations are performed
39.236,        0.000,       39.236,       39.236,            1,
Load state from file
0.049,        0.002,        0.046,        0.063,          968,
Apply slot
256.504,       81.008,      213.471,      591.902,           32,
Apply epoch slot
28.597,        0.000,       28.597,       28.597,            1,
Save state to file
```

cast:
```
All time are ms
Average,       StdDev,          Min,          Max,      Samples,
Test
Validation is turned off meaning that no BLS operations are performed
37.079,        0.000,       37.079,       37.079,            1,
Load state from file
0.042,        0.002,        0.040,        0.090,          968,
Apply slot
215.552,       68.763,      180.155,      500.103,           32,
Apply epoch slot
25.106,        0.000,       25.106,       25.106,            1,
Save state to file
```

cast+rewards:
```
All time are ms
Average,       StdDev,          Min,          Max,      Samples,
Test
Validation is turned off meaning that no BLS operations are performed
40.049,        0.000,       40.049,       40.049,            1,
Load state from file
0.048,        0.001,        0.045,        0.060,          968,
Apply slot
164.981,       76.273,      142.099,      477.868,           32,
Apply epoch slot
28.498,        0.000,       28.498,       28.498,            1,
Save state to file
```

cast+rewards+shr
```
All time are ms
Average,       StdDev,          Min,          Max,      Samples,
Test
Validation is turned off meaning that no BLS operations are performed
12.898,        0.000,       12.898,       12.898,            1,
Load state from file
0.039,        0.002,        0.038,        0.054,          968,
Apply slot
139.971,       68.797,      120.088,      428.844,           32,
Apply epoch slot
24.761,        0.000,       24.761,       24.761,            1,
Save state to file

```
 2020-09-16 20:59:33 +00:00
Jacek Sieka aed57df957
avoid hash tree root calculation when loading blocks from database (#1572) 2020-09-04 08:35:10 +02:00
Jacek Sieka 2081c4f505
avoid unnecessary seq allocations (#1573) 
"sigh"
 2020-08-27 08:32:51 +02:00
Zahary Karadjov 21b5ad3259
make clearn-medalla will preserve your validator files 2020-08-21 21:50:46 +03:00
tersec b45de824a4
convert 22 v0.12.1 spec refs to v0.12.2 and remove 1 unnecessary spec ref (#1384) 2020-07-28 14:55:46 +00:00
Jacek Sieka 8b01284b0e
cache block hash (#1329) 
hash_tree_root was turning up when running beacon_node, turns out to be
repeated hash_tree_root invocations - this pr brings them back down to
normal.

this PR caches the root of a block in the SignedBeaconBlock object -
this has the potential downside that even invalid blocks will be hashed
(as part of deserialization) - later, one could imagine delaying this
until checks have passed

there's also some cleanup of the `cat=` logs which were applied randomly
and haphazardly, and to a large degree are duplicated by other
information in the log statements - in particular, topics fulfill the
same role
 2020-07-16 15:16:51 +02:00
zah 0be77f9cbc
Restore compilation with custom presets (#1309) 
This also assigns precise types to the constants in the minimal
and mainnet presets in order to reduce the chance of compilation
errors when custom presets are used (previously, only the custom
presets have precisely assigned types for the constants).
 2020-07-13 16:44:58 +02:00
Zahary Karadjov 93b04bc214 Add an option for graffiti customization 2020-07-12 21:01:31 +03:00
tersec c64737e7f2
implement aggregated attestation receiving/validating (#1272) 
* implement aggregated attestation receiving/validating

* document the conditions without explicit implementations in isValidAggregatedAttestation()
 2020-07-02 16:15:27 +00:00
Zahary Karadjov a2e28bef27 Further removal of enum support; Bump modules 2020-06-22 20:14:14 +03:00
Jacek Sieka 7e0e4dc327
don't crash on unknown disconnection reason, fix disconnection reason enum (#1208) 2020-06-20 09:24:33 +02:00
Zahary Karadjov 811ba9aacd Fix SSZ issues on 32-bit platforms 2020-06-11 17:40:08 +03:00
Zahary Karadjov 17343442ea Implement more of the KeyStore spec and integrate it in the beacon node 2020-06-11 17:40:08 +03:00
Jacek Sieka 68b5638da4 allow non-power-of-2 limits in hashlist 
fixes "make eth2_network_simulation"

a bit sad: no test coverage except against our own tests
 2020-06-05 14:51:22 +03:00
Jacek Sieka 56ffb696be
reorder ssz (#1099) 
* reorder ssz

* split into hash_trees and ssz_serialization, roughly, for hashing and
IO
* move bitseqs into ssz (from stew)
* clean up imports

* docs, imports
 2020-06-03 15:52:02 +02:00
Jacek Sieka 1d9a58f1e8
a few more cleanups 2020-06-02 22:10:45 +02:00
Jacek Sieka 7e881a4c09
cleanups 2020-06-02 17:17:07 +02:00
Jacek Sieka 872d7ff493
support all basic types 2020-06-02 11:44:51 +02:00
Jacek Sieka 405f460f9b
int64 fixes 2020-06-01 17:35:42 +02:00
Jacek Sieka 2316ac15c3
index type 2020-06-01 14:52:52 +02:00
Jacek Sieka fec01829c4
cache balances, more type fixes 2020-06-01 14:52:52 +02:00
Jacek Sieka be92842944
more type compiler bug workarounds 2020-06-01 14:52:52 +02:00
Jacek Sieka a0b8b818f3
work around type issues 2020-06-01 14:52:52 +02:00
Jacek Sieka 4fc9a84c57
support array of uint64 2020-06-01 14:52:52 +02:00
Jacek Sieka f1dcee2749
wip: cache 
* cache lists and arrays of complex objects (5x block processing speed
on ncli_db)

trivial baseline cache that stores tree in flat memory structure
 2020-06-01 14:52:50 +02:00
Zahary Karadjov 0bcdabfcdf Detect another invalid SSZ input found through fuzzing 
The first offset of an SSZ object should always have a fixed constant
value. Otherwise, some unused bytes may appear between the fixed portion
and the dynamic portion.

Please note that this fix shutds down the minimal forward compatibility
currently supported by the SSZ format (and thus, the expected behavior
must be clarified in the SSZ spec).
 2020-05-30 14:47:35 +03:00
Zahary Karadjov 1fd87a2255 Remove SszReader.maxObjectSize 2020-05-28 19:15:16 +03:00
Zahary Karadjov d4fa2c8913 Avoid an unnecessary copy in readSszValue 2020-05-28 19:15:16 +03:00
Jacek Sieka 693bc15919 avoid some RVO bugs 2020-05-28 19:15:16 +03:00
Jacek Sieka f53b55cbe0 SSZ cleanup 
* be stricter about SSZ length prefix
* compute zeroHash list at compile time
* remove SSZ schema stuff
* move SSZ navigation to ncli
* cleanup a few leftover openArray uses
 2020-05-28 19:15:16 +03:00
Jacek Sieka 0d54073ec9 properly work around nim type bug in ssz 
the cast worked around the bug at compile time by means of casting, but
introduced a runtime error, because the pre-cast type was still being
used during deserialization
 2020-05-26 14:12:42 +03:00
Zahary Karadjov a8003e7e38 More complete integration of the List type; Detect more invalid inputs; Enable more tests 2020-05-24 19:00:34 +03:00
Zahary Karadjov accd5fe954 Don't use StackArray in ssz; Drop the support for strings 2020-05-24 19:00:34 +03:00
Zahary Karadjov a99977b772 Progress towards #991; Enable the distinct ssz.List type 2020-05-19 15:53:03 +03:00
Zahary Karadjov a739d7e8d6 Adapt SSZ to the latest FastStreams API 2020-05-13 12:18:42 +03:00
Jacek Sieka fb2e0ddbec
sync fixes (#1005) 
* sync fixes

* fix Status message finalized info
* work around sync starting before initial status exchange
* don't fail block on deposit signature check failure (fixes #989)
* print ForkDigest and Version nicely
* dump incoming blocks
* fix crash when libp2p peer connection is closed
* update chunk size to 16 to work around missing blocks when syncing

* bump libp2p

* bump libp2p

* better deposit skip message
 2020-05-11 18:08:52 +00:00
Jacek Sieka c74ba5c0c6
ssz: move ref support outside (#943) 
* ssz: move ref support outside

Instead of allocating ref's inside SSZ, move it to separate helper:

* makes `ref` allocations explicit
* less magic inside SSZ
* `ref` in nim generally means reference whereas SSZ was loading as
value - if a type indeed used references it would get copies instead of
references to a single value on roundtrip which is unexpected

TODO: EF tests would benefit from some refactoring since they all do the
same thing practically..

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
 2020-04-29 22:12:07 +02:00
tersec be475a82d7
catch invalid 0-raw-byte Bitlist SSZ inputs during parsing (#936) 
* catch invalid 0-raw-byte Bitlist SSZ inputs during parsing

* bump a couple more spec refs to v0.11.1
 2020-04-26 16:26:53 +00:00
Zahary Karadjov fdcbfdff05 Pass the test suite with a BeaconState ref type 2020-04-26 13:04:53 +03:00
tersec 7bc18423fa
detect improper 0-length lists of variable-sized objects in SSZ reading (#928) 
* detect improper 0-length lists of variable-sized objects in SSZ reading
 2020-04-23 19:39:23 +00:00
Jacek Sieka ed74770451
spec: regulate exceptions (#913) 
* spec: regulate exceptions

* a few more simple raises
 2020-04-22 07:53:02 +02:00
Dustin Brody 4f970a067a fix variable offset bounding/comparison 2020-04-17 16:37:52 +03:00
Dustin Brody 71e802c740 harden SSZ reader 2020-04-17 16:37:52 +03:00
Zahary Karadjov 8d639d9bd7 Avoid accidental memory allocations in SSZ 2020-04-16 01:01:18 +03:00
Zahary Karadjov f34fd46e3a Add snappy and protobuf-serialization; Use the latest FastStreams API 2020-04-16 01:01:18 +03:00
Dustin Brody e7de8aae20 fix/restore 0.11.1 BeaconState genesis interop: (a) use GENESIS_FORK_VERSION properly in BeaconState init; (b) use GENESIS_FORK_VERSION to be compatible with zcli/zrnt initial eth1 deposits; (c) let List[foo] work with 64-bit ints, so BeaconState.validators SSZ-serializes properly; and (d) update test_interop to use new spec/crypto API 2020-04-07 13:16:55 +03:00
Jacek Sieka 7902d070cd prefer endians2 2020-03-05 18:13:28 +02:00
Zahary Karadjov 4fb654f2af Remove some SSZ compilation warnings 2020-02-18 12:53:49 +02:00
Zahary Karadjov 1f1a244f3f Implement more checks and wire up the invalid SSZ tests 2020-01-29 10:10:28 +02:00
Zahary Karadjov 182593ccb5 Tenative fix for the test suite 2020-01-29 10:10:28 +02:00
Zahary Karadjov 2a3c237bbb Handle malformed SSZ inputs properly 2020-01-29 10:10:28 +02:00
Jacek Sieka fd4de5de0f
start untangling beaconnode (#589) 
* Move BeaconNode type to its own file (fewer imports)
* disentangle sync protocol/request manager
* fix some old nimisms
* de-fear some logs
* simplify eth1 data production
* add stack tracing to release builds
* drop release compile flag for testnet
 2019-11-25 15:36:25 +01:00
Jacek Sieka 14712bbbdc
random warning fixes 2019-11-15 12:05:18 +01:00
Zahary Karadjov 22591deced Safer testnet restarts; Working CLI queries for inspecting the genesis states 
When the connect_to_testnet script is invoked it will first verify that
the genesis file of the testnet hasn't changed. If it has changed, any
previously created database associated with the testnet will be erased.

To facilitate this, the genesis file of each network is written to the
data folder of the beacon node. The beacon node will refuse to start if
it detects a discrepancy between the data folder and any state snapshot
specified on the command-line.

Since the testnet sharing spec requires us to use SSZ snapshots, the Json
support is now phased out. To help with the transition and to preserve the
functionality of the multinet scripts, the beacon node now supports a CLI
query command that can extract any data from the genesis state. This is
based on new developments in the SSZ navigators.
 2019-11-11 23:29:36 +00:00
Zahary Karadjov a644839b79 Dynamic SSZ navigator 2019-11-11 23:29:36 +00:00
Mamy André-Ratsimbazafy 5f6d60abce remove a lot of XDeclaredButNotUsed spam from SSZ 2019-11-07 15:22:04 +00:00
Zahary Karadjov a336285922 Fix a potential OOB error in SSZ deserialization 2019-09-27 23:29:58 +03:00
Zahary Karadjov 2bbfa8c877 Changes related to the new ETH interop spec 
* Hello is no longer a handshake message
  (all handshakes related code was deleted for clarity)

* Deal with the single-parameter inlining defined in the new spec
 2019-09-10 09:49:54 -04:00
zah 7ebf685fa3 Work-around for #373 (#385) 2019-09-05 10:27:26 +00:00
Mamy André-Ratsimbazafy ad55889707
Fix SSZ typo fieldCaseDisciminator (declared in nim-serialization) 2019-08-24 16:52:58 +02:00
Zahary Karadjov 5ce50b3aca
Implement the latest networking spec 
https://github.com/ethereum/eth2.0-specs/pull/1328
 2019-08-07 05:22:28 +03:00
Zahary Karadjov 398ea55801
Implement the latest SSZ specification and integrate the official SSZ test suite 2019-08-05 03:18:48 +03:00