Commit Graph

1895 Commits

Author SHA1 Message Date
Mamy Ratsimbazafy 149ff49c8e
Remove correlated queries in finalization pruning, all use indexes (#2554) 2021-05-11 10:41:37 +02:00
Mamy Ratsimbazafy e6b559a35a
Slashing db pruning [Merge only after v2 has been default for 1 noticeable release] (#2452)
* Enable slashing DB pruning

* integrate slashing DB pruning with onSlotEnd

* rebase tests
2021-05-10 16:32:28 +02:00
Jacek Sieka 867d8f3223
Perform attestation check before broadcast (#2550)
Currently, we have a bit of a convoluted flow where when sending
attestations, we start broadcasting them over gossip then pass them to
the attestation validation to include them in the local attestation pool
- it should be the other way around: we should be checking attestations
_before_ gossipping them - this serves as an additional safety net to
ensure that we don't publish junk - this becomes more important when
publishing attestations from the API.

Also, the REST API was performing its own validation meaning
attestations coming from REST would be validated twice - finally, the
JSON RPC wasn't pre-validating and would happily broadcast invalid
attestations.

* Unified attestation production pipeline with the same flow for gossip,
locally and API-produced attestations: all are now validated and entered
into the pool, then broadcast/republished
* Refactor subnet handling with specific SubnetId alias, streamlining
where subnets are computed, avoiding the need to pass around the number
of active validators
* Move some of the subnet handling code to eth2_network
* Use BitArray throughout for subnet handling
2021-05-10 09:13:36 +02:00
Jacek Sieka 646923c3dd
add attestation stats tool to ncli_db (#2539)
This also makes future efforts to provide metrics and logs for
attestation efficiency easier

* Export rewards from epoch transition
* Use less memory for reward calculation (bool -> set[enum], field
alignment)
* Reuse reward memory when replaying, avoiding spike
* Allow replaying any range in ncli_db benchmark
2021-05-07 13:36:21 +02:00
Jacek Sieka 5cd5da74c4
Gossipsub unsubscribe fixes (fixes #2540) (#2545) 2021-05-07 06:31:28 +02:00
tersec 1d6c8ee9ab
store full state 4x less often (#2542) 2021-05-06 07:36:18 +02:00
tersec dd43a2c3b0
bump nim-eth2-scenarios to get merge SSZ test vectors (#2541) 2021-05-05 15:35:36 +00:00
Jacek Sieka 427c0f307c
avoid extraneous hash root calculation (#2537)
When applying a block, we'll currently compute a state root for the
state after slot processing but before block processing - this is
unnecessary when a block is being applied because the intermediate state
root is never observed.
2021-05-05 08:54:21 +02:00
Jacek Sieka 4d74c742da
move ENRForkID into `spec` (#2538)
* move ENRForkID into `spec`

also get rid of strformat in topic formation and fix some case
discrepancies

* also move `Eth2Metadata`
2021-05-04 17:28:48 +02:00
Jacek Sieka efdf759cc0
avoid some slashing protection queries (#2528)
This PR reduces the number of database queries for slashing protection
from 5 reads and 1 write to 2 reads and 1 write in the optimistic case.

In the process, it removes user-level support for writing the database
in the version 1 format in order to simplify the code flow, and prevent
code rot. In particular, the v1 format was not covered by any unit tests
and has no advantages over v2. The concrete code to read and write it
remains for now, in particular to support upgrades from v1 to v2.

The branch also removes the use of concepts which doesn't work with
checked exceptions - in particular, this highlights code that both
raises exceptions and returns error codes, which could be cleaned up in
the future.

* Cache internal validator ID
* Rely on unique index to check for trivial duplicate votes
* Combine two surround vote queries into one
* Combine API for checking and registering slashing into single function

The slashing DB is normally not a bottleneck, but may become one with
high attached validator counts.
2021-05-04 15:17:28 +02:00
tersec 290b889ce6
non-intrusive, novel portions of merge (#2535) 2021-05-04 11:54:19 +00:00
tersec e0f4d28116
rename initialize_beacon_state to initialize_beacon_state_from_eth1 (#2536) 2021-05-04 12:19:11 +02:00
Ștefan Talpalaru 51fde853d5
Merge branch 'stable' into unstable 2021-05-03 15:26:56 +02:00
Zahary Karadjov f369682b3f
v1.2.2 2021-05-03 11:13:12 +03:00
Chris Hobcroft a3b9990beb
Fixed confusing typo (#2533)
"One or more" appears to be a better description than "One of more"
2021-05-02 17:01:49 +02:00
Zahary Karadjov e60dd8e4f6 Stronger validation for third-party Eth1Data votes 2021-05-01 10:24:31 +03:00
Jacek Sieka ce49da6c0a
Introduce unittest2 and junit reports (#2522)
* Introduce unittest2 and junit reports

* fix XML path

* don't combine multiple CI runs

* fixup

* public combined report also

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-28 18:41:02 +02:00
Dustin Brody 7f42d38219 rename initialize_beacon_state{_from_eth1,}; suppress warnings when doppelganger detection disabled 2021-04-28 00:12:41 +03:00
Eugene Kabanov 0c3302b826
REST API test framework and tests. (#2492)
* REST API test framework and tests.

* Fix ValidatorIndex tests to properly handle int32, but not uint32 values.

* Fix tests to follow latest REST fixes.

* refactor restapi.sh

and add it to the test suite

* Fix issues.
Add delay timeout which is required.

* Fix restapi.sh script for Windows.

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-27 23:46:24 +03:00
Jacek Sieka 7dba1b37dd
remove attestation/aggregate queue (#2519)
With the introduction of batching and lazy attestation aggregation, it
no longer makes sense to enqueue attestations between the signature
check and adding them to the attestation pool - this only takes up
valuable CPU without any real benefit.

* add successfully validated attestations to attestion pool directly
* avoid copying participant list around for single-vote attestations,
pass single validator index instead
* release decompressed gossip memory earlier, specially during async
message validation
* use cooked signatures in a few more places to avoid reloads and errors
* remove some Defect-raising versions of signature-loading
* release decompressed data memory before validating message
2021-04-26 22:39:44 +02:00
Zahary Karadjov 8cd570fc52
Merge branch 'stable' into unstable 2021-04-20 18:17:54 +03:00
Zahary Karadjov 2faaf391a0
v1.2.1 2021-04-20 16:55:37 +03:00
Jacek Sieka 54d6884c89
fix sync issue when upgrading from 1.1.0-inited db
This patch writes a full genesis state to `kvstore` if one was missing,
which fixes 1.2.0 restarting sync when upgrading from 1.1.0, or when
downgrading to a pre-1.1.0 release.
2021-04-20 16:55:18 +03:00
Zahary Karadjov 9b57535f33
Merge branch 'stable' into unstable 2021-04-19 19:55:20 +03:00
Zahary Karadjov e1a8049ed5
v1.2.0 2021-04-19 19:54:24 +03:00
Ștefan Talpalaru 5b32f7ca17 Nim-1.2.12 2021-04-19 10:53:58 +03:00
Jacek Sieka daf98e4330 fix committee vs subnet confusion 2021-04-18 14:17:45 +03:00
tersec 99fccaee6e
more abstraction over BeaconState (#2509)
* more abstraction over BeaconState

* use HashedBeaconState copy of htr
2021-04-16 08:49:37 +00:00
Ștefan Talpalaru 85acf55ad8
remove duplicate metric (#2507) 2021-04-15 13:42:40 +02:00
Jacek Sieka f1f424cc2d attestation processing speedups
* avoid creating indexed attestation just to check signatures - above
all, don't create it when not checking signatures ;)
* avoid pointer op when adding attestation to pool
* better iterator for yielding attestations
* add metric / log for attestation packing time
2021-04-14 21:51:17 +03:00
Zahary Karadjov b0912b8eaf
Remove some unused parameters 2021-04-14 16:15:22 +03:00
tersec 050e3ac48b
abstract over more BeaconState usage (#2496) 2021-04-14 11:34:35 +02:00
Dmitriy Ryajov 3e3137ec14 pass max peers to builder 2021-04-14 10:15:29 +03:00
Jacek Sieka 4ed2e34a9e Revamp attestation pool
This is a revamp of the attestation pool that cleans up several aspects
of attestation processing as the network grows larger and block space
becomes more precious.

The aim is to better exploit the divide between attestation subnets and
aggregations by keeping the two kinds separate until it's time to either
produce a block or aggregate. This means we're no longer eagerly
combining single-vote attestations, but rather wait until the last
moment, and then try to add singles to all aggregates, including those
coming from the network.

Importantly, the branch improves on poor aggregate quality and poor
attestation packing in cases where block space is running out.

A basic greed scoring mechanism is used to select attestations for
blocks - attestations are added based on how much many new votes they
bring to the table.

* Collect single-vote attestations separately and store these until it's
time to make aggregates
* Create aggregates based on single-vote attestations
* Select _best_ aggregate rather than _first_ aggregate when on
aggregation duty
* Top up all aggregates with singles when it's time make the attestation
cut, thus improving the chances of grabbing the best aggregates out
there
* Improve aggregation test coverage
* Improve bitseq operations
* Simplify aggregate signature creation
* Make attestation cache temporary instead of storing it in attestation
pool - most of the time, blocks are not being produced, no need to keep
the data around
* Remove redundant aggregate storage that was used only for RPC
* Use tables to avoid some linear seeks when looking up attestation data
* Fix long cleanup on large slot jumps
* Avoid some pointers
* Speed up iterating all attestations for a slot (fixes #2490)
2021-04-13 20:24:02 +03:00
Dustin Brody 398c151b7d
revert change 2021-04-13 18:50:06 +02:00
Dustin Brody d6fa4d06bc
abstract over more BeaconState usage 2021-04-13 18:47:44 +02:00
cheatfate c2c3fdb471 Simplification and compiler error fix. 2021-04-13 17:07:41 +03:00
cheatfate fc5ca8ec35 Address #2487 and #2488. 2021-04-13 17:07:41 +03:00
cheatfate 79d4735355 Address #2491. 2021-04-13 17:07:41 +03:00
cheatfate b0a3008a1b Address #2486. 2021-04-13 17:07:41 +03:00
cheatfate 3417871370 Address #2489. 2021-04-13 17:07:41 +03:00
cheatfate 477decbcf5 Address #2490. 2021-04-13 17:07:41 +03:00
tersec 498c998552
abstract over most withStateVars/withState state var usage (#2484)
* abstract over most withStateVars/withState state var usage

* cleanups
2021-04-13 15:05:44 +02:00
Dmitriy Ryajov 1d6518eaaa use `new` instad of `init` 2021-04-12 20:28:39 +03:00
Dmitriy Ryajov 920de91c37 use libp2p builder 2021-04-12 20:28:39 +03:00
cheatfate 9cd946a192 Proper fix for ValidatorIndex supported values. 2021-04-09 21:42:13 +03:00
cheatfate af1a4d0c05 Simplify validator_id filter's logic. 2021-04-09 21:42:13 +03:00
cheatfate 5268d03b68 Finish error message to constants process. 2021-04-09 21:42:13 +03:00
cheatfate b3ad573b1f Make error messages to be constants. 2021-04-09 21:42:13 +03:00
cheatfate 934cc453d9 Add "BETA version" note to the `--rest` server command line parameters descriptions. 2021-04-09 21:42:13 +03:00