Commit Graph

49 Commits

Author SHA1 Message Date
Jacek Sieka 7165e0ac31
Reset cached indices when resetting cache on SSZ read (#2480)
* Reset cached indices when resetting cache on SSZ read

When deserializing into an existing structure, the cache should be
cleared - goes for json also. Also improve error messages.
2021-04-08 13:11:04 +03:00
Jacek Sieka 3cb31e66b4
set upper bound on EpochRef cache (#2403)
* set upper bound on EpochRef cache

* max 32 EpochRef instances
* less memory waste in BlockRef by removing EpochRef seq that is mostly
unused (~20mb)
* less memory waste in dag block lookup by not keeping an extra copy of
digest (~70mb)
* fix `==` and `$` for Eth2Digest
* remove `ChainDAG.tmpState` (~50mb?)

all in all, this branch cuts mainnet memory usage by ~160-180mb and puts
limits on EpochRef cache usage - where normally it hovered around 950mb
before, it's now sitting at 600-700mb on my machine.

* docs
2021-03-17 11:17:15 +01:00
tersec 90708a8287
Prefer converting int` to uint64 and switch foo.len.uint64 to .len64 (#1375)
* avoid converting from uint64 to int, and where most feasible, int type conversion at all

* .len.uint64 -> .len64

* fix 32-bit compilation

* try keeping state_sim loop variable/bounds as int for 32-bit Azure

* len64 -> lenu64
2020-07-26 20:55:48 +02:00
Jacek Sieka 6fe0a623f5
Crypto rng (#1284)
* use bearssl rng throughout

* bump

* bump

* move keygen out of crypto
2020-07-07 17:51:02 +02:00
Jacek Sieka 68b5638da4 allow non-power-of-2 limits in hashlist
fixes "make eth2_network_simulation"

a bit sad: no test coverage except against our own tests
2020-06-05 14:51:22 +03:00
Jacek Sieka 872d7ff493
support all basic types 2020-06-02 11:44:51 +02:00
Jacek Sieka 405f460f9b
int64 fixes 2020-06-01 17:35:42 +02:00
Jacek Sieka 4fc9a84c57
support array of uint64 2020-06-01 14:52:52 +02:00
Jacek Sieka f1dcee2749
wip: cache
* cache lists and arrays of complex objects (5x block processing speed
on ncli_db)

trivial baseline cache that stores tree in flat memory structure
2020-06-01 14:52:50 +02:00
Zahary Karadjov a8003e7e38 More complete integration of the List type; Detect more invalid inputs; Enable more tests 2020-05-24 19:00:34 +03:00
Zahary Karadjov accd5fe954 Don't use StackArray in ssz; Drop the support for strings 2020-05-24 19:00:34 +03:00
Jacek Sieka a38eddcaac
remove ssz stint support (#1046) 2020-05-20 19:05:22 +02:00
Zahary Karadjov a99977b772 Progress towards #991; Enable the distinct ssz.List type 2020-05-19 15:53:03 +03:00
Jacek Sieka afa08c8e3c
crypto: cleanup (#882)
* crypto: cleanup

* fix several Defect-on-user-input
* make crypto interface more similar to secp
* use `crypto.nim` in all of nbc

* digest: raises

* fix

* vendor
2020-04-11 10:51:07 +02:00
Zed 6ba7b4b117 Generate markdown test reports 2020-03-13 14:38:59 +00:00
Zahary Karadjov 8ab0248209
Revert "tighten a few imports"
This reverts commit b24a9200d9.
2019-12-10 02:18:47 +02:00
Jacek Sieka b24a9200d9 tighten a few imports 2019-12-09 12:58:28 +00:00
Dustin Brody 570de0839d measure/find slow tests (#624) 2019-12-05 11:27:00 +01:00
Dustin Brody 1494bcc262 replace links to Apache and MIT licenses with HTTPS versions (#592) 2019-11-25 16:30:02 +01:00
Dustin Brody f5b8931fad switch assert(...) to doAssert(...) and minor cleanup 2019-11-22 16:39:05 +00:00
Dustin Brody df1d00f13a remove UnusedImport spam (#549)
* remove UnusedImport spam

* re-add random import to beacon_node
2019-11-14 11:47:55 +01:00
Zahary Karadjov 22591deced Safer testnet restarts; Working CLI queries for inspecting the genesis states
When the connect_to_testnet script is invoked it will first verify that
the genesis file of the testnet hasn't changed. If it has changed, any
previously created database associated with the testnet will be erased.

To facilitate this, the genesis file of each network is written to the
data folder of the beacon node. The beacon node will refuse to start if
it detects a discrepancy between the data folder and any state snapshot
specified on the command-line.

Since the testnet sharing spec requires us to use SSZ snapshots, the Json
support is now phased out. To help with the transition and to preserve the
functionality of the multinet scripts, the beacon node now supports a CLI
query command that can extract any data from the genesis state. This is
based on new developments in the SSZ navigators.
2019-11-11 23:29:36 +00:00
Zahary Karadjov a644839b79 Dynamic SSZ navigator 2019-11-11 23:29:36 +00:00
Mamy Ratsimbazafy ca4f29caca v0.8.1 tests refactor (#326)
* Introduce new mocking proc to replace:
- makeFakeValidatorPrivKey
- hackPrivKey
- getNextBeaconProposerIndex
- addBlock
- makeBlock

* Add comments on datastructure unsynced with the spec

* Add merkle tree constructor and initial mocking for deposits (missing merkle proofs)

* [Mock] Implement sparse merkle tree and merkle proof builder

* [Mocking] Genesis deposits

* Add compact_committees_roots init + mock genesis state

* [Tests] Add first deposit test using the new mocking procedures

* [Tests -deposits] add at and over 32 ETH deposit tests

* [Tests - deposits] Add test for validator top-up

* [Tests -deposits] Mention the TODO to test for invalid conditions

* [Tests] Add stub to test "is_valid_genesis_state"

* [Merkle proofs] Implement round-trip checks

* Deactivate roundtrips test

* SSZ - use EF convention for hash_tree_root / hashTreeRoot

* [Tests - Attestation] Attestation mocking + initial test

* Add mocking + 3 new tests for valid attestations + mention future invalid attestation tests

* Add crosslinks test (1 failing to attestations in block being duplicated in state transition)

* Single attestation crosslink test - workaround https://github.com/status-im/nim-beacon-chain/issues/361

* Add test for failed crosslink penalty

* Rebase fixes + add refactored tests to test suite

* justif-finalization helpers first batch

* Add 234 finalization tests

* Fix justif test, Rule I 234 finalization does not happen with sufficient support.
(Also unittest check template does not fail properly in some cases)

* Add tests for all finalization rules

* Properly delete nim-byteutils following c91727e7e5 (diff-7c3613dba5171cb6027c67835dd3b9d4)

* use digest helper for deposit root
2019-08-28 12:07:00 +00:00
Zahary Karadjov f477eb6877
Add a `sszList` helper for writing tests involving SSZ lists 2019-08-20 16:34:36 +03:00
Zahary Karadjov 398ea55801
Implement the latest SSZ specification and integrate the official SSZ test suite 2019-08-05 03:18:48 +03:00
Zahary Karadjov 31baa77742 Restore building with chronicles_sinks=json 2019-06-24 19:38:19 +03:00
Mamy Ratsimbazafy e5047d93a4 [Tests] SHA256 + official shuffling vectors (#263)
* Change digests to SHA2-256 (from Keccak256)

* Fix sha256 digest initialization - pass shuffling test

* Add comments + remove old shuffling tests

* Remove stale tree hashing tests

* Small toOpenArray optim + notes on in-place hashing

* Revert "Revert "Update test repo""

This reverts commit f385467cd3.

* Revert "Revert "Fix SIGFPE on shuffling for 0""

This reverts commit 226d380f87.

* Revert "Revert "Implement shuffling test (pending typedesc and shuffling algo fix)""

This reverts commit 813cb6fbb8.

* withEth2hash templates now needs ctx.init()

* Use init-update-finish to avoid burnMem
2019-05-10 08:14:01 +00:00
Dustin Brody 46b4154ce8 begin 0.6.0 (#256)
* begin 0.6.0: new get_domain/increase_balance/reduce_balance, BeaconState.validator_balances -> BeaconState.balances, some renamed constants, transaction processing changes, SlashableAttestation field name changes, 0.6.0 get_beacon_proposer_index always uses given state's slot, update tests subrepo

* mark get_bitfield_bit/bls_verify_multiple/stat-list-lengths/is_active_validator/is_surround_vote/slot_to_epoch/int_to_bytes/etc as unchanged in 0.6.0; rm Eth1DataVote/maybe_reset_eth1_period and thus adjust expected tree hash test results

* mark verify_bitfield/bls_verify/deposit-contract/VoluntaryExit/PendingAttestation/Historicalbatch/Fork as 0.6.0; update DOMAIN_BEACON_BLOCK to DOMAIN_BEACON_PROPOSER

* update Crosslink to 0.6.0 (also requires tree hashing test result change, so isolate in individual commit)

* mark verify_merkle_branch/get_delayed_activation_exit_epoch/ProposerSlashing/Attestation/AttestationDataAndCustodyBit/hash/integer_squareroot/get_epoch_start_slot/is_double_vote/get_randao_mix/generate_seed as 0.6.0; update reward and penalty quotients; SlashableAttestation -> IndexedAttestation; rm get_fork_version; ATTESTATION_INCLUSION_REWARD_QUOTIENT -> PROPOSER_REWARD_QUOTIENT
2019-04-29 10:48:30 -06:00
Dustin Brody c53de3e550 Caching updates/refactoring & state_sim defaults updates (#235)
* rm now-superceded shuffling cache (shuffling is only called from get_crosslinks), which was badly architected due to trying to exist in state; rm one more vestige of previous light-client regime (one more to go, from datatypes)

* fix wrong shuffling list size (active validator size, not validator size) to make consistent with 0.5.1 (will be inconsistent with testnet0); fix typo and change defaults in state_sim

* doAssert a couple of constant relationships necessary to avoid underflow; rm non-spec, unused helper function get_new_recent_block_roots

* refactor separate crosslink_committee_cache and winning_root_participants_cache(s) into StateData object; remove last vestige of previous shuffling cache

* separate out caching parts of StateData to new StateCache object
2019-04-05 08:18:13 -06:00
Jacek Sieka 605dd0a0e9 Some speedups (#226)
* ssz: avoid memory allocations

* a bit fishy with the 32-item stack.. this should be a smallvector

* digest: avoid another burnmem

* avoid a few allocations by using iterator
2019-04-03 15:46:22 +00:00
Jacek Sieka 1b0e67c88c
ssz: update to 0.5.1:ish (#202)
* ssz: update to 0.5.1:ish
* slightly fewer seq allocations
* still a lot of potential for optimization
* fixes #174
* ssz: avoid reallocating leaves (logN merkle impl)
2019-03-25 10:46:31 -06:00
Dustin Brody 50a0948bc0 Mostly convert to using distinct Slot type (#172) 2019-03-12 16:21:32 -06:00
Zahary Karadjov 903cb8a8b5 A LibP2P-based networking stack;
To enable it, comment out the 'withLibp2p' line in nim.cfg

The history was squashed in order to remove an accidentally
commited binary file.

Other changes:

* SSZ was adapted to use the common serialization framework

* gossibsup.subscribe is not using async handlers at the moment
  and this allowed me to simplify it
2019-03-12 13:36:55 +02:00
Jacek Sieka f5a641178d
fix tests after rename 2019-02-06 21:11:14 +01:00
Yuriy Glukhov 7f1bddb267 Use nim-eth (#82) 2019-02-05 20:21:18 +01:00
Mamy Ratsimbazafy e5d152c6d6
Update BLS to the latest scheme. (#80)
* Attempt to switch to nim-blscurve.

* "Fix" BLS test for new tests

* Missing domain param in validator keygen
2019-02-05 17:13:29 +01:00
Dustin Brody ad3de5fde9 Uint24 -> ValidatorIndex; Fork.pre_fork_version -> Fork.previous_version; Fork.post_fork_version -> Fork.current_version 2019-01-28 20:15:00 -08:00
Jacek Sieka 523a990adc
ssz: switch to little-endian
https://github.com/ethereum/eth2.0-specs/pull/139
2019-01-22 11:56:01 -06:00
Dustin Brody f30b4f822e ValidatorRecord -> Validator 2019-01-18 12:47:30 +02:00
Jacek Sieka 142aa8ca8e ssz: finish implementation (#42)
* ssz: finish implementation

* add object support, simplify implementation
* fix extra round of hashing in tree_hash_root

* ssz: cleanups

* work around Nim range bug for Uint24, cleanups
2018-12-17 19:03:53 +01:00
Jacek Sieka 5b35b6c2dc
hash -> root
ethereum/eth2.0-specs#285
2018-12-11 15:53:18 -06:00
Jacek Sieka e8ec26debb
spec updates
* bunch of renames
* bunch of new constants
* more long variable names!
2018-12-05 20:07:04 -06:00
Jacek Sieka 6669e698b5
spec updates
* more work on block processing
* fix some unsigned issues
* fix ssz issue when last chunk is not full
* work around empty const seq compiler bug
* XXX -> TODO
2018-11-29 16:16:07 -06:00
Jacek Sieka b01c51b3e2
move spec stuff to spec directory
helps maintain sanity in the face of spec changes
2018-11-28 13:49:03 -06:00
Mamy Ratsimbazafy 9989f2fe56
"Stub" hashSSZ function for BeaconBlock (#18)
* FIx hash_ssz in the generic case

* Add a stub hash function for BeaconBlocks
2018-11-22 11:17:05 +01:00
Jacek Sieka 6df3ec952b
hash_ssz: simplify per latest spec update
* update per https://github.com/ethereum/eth2.0-specs/pull/140
* join with ssz to form a single ssz module (like spec)
2018-11-20 10:43:09 -06:00
Jacek Sieka 577598cca7 ssz: fix buffer overflows, explicit error reporting (#7)
* ssz: fix buffer overflows, explicit error reporting

* ssz: harden against unaligned data
2018-10-08 12:56:34 +02:00
mratsim 6ddab7e9c0 basic SimpleSerialize + tests 2018-09-20 17:45:02 +02:00