332e248d91
rpc: setLogLevel ( #1991 )
...
* rpc: setLogLevel
also change default rpc port to match makefile and manual
* set topic-level formats too
and make feature less secret
2020-11-12 11:46:02 +01:00
43c6c5a0b3
Gossip message id should decompress data ( #1998 )
...
* also fix missing return on empty message
2020-11-12 11:45:28 +01:00
5644cfae8f
increase max db object size ( fixes #1996 )
2020-11-12 08:42:45 +01:00
c5710d97fc
default to gossipsub 1.1 ( #1992 )
2020-11-11 13:30:18 +01:00
389c11743a
Review TODO items and self-assign the most important ones
2020-11-10 20:41:04 +02:00
6e40ca5e15
Use separate cache for getting epochref
...
...else tmpState gets overwritten when producing blocks and requesting
epochRef for finalized state.
2020-11-10 17:40:32 +02:00
ea4d94c65c
Add guard task to monitor synchronization worker Future. ( #1972 )
...
* Add guard task to monitor synchronization worker Future.
* Change log events from debug to warning level.
2020-11-10 14:47:26 +01:00
7eaae89263
keep AllTests-mainnet.md consistent; ensure don't send overly large messages aren't sent ( #1973 )
2020-11-09 18:11:09 +00:00
a010ef2061
update tests from v1.0.0-rc.0 to v1.0.0 ( #1976 )
2020-11-09 17:44:18 +00:00
bd1047b715
0.6.0 release fixes
...
* Updated README
* No double v in nimbus_beacon_node --version
* No md5 checksums
2020-11-09 17:09:49 +02:00
271df8b604
bump 1.0.0rc-0 spec refs to 1.0.0 ( #1974 )
2020-11-09 14:18:55 +00:00
61a3209c24
remove version quotes
2020-11-09 13:37:58 +01:00
94e9ce3c1f
Add metadata for the toledo network
2020-11-09 11:38:52 +02:00
4b603d7f5a
Change the default log level to INFO
2020-11-09 11:38:52 +02:00
e9b9cd75ee
Rename binaries; Mimic the original repo layout in the distribution
2020-11-09 11:38:52 +02:00
bf58915500
release fixes ( #1966 )
2020-11-07 08:46:53 +01:00
d2476b643a
bump version ( #1964 )
...
* bump version
* update version format to "standard" style
* remove zinken/attacknet (both networks deprecated)
* add mainnet warning
2020-11-07 08:45:55 +01:00
fb6ce5c804
Enable discv5.1 for general beacon_node and inspector builds
2020-11-07 01:27:02 +02:00
ac1072b4bb
Add ./run-medalla-beacon-node script
2020-11-07 01:25:02 +02:00
4a710252d8
Check the web3 provider for network compatibility
2020-11-07 01:25:02 +02:00
24f8b7d483
[skip ci] Clean up some accidentally commited code
2020-11-06 11:53:44 +02:00
a8a66fe3f6
Fix #1784 ; Handle Infura HTTPS URLs
2020-11-06 11:15:32 +02:00
f596a24c4c
Mainnet deposit contract monitoring
2020-11-06 11:15:32 +02:00
95f5f76180
Datatype cleanup ( #1953 )
...
* clear up spec todo
* test fix
* remove unnecessary toSszType
* type
* one more
2020-11-04 21:52:47 +00:00
fc7885b27e
Store block summary in database
...
This introcudes a cache for block summaries, useful for instantiating
the block dag on startup, bringing medalla startup times down from
minutes to seconds.
This is something of a temporary band-aid that would be obsoleted by a
finalized block store.
2020-11-04 11:28:55 +02:00
7823c25ba8
Hotfix for a failing assert during block production
2020-11-04 11:06:58 +02:00
36b54fe091
Update eth2_network.nim
...
More informative and friendlier log message (from the user's perspective). Also helps avoid the potential for confusion with the validator key.
2020-11-03 23:23:32 +02:00
6ee80651c7
Rename mainchain_monitor to eth1_monitor
2020-11-03 23:23:10 +02:00
2f29e3e7f3
Further reduce the number of RPC requests; Significantly faster ETH1 syncing
2020-11-03 23:23:10 +02:00
14b2d4324d
openarray -> openArray
2020-11-03 23:23:10 +02:00
18639c3eff
Don't require requests that might fail on non-archive Geth nodes
2020-11-03 23:23:10 +02:00
79f1b6117e
Remove dead code from the beacon node binary; Fix 'make test'
2020-11-03 23:23:10 +02:00
07206ca264
An efficient algorithm for incremental generation of multiple merkle proofs
2020-11-03 23:23:10 +02:00
d913273dcc
Fix #1823
2020-11-03 23:23:10 +02:00
937decb0e9
Revert "Gossip11 post audit ( #1917 )" ( #1947 )
...
This reverts commit 63173ab2c1
2020-11-03 11:45:40 +01:00
9255945fb0
Update mainchain_monitor.nim ( #1945 )
...
Make it clear that this failure concerns the eth1 chain
2020-11-02 21:16:24 +01:00
002a3dfa54
graceful shutdown: move the hook registration ( #1940 )
...
...closer to where it starts having an effect.
2020-11-02 19:02:27 +01:00
2f049dc6e7
Fix init condition for fork choice ( #1941 )
...
... or we choose the wrong head on startup, sometimes.
2020-11-02 18:51:08 +01:00
ca63d48b82
tone down validator exit logging ( #1939 )
...
Validators exiting is normal, no need to scream about it
* avoid reallocating seq on big exit queue
* avoid fetching state cache when updating head (it's rarely needed)
* remove incorrectly implemented live validator counts (avoids memory
allocs)
2020-11-02 18:34:23 +01:00
7294d98ed1
Update validator_duties.nim (Improve node out of sync message) ( #1933 )
...
* Update validator_duties.nim
Improve node out of sync message
* Update validator_duties.nim
2020-11-02 08:52:16 +01:00
67113373f9
Work around BLST build issues
2020-10-30 23:56:04 +02:00
f76679810b
Fix #1855 ; Add support for wallet recovery passwords
2020-10-30 23:56:04 +02:00
63173ab2c1
Gossip11 post audit ( #1917 )
...
* bump libp2p gossip
* use gossip 1.1
* use proper libp2p commit hash
* switch to master libp2p, has audit fixes now
2020-10-30 14:26:35 +00:00
3f5c7c36bc
Some syncing fixes ( #1919 )
...
* Add exponential rewind on MissingParent.
* Try to avoid peers which are useless for syncing.
Fix forward sync restart at proper point.
Fix getLocalWallSlot() to not return slots from the future.
* Fix incorrect logs.
* Fix logging.
Enable peer's status messages log on DEBUG level.
* Fix watch task to monitor operation progress, but not local head progress.
* Add more logging information.
Remove recurring failures detection mechanism.
2020-10-30 13:33:52 +01:00
81f4fe0783
limit fork choice initialization horizon ( #1910 )
...
This helps manage the long startup time on chains that are not
finalizing for long periods of time
2020-10-29 12:09:03 +01:00
020a32ffa4
minor api cleanups ( #1913 )
...
* readd getNodeVersion
* implement get_v1_config_fork_schedule
* fix formatting of several types
2020-10-28 19:51:38 +01:00
19c51e5900
use MAXIMUM_GOSSIP_CLOCK_DISPARITY for BeaconBlock verification ( #1918 )
2020-10-28 14:04:21 +01:00
ff3f8741de
handle fork choice selecting a head unknown to chaindag ( #1907 )
...
* move head slot metric to head selection
* avoid redundant head selection in validator duties
2020-10-28 07:55:36 +00:00
6050eebec0
document not-nil issues
2020-10-27 13:01:45 +01:00
eee01a32d5
Regression fix of eth2_network_simulation on Windows. ( #1900 )
...
* Concentrate all sensitive writeFile/createPath calls in one place.
Fix eth2_network_simulation for Windows.
* Remove artifacts.
* fix import
Co-authored-by: Jacek Sieka <jacek@status.im>
2020-10-27 12:04:17 +01:00
c82ff24b5c
Syncing fixes ( #1909 )
...
* Fix continuous sync queue rewinds on slow PCs.
Fix recurring disconnects on low peer score.
* Calculate average syncing speed, not the current one.
Move speed calculation to different task.
* Address review comments.
2020-10-27 10:25:28 +01:00
ee2ebc96a8
move rpc api to own folder, mimic upstream structure ( #1905 )
...
also implements a few more endpoints
2020-10-27 10:00:57 +01:00
677f099110
BLS signature v4 ( #1912 )
2020-10-27 09:37:04 +01:00
8102d5dc4d
Bump eth2-testnets for more discv5.1 bootstrap nodes on medalla ( #1911 )
...
And improve logging UX a bit by ignoring comments or newlines
2020-10-27 08:44:46 +01:00
756b62fdf5
Bump nim-eth to get UseDiscv51 flag ( #1895 )
...
* Bump nim-eth to get UseDiscv51 flag
* Switch medalla to discovery v5.1, other targets to v5.0
* Bump nim-eth for better discv5.1 logging
* Bump eth2-testnets for updated medalla bootnodes
2020-10-23 18:59:58 +02:00
7c0b4d28d2
speed up reward/penalty calculation
...
Calculating rewards/penalties is slow due to how we compute sets of
attestations validators then use the sets for inclusion checks, to see
who attested. The dominant function during validated block processing /
epoch processing is hash set building and lookup.
This PR inverts the flow by removing the sets and creating a single
large validator status list, then applying all relevant state
attestations, then updating rewards and penalties.
This provides a 10x speedup to epoch processing which in turn speeds up
both empty slot and block processing - for example, on startup, we
replay all non-finalized blocks to prime fork choice - the same when
validating attestations or replaying states on reorg.
2020-10-23 19:23:36 +03:00
499e5ca991
misc memory and perf fixes ( #1899 )
...
* misc memory and perf fixes
* use EpochRef for attestation aggregation
* compress effective balances in memory (medalla unfinalized: 4gb ->
1gb)
* avoid hitting db when rewinding to head or clearance state
* avoid hitting db when blocks can be applied to in-memory state -
speeds up startup considerably
* avoid storing epochref in fork choice
* simplify and speed up beacon block creation flow - avoids state reload
thanks to head rewind optimization
* iterator-based committee and attestation participation help avoid lots
of small memory allocations throughout epoch transition (40% speedup on
epoch processing, for example during startup)
* add constant for threshold
2020-10-22 12:53:33 +02:00
decfd66a78
bump libp2p: seq[byte] message IDs and improve reject/ignore handling ( #1898 )
...
* bump libp2p: use seq[byte] message IDs and improve reject/ignore handling
* bump libp2p to pick up debug message details
2020-10-21 20:03:12 +00:00
9810a31447
Fix #1838 ( #1892 )
2020-10-20 15:01:21 +02:00
a136c2e95a
bump libp2p; integrate pubsub.ValidationResult into extended validation ( #1893 )
2020-10-20 12:31:20 +00:00
e87e0ef179
update to 1.0.0-rc.0 message ID; unexport funcs; enable test in mainnet; remove unused subscribe-with-nondummy-handler function; fix gossip received metric ( #1884 )
2020-10-20 08:54:11 +00:00
c446eb7cff
update v1.0.0-rc.0 preset spec references ( #1891 )
...
* update ve1.0.0-rc.0 preset spec references
* remove runtime preset ETH1_FOLLOW_DISTANCE from preset files; remove two CI build items to try to keep Travis from timing out
2020-10-19 13:53:23 +00:00
5b13a5ba10
update attestation extended validation to v1.0.0-rc.0 ( #1889 )
...
* update attestation extended validation to v1.0.0-rc.0
* attestation block and target must be within same epoch
* remove duplicate attestation epoch/target epoch check
* in spec now
2020-10-19 09:25:06 +00:00
df43b8aa8b
save some more states after all ( #1887 )
...
Don't save states when replaying history, but do save states when
applying new blocks (!)
2020-10-18 15:47:39 +00:00
dbc90e998a
Add Disc v5.1 bootstrap nodes for Zinken
...
Also prepares the Makefiles for supporting Medalla
through a separate binary
2020-10-18 12:44:52 +03:00
d98be078c3
Address #1690 Min password entropy
2020-10-16 19:34:26 +03:00
9899578e1c
Update get_eth1_data from v0.11.1 to v1.0.0-rc
2020-10-16 11:46:53 +03:00
9a543e0af7
partial hotfix for #1879 crash
2020-10-16 11:46:19 +03:00
948d15ea95
Follow-up on #1687 Don't ask for password on invalid keystores
2020-10-16 11:45:39 +03:00
db70f066ba
fixed all crashes which can be triggered through the validator RPC API - fixes #1680
2020-10-15 20:19:43 +03:00
7ed885f144
configuring a writer for the dynamic log output device for the VC binary as well (same as in the BN)
2020-10-15 20:19:10 +03:00
7ad59b1852
Bump nim-stew.
2020-10-15 20:18:33 +03:00
1742a82ca7
Proper fix for writeFile() places.
...
Comment unused/insecure procedure.
2020-10-15 20:18:33 +03:00
84fe294c00
Fix protection of data folder and security files using Windows ACL.
2020-10-15 20:18:33 +03:00
5f6bdc6709
Store all deposit-derived data in memory
2020-10-15 20:15:51 +03:00
7a577b2cef
More tests for getBlockRange
2020-10-15 20:15:51 +03:00
080609eee1
Address #1366 Avoid uint64 overflow in getBlockRange when skipStep is large
2020-10-15 20:15:51 +03:00
2152dc6136
Simplify the mainchain monitor
2020-10-15 20:15:51 +03:00
ce1fda1195
Store the deposits and the immutable validator data in Sqlite
2020-10-15 20:15:51 +03:00
e69af00e3a
Don't penalize flooding peers; Just rate limit them
2020-10-15 20:15:51 +03:00
4d66914f5a
Fix the test suite
2020-10-15 20:15:51 +03:00
f0892ce934
Fix compilation problems on Windows
2020-10-15 20:15:51 +03:00
e6320e5881
Address #1584 Don't keep all deposits in memory (persist them to disk)
2020-10-15 20:15:51 +03:00
bc8acdb9de
Switch to an in-house clearScreen to avoid problems with exception tracking
2020-10-15 20:15:51 +03:00
69e498dc00
Address #1689 and #1517 Usability and security improvements in wallet creation
2020-10-15 20:15:51 +03:00
02a3770803
Adds a fuzzing test for the validator client CLI
...
Also removes an obsolete way to add validators through the command line
2020-10-15 20:15:51 +03:00
8ce0fc3a89
Address #1687
...
This addresses the issues by detecting and rejecting keystores with
incorrect PBKDF2 and SCrypt params. It also bumps the version of
nim-json-serialization to include a bugfix for incorrect parsing
of json files featuring comments.
2020-10-15 20:15:51 +03:00
cd949a2b81
Address #1691 More pervasive handling of Ctrl+C
2020-10-15 20:15:51 +03:00
8a6281aad2
Simple cost model for sync requests; Penalize peers perfoming flooding or invalid requests
2020-10-15 20:15:51 +03:00
6b9419e547
fix db growth on attestation processing ( #1860 )
...
It turns out that we often save lots of states in the database that are
the result of empty slot processing only - here, we make sure to only
save a state if a block follows - this fixes several issues:
* empty slot states are not always pruned leading to state database size
explosion
* storing states is (very) slow which slows down processing in general,
so we should only do it when it's likely to be useful
* attestation processing doesn't get stuck on saving random states that
won't appear in the chain history
2020-10-15 14:28:44 +02:00
4e73d10f86
removed 2 buggy RPC API endpoints - there are 2 well functioning ones in validator_api.nim anyway ( #1867 )
2020-10-15 14:26:59 +02:00
da59f45a90
update spec/datatypes spec refs from v0.12.3 to v1.0.0 ( #1866 )
2020-10-14 10:00:27 +00:00
513ba72b9a
add v1.0.0-rc.0 support behind compile-time flag ( #1852 )
...
* add v1.0.0-rc.0 support behind compile-time flag
* keep runtime presets consistent
2020-10-13 17:21:25 +00:00
3ee2dd8da4
p2p-interface spec ref bump (except non-updated places) ( #1862 )
2020-10-12 14:37:14 +00:00
1994ffe5a0
update 130+ spec references from v0.12.3 to v1.0.0-rc1 ( #1854 )
2020-10-12 08:59:24 +00:00
00a8a68671
Address #1695
...
Better error messages when the beacon node is asked to listen on a
reserved port (0) or an already taken one.
2020-10-09 16:39:03 +03:00
d69d89866e
fix logtrace in local network sim ( #1840 )
2020-10-09 10:58:54 +02:00
b79e5f8af5
update nim-beacon-chain to nimbus-eth2 in beacon_chain/, ncli/, tests/, and README.md ( #1843 )
2020-10-08 19:02:05 +00:00
c0aa2f1e58
update process_slashings() to use PROPORTIONAL_SLASHING_MULTIPLIER ( #1841 )
2020-10-08 18:00:07 +00:00
3bd7ab4c20
Do not reward empty responses. ( #1827 )
...
Request status from peers every StatusExpirationTime.
2020-10-08 14:50:48 +02:00
f08f44b9a2
in exit pool, bundle receive messages into beaconblocks ( #1812 )
...
* in exit pool, filter out already-packaged messages; bundle remaining messages into beaconblocks
* filter messages at block construction time
* allow adding up to intended capacity of buffers, beyond per-block limits
* document rationale/design for filtering mechanism
2020-10-07 16:57:21 +00:00
b46c8410a6
Retire Spadina and prepare for the Zinken network
2020-10-07 18:12:21 +03:00
6b9090398e
[JSON-RPC] Prevent silent uint conversion ( #1819 )
...
* Prevent silent uint conversion
* bump json-rpc
2020-10-07 15:02:54 +02:00
deaddc1fc0
Address review comments
2020-10-07 09:32:03 +03:00
aed291128a
Add support for starting from weak subjectivity checkpoints
...
Also removes the `genesis.ssz` file stored in the data folder.
The `medalla-fast-sync` target has been adapted to use the new features.
2020-10-07 09:32:03 +03:00
2c06492407
Address #1728
2020-10-06 15:46:22 +03:00
950bcd0f6d
Address #1729 : Min password length
2020-10-06 15:46:22 +03:00
78953fd9b5
Address #1729 : NFKD Normalization
2020-10-06 15:46:22 +03:00
fec4b5014d
Address #1694
2020-10-06 15:46:22 +03:00
47d4899911
Wallet creation does not allow overwriting an existing wallet
2020-10-06 15:46:22 +03:00
9d23094c68
Address #1692
2020-10-06 15:46:22 +03:00
27fc8bcc80
Address #1538 and do not penalize peers for empty response. ( #1811 )
2020-10-06 14:10:02 +02:00
6bb6b66766
Partial LTO ( #1789 )
...
* initial partial LTO example
* Add select private key related files to ignore LTO
* Apply to keystores
2020-10-06 13:38:18 +02:00
c92d228ec8
Fix non-split attesting protection ( #1804 )
...
* address #1679 - att protection in non-split mode
* AttachedValidator public_key -> pubkey
* Always build with slashing protection
* Remove now unnecessary forwarding impl proc
2020-10-06 10:51:33 +02:00
5354a0a3d5
fix #1585 and restore storeLEBytes on big-endian systems ( #1796 )
...
* fix #1585 and restore storeLEBytes on big-endian systems
* switch to using stew/endians2
* create static endians2-like shim in bitseqs
2020-10-06 08:00:11 +00:00
8c59b92081
fixed #1663 - Interger overflow in compute_start_slot_at_epoch through RPC ( #1810 )
...
* fixed #1663 - Interger overflow in compute_start_slot_at_epoch through RPC
* changed the way the overflow check is done - took the approach from PR #1797 - see the comment in PR #1810 for more details
2020-10-05 21:58:30 +00:00
00a500ef3c
Fix rebase issues.
2020-10-05 22:19:50 +03:00
f091c8d4df
Add chronicles.formatIt for PublicKey.
...
Fix logs.
Rename checkFilePermissions to checkSensitiveFilePermissions.
2020-10-05 22:19:50 +03:00
dd8b73c85c
Remove `keypath` and set version to 1 for network keystorage.
2020-10-05 22:19:50 +03:00
9441e912cb
Merge different places of keyboard input to keyboardCreatePassword() and keyboardGetPassword().
...
Fix Windows problem with english words array.
Add checksum for english words array verification.
2020-10-05 22:19:50 +03:00
868ce14762
Fix compilation issue.
2020-10-05 22:19:50 +03:00
82228fe471
Fix last createDir.
2020-10-05 22:19:50 +03:00
86139839f1
Add permissions checks and handling to wallets and bls keystores.
2020-10-05 22:19:50 +03:00
fc09c2b3d4
Fix compilation problem.
2020-10-05 22:19:50 +03:00
1d76683a8b
Fix os dependency.
2020-10-05 22:19:50 +03:00
20f695515a
Fix localtestnet simulation.
2020-10-05 22:19:50 +03:00
add22a20e1
Update local_testnet and simulation scripts to use netkey-file and insecure-netkey-password.
...
Add more logging
2020-10-05 22:19:50 +03:00
e1182f8000
Add insecure password for automated testing.
...
Fix checkDataDir to run before setupLogging.
2020-10-05 22:19:50 +03:00
e43af4e978
Fix CLI option.
2020-10-05 22:19:50 +03:00
40f2b74f73
Add keystore management and interactive password handling.
2020-10-05 22:19:50 +03:00
ae72c08573
Fix utils problem.
2020-10-05 22:19:50 +03:00
cec4ad2717
Refactor keystore.nim to include network key storage too.
...
Use constant-time hex encoding/decoding procedures.
Add tests for network key storage.
2020-10-05 22:19:50 +03:00
c5c788a9db
Secure network key file and data directory.
2020-10-05 22:19:50 +03:00
d9738b43b3
Revert "Address issue #1538."
...
This reverts commit 1be3fd53b1
2020-10-05 19:25:47 +03:00
1be3fd53b1
Address issue #1538 .
2020-10-05 13:23:25 +03:00
67a93f558b
Address issue #1714 .
2020-10-05 13:22:45 +03:00
99afafecd7
fix quadratic seq assignment in fork choice ( #1805 )
...
this would reallocate the attestation queue on every attestation and
other call to update_time, causing quite the overhead (~10% cpu spent
when gossiping)
2020-10-03 23:43:27 +02:00
b57693ec0d
Logging update ( #1795 )
...
* Fix discovery log message trigger
* Bump chronicles - include https://github.com/status-im/nim-chronicles/pull/89 for better NOTICE/WARNING color
2020-10-03 08:35:45 +02:00
5e95fd7468
Revert "update to v0.12.3 message ID for Spadina launch ( #1762 )" ( #1801 )
...
This reverts commit a2270a5f27
2020-10-02 19:50:21 +00:00
f5340998b9
Add a fuzzing test for the beacon node CLI parser
2020-10-02 12:42:32 +03:00
ee114deb82
fix #1782 by checking for empty attester index sets ( #1786 )
2020-10-01 21:05:22 +02:00
0280d6c73e
Revisiting log levels ( #1788 )
...
* Update log level - https://github.com/status-im/nim-beacon-chain/issues/1779 https://github.com/status-im/nim-beacon-chain/issues/1785
* Address review comments
* Document the logging strategy [skip ci]
2020-10-01 20:56:42 +02:00
13d4831664
fix #1783 by not assuming that an attached validator will attest ( #1787 )
2020-10-01 14:48:24 +00:00
7eaaab908c
fix output of proposer slashing test fixture ( #1780 )
...
* fix output of proposer slashing test fixture
* run make test
* a few more v0.12.3 spec refs
2020-09-30 13:12:03 +00:00
0c921b7bc0
Multinet fixes ( #1774 )
...
* add missing hardcoded preset value
* wip
* wip
2020-09-29 10:12:57 +02:00
f6b8e26567
Fix typos [skip ci] ( #1771 )
2020-09-28 20:43:09 +02:00
2e6fd8976a
Ctrl+C now stops the BN when waiting for genesis ( #1761 )
...
* Ctrl+C now stops the BN when waiting for genesis
* proper shutdown when waiting for genesis if Ctrl+C is used
2020-09-28 17:19:57 +02:00
a2270a5f27
update to v0.12.3 message ID for Spadina launch ( #1762 )
...
* update to v0.12.3 message ID for Spadina launch
* remove base64 import
2020-09-28 17:07:10 +02:00
23bec99341
Let also discovery used listen-address cli option instead of always any address ( #1658 )
2020-09-27 22:00:24 +02:00
c472d5368b
avoid attestation pool copy in check ( #1755 )
2020-09-25 19:51:44 +02:00
Jacek Sieka
Zahary Karadjov
Eugene Kabanov
tersec
Ștefan Talpalaru
kdeme
Sacha Saint-Leger
Giovanni Petrantoni
Mamy Ratsimbazafy
Viktor Kirilov
Giovanni Petrantoni