Commit Graph

362 Commits

Author SHA1 Message Date
Eugene Kabanov eee01a32d5
Regression fix of eth2_network_simulation on Windows. (#1900)
* Concentrate all sensitive writeFile/createPath calls in one place.
Fix eth2_network_simulation for Windows.

* Remove artifacts.

* fix import

Co-authored-by: Jacek Sieka <jacek@status.im>
2020-10-27 12:04:17 +01:00
Zahary Karadjov 02a3770803 Adds a fuzzing test for the validator client CLI
Also removes an obsolete way to add validators through the command line
2020-10-15 20:15:51 +03:00
Zahary Karadjov deaddc1fc0 Address review comments 2020-10-07 09:32:03 +03:00
Zahary Karadjov aed291128a Add support for starting from weak subjectivity checkpoints
Also removes the `genesis.ssz` file stored in the data folder.
The `medalla-fast-sync` target has been adapted to use the new features.
2020-10-07 09:32:03 +03:00
Zahary Karadjov fec4b5014d Address #1694 2020-10-06 15:46:22 +03:00
cheatfate 82228fe471 Fix last createDir. 2020-10-05 22:19:50 +03:00
cheatfate add22a20e1 Update local_testnet and simulation scripts to use netkey-file and insecure-netkey-password.
Add more logging
2020-10-05 22:19:50 +03:00
cheatfate e1182f8000 Add insecure password for automated testing.
Fix checkDataDir to run before setupLogging.
2020-10-05 22:19:50 +03:00
cheatfate e43af4e978 Fix CLI option. 2020-10-05 22:19:50 +03:00
cheatfate c5c788a9db Secure network key file and data directory. 2020-10-05 22:19:50 +03:00
Zahary Karadjov f5340998b9 Add a fuzzing test for the beacon node CLI parser 2020-10-02 12:42:32 +03:00
Kim De Mey 23bec99341
Let also discovery used listen-address cli option instead of always any address (#1658) 2020-09-27 22:00:24 +02:00
Viktor Kirilov e926db22b9 initial implementation of attestation aggregation for the validator API 2020-09-24 11:31:14 +03:00
Eugene Kabanov c7c9b9d5f1
Syncing V2 (#1602)
* Syncing workers now not bound to peers.
Sync status is now printed in statusbar.

* Add `SyncQueue.outSlot` to statusbar too.

* Add `inRangeEvent` and `rangeAge` parameter.

* Fix rangeAge is not depends on SyncQueue latest slot.
Fix syncManager to start from latest local head slot.

* Add notInRange event.

* Remove suspects field.
2020-09-11 14:46:01 +02:00
Viktor Kirilov d9f9949ef0 use a separate process for the private keys (Off by default) - there is a new signing_process binary which loads all validators of the beacon node and the BN dictates through stdin of the signing process what to be signed and when and reads from stdout of the process 2020-09-02 12:47:00 +03:00
Viktor Kirilov 65d7787b1e 50/50 bn/vc split for the validator keys ON by default for the testnet scripts 2020-09-01 16:39:07 +03:00
Jacek Sieka a7a279d615
add option to disable discv5 (#1509) 2020-08-24 13:52:06 +02:00
Jacek Sieka 10da7fe9da remove eth from default status bar
not viable at higher validator counts - linear scan + forced public key
init makes it extremely slow
2020-08-10 17:01:53 +03:00
Zahary Karadjov c293254ded Add 'deposits import' command; Switch to NJS when loading the keystores and improve the data validation 2020-08-02 23:00:43 +03:00
Zahary Karadjov d58668157a Use consistent naming for the synonyms 'deposits-dir' and 'validators-dir' 2020-08-02 23:00:43 +03:00
Zahary Karadjov 0c60e452e7 Allow the deposit contract deployment block to be specified as a number (needed for Medalla) 2020-07-28 18:13:01 +03:00
Zahary Karadjov 40ea9e9886 Integrate the Wallet support with the Deposits creation; Produce Launchpad-compatible deposits 2020-07-28 07:36:25 +03:00
Zahary Karadjov fcd412f7a1 Finish the 'create wallet' command; Addresses #1319 2020-07-28 07:36:25 +03:00
Ștefan Talpalaru e9193fc9da
eth2_network_simulation: fixes (#1368)
- delete "tests/simulation/{data,validators}" by default, because old
  validator keys can lead to a crash with a cryptic error message
- actually start the Prometheus daemon on `make eth2_network_simulation`
- kill any running Prometheus daemon on exit
- fix some shell script syntax incompatible with Bash
2020-07-23 17:58:54 +02:00
zah 8970a22fdd
Add a non-optional Json file log for the beacon node as stopgap measure (#1322)
* Add a non-optional Json file log for the beacon node as stopgap measure

* Fix the build with -d:testnet_servers_image
2020-07-15 15:15:55 +02:00
Zahary Karadjov 93b04bc214 Add an option for graffiti customization 2020-07-12 21:01:31 +03:00
Zahary Karadjov 87928e0069
Documentation; 'deposits create' can now ask for a private key 2020-07-10 18:18:14 +03:00
Zahary Karadjov 540b2828b2
Adapt the local sim scripts to use the new run-time presets 2020-07-10 01:08:54 +03:00
Zahary Karadjov c4af4e2f35
Working test suite with run-time presets 2020-07-08 02:02:14 +03:00
Zahary Karadjov 98761afc64
Run beacon_node --network=altona without initial Eth1 syncing 2020-07-02 18:52:48 +03:00
Zahary Karadjov e342b96d2e
Embed the Altona metadata in the NBC executable 2020-07-02 18:14:11 +03:00
Zahary Karadjov 62e4efe34b Re-wire the sending of deposits by the 'deposits create' command 2020-06-24 23:06:05 +03:00
Zahary Karadjov 384e512031 Implement EIP-2386 wallets; Progress towards a CLI for interactive wallet creation
For more information:
4494da0966/EIPS/eip-2386.md (specification)
2020-06-24 23:06:05 +03:00
Ștefan Talpalaru bdd56b626f
Merge pull request #1206 from status-im/readme
Update readme and manual
2020-06-24 01:04:25 +02:00
Zahary Karadjov 7211a99086 Smooth out the differences between Ganache and Infura; Working local sim and Altona target 2020-06-22 17:30:04 +03:00
Zahary Karadjov 1def383ad1 Reword the Tmux setup to handle all simulation scenarios in more visible way 2020-06-22 17:30:04 +03:00
Zahary Karadjov e9d68e2f7b Add simulation mode that bootstraps from an Eth1 ganache instance 2020-06-22 17:30:04 +03:00
李婷婷 fd35b61ecb fix typo 2020-06-19 23:00:18 +08:00
Jacek Sieka 49e9167b28 clean up dump feature
* don't write blocks that get added to database
* don't write states
* write to folders
* add state dumping feature to `ncli_db` to get any known state from the
database
2020-06-16 13:44:37 +00:00
Zahary Karadjov 6e88a07b51 Fix the validator client build 2020-06-11 17:40:08 +03:00
Zahary Karadjov 2acda1c115 Provide a default value for secretsDir (similar to validatorsDir) 2020-06-11 17:40:08 +03:00
Zahary Karadjov a8113cf2bc Restore the local sim to a working state 2020-06-11 17:40:08 +03:00
Zahary Karadjov 17343442ea Implement more of the KeyStore spec and integrate it in the beacon node 2020-06-11 17:40:08 +03:00
Viktor Kirilov 3bae40ae91 - extracted the commands to run a VC into a separate run_validator.sh script
- each BN gets half of its previous validators as inProcess and the other half goes to the respective VC for that BN - using separate data dirs where the keys are copied
    - also removed a few command line options which are no longer necessary
- block proposals originating from a VC are propagated from one BN to the rest properly
- other cleanup & moving code back to  since it is no longer used elsewhere
2020-06-10 13:50:50 +03:00
Jacek Sieka 1e71388093
Include finalization, head and time (in that order) in standard bar (#1142)
These are the 3 most important things in the life of a beacon-node -
finalization checkpoint for security, head + time for seeing that we're
synced and working.

The order given is the expected increasing order that things should be
moving in - finalization lags head, and head lags clock - the structure
helps the user remember, but also see the pattern where time is getting
out of sync with head.

A validator would also want to know when the next action is going to be
(ie next attestation / block production) but that's for another day.
2020-06-08 18:04:11 +02:00
Jacek Sieka bb698a7e16
lots of peers (#1130) 2020-06-05 22:25:01 +02:00
Zahary Karadjov 0c78fc39e7
Use the latest LibP2P 2020-06-05 19:34:12 +03:00
Jacek Sieka f53b55cbe0 SSZ cleanup
* be stricter about SSZ length prefix
* compute zeroHash list at compile time
* remove SSZ schema stuff
* move SSZ navigation to ncli
* cleanup a few leftover openArray uses
2020-05-28 19:15:16 +03:00
Viktor Kirilov cefd525ab3 more work on the BN/VC split
- fixed comments from the last review
- getting more data VIA RPC, moved some code back into the BN only
- attestation duties being requested as well
2020-05-28 13:01:42 +03:00
Viktor Kirilov 8760494c72 first batch of work towards the VC/BN split:
- we have a new binary which connects via RPC to the respective BN and has an internal clock - waking it up on every slot
- the BN has a new option called --external-validators and currently in order to have the VC binaries to run we need to pass EXTERNAL_VALIDATORS=yes to make
- factored some code out of beacon_node.nim for easier reuse in validator_api.nim and validator_client.nim
- the VC loads its associated private keys from the datadir for its BN
- most of the validator API calls have been implemented as a stub.
- the VC polls its BN at the start of each epoch - getting a list of all active validators for the current epoch - and then continues to request blocks and sign them with its appropriate validators when necessary
2020-05-25 16:23:15 +03:00
Zahary Karadjov 7237a370ba Implement https://github.com/status-im/nim-beacon-chain/issues/872
Please see the newly added 'schlesi-dev' Makefile target.
It demonstrates how the log level can be specified for individual topics.

Additionally, when connecting to testnets like 'schlesi' there will be
two additional log files produced in the working directory:

* json-log.txt
* text-log.txt (in the textblocks format)
2020-04-28 13:49:30 +03:00
Zahary Karadjov 740b76d152 Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim
BEWARE! This commit will trigger a stack overflow during local sim
2020-04-26 13:04:53 +03:00
Jacek Sieka ed74770451
spec: regulate exceptions (#913)
* spec: regulate exceptions

* a few more simple raises
2020-04-22 07:53:02 +02:00
Zahary Karadjov d2b7ee27de The RPC service listens on localhost by default 2020-03-22 19:00:09 +02:00
Zahary Karadjov adcec61081 Initial implementation of a JSON-RPC service 2020-03-22 19:00:09 +02:00
Mamy André-Ratsimbazafy bfd4df912e Update to BLS v0.10.1 2020-03-13 12:20:41 +00:00
Zahary Karadjov 7f395e554a WIP Working local network sim with discovery
* The bootstrap_nodes.txt file in the node's data dir is now optional
* Log more data on start-up
* Use the latest ENR APIs
* Fix simulation build errors
2020-02-18 12:53:49 +02:00
Zahary Karadjov ee7c2c9dff Unify the bootstrap nodes handling code
We no longer discriminate between ENR, MultiAddress or ENode
bootstrap records (all of them are remapped to ENodes).

The discovery loop will stochastically try to reconnect to
accidentally disconnected nodes.
2020-02-18 12:53:49 +02:00
Dustin Brody bde1eefdcf improve option naming 2020-02-17 22:00:41 +00:00
Dustin Brody e13846bec6 allow beacon_node to verify finalization when appropriate, and enable by default for make eth2_network_simulation 2020-02-17 22:00:41 +00:00
Zahary Karadjov 07d329acb8 Wire-up the peer dialing loop and introduce --max-peers option 2020-01-21 01:26:57 +02:00
Zahary Karadjov 52878405b7 Integrate Discovery V5 and support ENR bootstrap records 2020-01-21 01:26:57 +02:00
Jacek Sieka a3145f0f5c cleanups
* move stuff out of beaconnode
* fix use-after-free in query
2019-12-10 14:47:52 +00:00
Jacek Sieka afc0686b36 save ssz dummps during processing (#471) 2019-12-03 11:32:27 +00:00
Zahary Karadjov 4c2ded25a6 Add detailed tracing for blocks by range requests 2019-11-26 02:45:25 +02:00
Zahary Karadjov ac5aabd569 Use the latest confutils 2019-11-11 23:29:36 +00:00
Zahary Karadjov ed9cb274a2 Restore a change mysteriously lost during rebasing 2019-11-11 23:29:36 +00:00
Ștefan Talpalaru c1a63035b9 beacon_node: "--version" option [skip ci]
fixes #466
2019-11-09 17:43:42 +00:00
Yuriy Glukhov 568e669b1c Fixed deposit/contract 2019-11-07 17:23:06 +00:00
Zahary Karadjov 72c42838cb
Fix a typo in start.sh; Display the full slot number in the statusbar [skip ci] 2019-10-29 21:48:32 +02:00
Zahary Karadjov f1ea0cec32
Add support for testnets with mixed quickstart/random deposits 2019-10-29 19:16:32 +02:00
Zahary Karadjov ba0037738b
Removed all code related to the old network metadata files
Also switched to a more "standard" naming convention for the
command-line parameters.
2019-10-29 01:04:52 +02:00
Jacek Sieka ee5d1c1467
minimize imports, specially for spec - cuts 2/3 of ncli build time (#500)
* minimize imports, specially for spec - cuts 2/3 of ncli build time
* ptr_arith->ptrops
2019-10-25 12:59:56 +02:00
Zahary Karadjov c23b011c77 Cosmetic improvements to the status bar; Attempt to fix the CI 2019-10-07 23:56:34 +09:00
Zahary Karadjov f72a58595f Initial version of status bar 2019-10-07 23:56:34 +09:00
Zahary Karadjov 0eaa433e84
Automated scripts for resetting the Status testnets
For detailed instructions, please see

https://github.com/status-im/nimbus-private/blob/master/testnets-maintenance.md
2019-09-26 18:58:51 +03:00
Jacek Sieka 42a469603d
simplify startup, describe a few startup tricks 2019-09-11 20:29:57 -04:00
Jacek Sieka 65ff8f2886
remove networkId (unused) 2019-09-11 20:29:57 -04:00
Jacek Sieka ab2cce3951
add quickStart option that avoids network metadata 2019-09-11 20:29:57 -04:00
Zahary Karadjov f583135a39
The --stateSnapshot option now accepts ssz files as well 2019-09-10 10:13:49 -04:00
zah 93cdb43f1e Integrate nim-metrics and add some metrics from the ETH2 spec (#394) 2019-09-07 19:48:05 +02:00
Jacek Sieka 7b73b40bab interop updates
* add interop launcher scripts
* stick validator_keygen into beacon_node
* fix lmd ghost slot number on missing block
* use mocked eth1data when producing blocks
* use bls public key method for withdrawal credentials
* fix deposit domain
* prefer lowercase for a bunch of toHex
* build simulation binary in data folder to avoid data types confusion
2019-09-02 22:14:18 +03:00
Yuriy Glukhov 958e4cd3b4 Getting genesis through web3 2019-07-31 16:39:26 +03:00
Zahary Karadjov 12e9367f78
Improved error handling; Simple test case for connecting 2 peers 2019-06-24 05:34:22 +03:00
Zahary Karadjov f4a96bc3f3
[WIP] Restore the Lib2P2 builds and implement the latest wire spec
Depends on https://github.com/status-im/nim-eth/pull/54
2019-06-24 05:34:21 +03:00
Ștefan Talpalaru 53012b9a25
NAT traversal 2019-04-18 15:48:17 +02:00
zah e16d8faee2 Switch to a two-steps build process (#228) 2019-03-29 10:44:19 -06:00
Zahary Karadjov 8e0025035c all validators are free for picking by default 2019-03-29 15:38:59 +02:00
Zahary Karadjov b3d23c052c Switch to 30 seconds slot duration; UX improvements
* On the testnet site, the first N instead of the last N validators
  will be considered reserved for users

* The number of validators is consistently named `totalValidators`
  in all commands

* Proper keys are generated by default in validator_keygen
2019-03-27 18:13:39 +02:00
Zahary Karadjov a3ce9d17ec Expose the default port 2019-03-27 18:13:39 +02:00
Zahary Karadjov 6629897567 We don't need the more complicated port setup anymore 2019-03-27 18:13:39 +02:00
Zahary Karadjov c0fa229254 Improved error handling in the networking code 2019-03-26 17:40:20 +02:00
Zahary Karadjov d94d4f2606 Dumbed down the validator key import procedure
The previous idempotent scheme proved to be too slow, making it
harder to quickly re-assign validators on the servers.
2019-03-26 17:37:31 +02:00
Zahary Karadjov 0a027e410a Address review concerns and make some steps towards json logging 2019-03-22 17:04:17 +02:00
Zahary Karadjov 06301458bf Don't add the 'node' log property when connecting to the testnet 2019-03-22 17:04:17 +02:00
Zahary Karadjov 42ca6afae2 Helpful scripts for resetting the testnets 2019-03-19 21:50:22 +02:00
Zahary Karadjov 6a35d3584d Automate the creation of the network metadata files
With these changes, running a simulation is very close to running
an actual testnet. Some checks have been added in the client to
make sure you are not connecting to an incompatible network (e.g.
a network running with a different number of shards).
2019-03-19 20:27:54 +02:00
Jacek Sieka d7f972a47e allow setting advertised external ip
* detect it in start.sh
2019-03-19 10:50:09 +02:00
Zahary Karadjov 6bb38095c9 Command-line and data storage handling for testnets 2019-03-18 12:45:29 +02:00
Jacek Sieka 4d55cf8eea beacon node sim: various improvements (fixes #111) (#156)
* allow running more or fewer validators
* use deterministic key generation for tests to avoid exhausting system
RNG
* update README with simulator docs
* write the data of each validator to separate file, instead of a big
chainstart.json (makes it easier to run different validator counts)
2019-03-07 13:59:28 +00:00
Jacek Sieka 0e9cc20cfe beacon node simplifications (#116)
* remove previous randao scheme code
* fix some epoch logging
* move genesis time selection to state generation
2019-02-15 17:33:32 +01:00
Dustin Brody 8fbf87aa8c complete switching to v0.2.0 RANDAO scheme, with verification enabled 2019-02-14 13:41:04 -08:00
Mamy Ratsimbazafy e5d152c6d6
Update BLS to the latest scheme. (#80)
* Attempt to switch to nim-blscurve.

* "Fix" BLS test for new tests

* Missing domain param in validator keygen
2019-02-05 17:13:29 +01:00
Zahary Karadjov 43ae45fe7e Add --logLevel command-line flag; Fix running the start.sh script from the beacon-chain repo 2019-01-17 14:38:27 +02:00
Zahary Karadjov 8f9a5441f1 More reliable network simulation
You'll need the latest versions of nim-eth-p2p, nim-serialization
and nim-json-serialization.

Before starting the simulation script, make sure to delete any previous
json files from the simulation folder:

```
rm tests/simulation/*.json
tests/simulation/start.sh
```

This should survive the creation of few blocks before diying with a
block validation error.
2019-01-05 14:35:47 +02:00
Jacek Sieka 12a819c110
spec updates (#48)
* spec updates
* balances move out to separate seq
* bunch of placeholders for proof-of-custody / phase1
* fix inclusion distance adjustment
* modify state in-place in `updateState` (tests spent over 80% time
copying state! now it's down to 25-50)
* document several conditions and conversations
* some renames here and there to follow spec
2018-12-27 14:14:37 -06:00
Zahary Karadjov abb199d6dc Chain creation and network simulation start script 2018-12-19 15:39:00 +02:00
Yuriy Glukhov 10ed2bd5b9 Comments addressed, cosmetics 2018-12-09 11:17:54 +02:00
Yuriy Glukhov 61f81fb0c4 Some bits of randao 2018-12-09 11:17:54 +02:00
Yuriy Glukhov f12b679880 addLocalValidators impl 2018-12-05 18:58:28 +02:00
Zahary Karadjov e4d1bebf4d Get the code to compiling state 2018-11-29 03:17:57 +02:00
Zahary Karadjov 4920c0a357 work-in-progress beacon node skeleton (not compiling at the moment) 2018-11-28 22:55:04 +02:00