Commit Graph

265 Commits

Author SHA1 Message Date
Ștefan Talpalaru 92750d5313 reduce peak memory usage during build 2020-12-10 17:17:12 +02:00
Ștefan Talpalaru ad54e9d3bf
Prometheus/Grafana changes (#2092)
- grafana/prometheus.yml.example
- drop the custom label
- raise the minimum supported Grafana version to 7.2
- increase scrape_interval
- add validator reward rate panel (Zahary's)
2020-12-02 21:47:00 +01:00
Ștefan Talpalaru 9c5cef346b set file and dir permissions 2020-11-17 23:28:26 +02:00
Zahary Karadjov 500301c6c7
Restore the NODE_ID functionality and use the old data directories 2020-11-10 10:11:13 +02:00
tersec a010ef2061
update tests from v1.0.0-rc.0 to v1.0.0 (#1976) 2020-11-09 17:44:18 +00:00
Zahary Karadjov 1774da15f1 Implement help message for the run-*-beacon-node scripts 2020-11-09 17:09:49 +02:00
Zahary Karadjov e9b9cd75ee Rename binaries; Mimic the original repo layout in the distribution 2020-11-09 11:38:52 +02:00
yslcrypto 6f9d0fa29e update beth1 client guide link 2020-11-07 15:16:53 +01:00
Zahary Karadjov 0ea68d61ed
Add another missing file 2020-11-07 13:51:14 +02:00
Ștefan Talpalaru f3254fd754
dist: add version and book (#1965) 2020-11-06 17:26:03 +01:00
Ștefan Talpalaru c7d8b81c86 dist: add Medalla binary and starting script 2020-11-06 01:29:03 +02:00
cheatfate 4caaf86332
Fix permissions on script. 2020-10-27 13:58:56 +02:00
Eugene Kabanov eee01a32d5
Regression fix of eth2_network_simulation on Windows. (#1900)
* Concentrate all sensitive writeFile/createPath calls in one place.
Fix eth2_network_simulation for Windows.

* Remove artifacts.

* fix import

Co-authored-by: Jacek Sieka <jacek@status.im>
2020-10-27 12:04:17 +01:00
tersec c446eb7cff
update v1.0.0-rc.0 preset spec references (#1891)
* update ve1.0.0-rc.0 preset spec references

* remove runtime preset ETH1_FOLLOW_DISTANCE from preset files; remove two CI build items to try to keep Travis from timing out
2020-10-19 13:53:23 +00:00
cheatfate f632a61cb9 Fix posix version. 2020-10-15 20:18:33 +03:00
cheatfate e0fcda090f Fix executable permissions. 2020-10-15 20:18:33 +03:00
cheatfate 84fe294c00 Fix protection of data folder and security files using Windows ACL. 2020-10-15 20:18:33 +03:00
Zahary Karadjov aed291128a Add support for starting from weak subjectivity checkpoints
Also removes the `genesis.ssz` file stored in the data folder.
The `medalla-fast-sync` target has been adapted to use the new features.
2020-10-07 09:32:03 +03:00
cheatfate 20f695515a Fix localtestnet simulation. 2020-10-05 22:19:50 +03:00
cheatfate 28630a1d71 Create DATA_DIR explicitly with 0750 permissions. 2020-10-05 22:19:50 +03:00
cheatfate add22a20e1 Update local_testnet and simulation scripts to use netkey-file and insecure-netkey-password.
Add more logging
2020-10-05 22:19:50 +03:00
Zahary Karadjov f5340998b9 Add a fuzzing test for the beacon node CLI parser 2020-10-02 12:42:32 +03:00
tersec 7eaaab908c
fix output of proposer slashing test fixture (#1780)
* fix output of proposer slashing test fixture

* run make test

* a few more v0.12.3 spec refs
2020-09-30 13:12:03 +00:00
tersec 5d1ea5e440
match log filenames more precisely for logtrace analysis (#1617)
* match log filenames more precisely for logtrace analysis
2020-09-09 13:08:21 +00:00
Viktor Kirilov d9f9949ef0 use a separate process for the private keys (Off by default) - there is a new signing_process binary which loads all validators of the beacon node and the BN dictates through stdin of the signing process what to be signed and when and reads from stdout of the process 2020-09-02 12:47:00 +03:00
Viktor Kirilov 65d7787b1e 50/50 bn/vc split for the validator keys ON by default for the testnet scripts 2020-09-01 16:39:07 +03:00
tersec b4f9908d9d
run logtrace in finalization CI (#1512)
* run logtrace in finalization CI

* use specified DATA_DIR rather than assuming default

* convert rest of hardcoded local_testnet_data references

* logtrace asr needs DEBUG-level logs

* clean up some shell (not) quoting issues
2020-08-17 08:36:29 +02:00
tersec bc6eefe31e
add --enable-logtrace argument to launch_local_testnet (#1502)
* add --enable-logtrace argument to launch_local_testnet

* scan for all available logfiles

* remove specific filename references

* update v0.11.3 spec ref to v0.12.2
2020-08-16 11:12:19 +02:00
Mamy Ratsimbazafy 454b9d0724
Bump nim-blscurve (#1491)
* Bump BLSCurve

* Use unified aggregation API

* use new blscurve with unified aggregate API

* bump

* fix toRaw

* replace state_sim combine with AggregateSignature

* Fix 32-bit

* Fix 32-bit for real and test deactivating ccache for fno-tree-lopp-vectorize flag

* change compilation switches to narrow down Linux issue

* Use -fno-tree-vectorize to disable both tree-loop-vectorize and tree-slp-vectorize

* blscurve now disables both Loop and SLP vectorization

* Add tests for the miracl/milagro fallback

* Travis has max log size of 4MB

* Test with Miracl in the finalization test

* fix state_sim log level

* Coment out the slow fallback tests
2020-08-15 19:33:58 +02:00
tersec af3355e0f8
create local testnet mode for eth2_network (#1494) 2020-08-12 14:16:59 +00:00
Zahary Karadjov d58668157a Use consistent naming for the synonyms 'deposits-dir' and 'validators-dir' 2020-08-02 23:00:43 +03:00
Ștefan Talpalaru 6a30c4a67d
launch_local_testnet.sh: add "--reuse-existing-data-dir" (#1396)
- also unify the indentation towards the least readable denominator, due
  to ".editorconfig" making my Vim display tabs as 2 spaces wide.
2020-07-30 02:46:10 +02:00
Ștefan Talpalaru fa9f35e148
Jenkins: run local testnet test on macOS (#1391) 2020-07-29 14:08:27 +02:00
Zahary Karadjov f4c19e303a Non-interactive generation of keystores in the local sim 2020-07-28 07:36:25 +03:00
Zahary Karadjov 40ea9e9886 Integrate the Wallet support with the Deposits creation; Produce Launchpad-compatible deposits 2020-07-28 07:36:25 +03:00
tersec 20a2525390
v0.12.2 beacon chain protocol update (#1378) 2020-07-27 12:59:57 +02:00
Ștefan Talpalaru c47532f2b0
deal with a temporary loss of network connectivity (#1354)
* don't kill the program if not connected to a bootstrap node within 30 seconds

* recover faster from loss of network connectivity

* connectWorker(): sleep 1s between dials

* launch_local_testnet.sh: increase BOOTSTRAP_TIMEOUT

* don't use metric value in program logic

* refactor some ungainly variable names
2020-07-23 22:51:56 +02:00
Zahary Karadjov 5a4f6b7e5e
Restore the SSZ fuzzign tests to working state 2020-07-22 16:20:07 +03:00
Ștefan Talpalaru ffd4bc1dd4
launch_local_testnet.sh: make node 0 the bootstrap node 2020-07-16 19:04:57 +02:00
Zahary Karadjov d16676b294
Bake the metadata for testnet0 and testnet1 as well; Remove the old scripts 2020-07-10 17:39:54 +03:00
Zahary Karadjov 540b2828b2
Adapt the local sim scripts to use the new run-time presets 2020-07-10 01:08:54 +03:00
Viktor Kirilov 1482b0430d - work towards more REST API endpoints being implemented
- testnets can now be launched with a separate validator client - make altona SCRIPT_PARAMS="--separateVC"
- reverted the ctrl+C signal handler code reuse - not necessary for the VC anyway (default is good enough)
- added a bit more logging in the VC
- removed unnecessary code in the VC - connect() just parses the address & port...
- fixed a couple more VC issues - when fetching the duties for an epoch fails on the BN side ==> the VC shouldn't be left in a broken state
- documented the currently supported json-rpc endpoints
- added more checks on the BN side for the API - bounds-checking the requests & also checking if the BN itself is synced
- other cleanup

currently a local sim doesn't finalize, but participation in the altona network with a separate VC is painless and works just as well as with in-process validators in a BN
2020-07-08 13:29:03 +03:00
Ștefan Talpalaru 20ac6d46fe
connect_to_testnet.nims: --extraBeaconNodeOptions
make SCRIPT_PARAMS="--extraBeaconNodeOptions='--metrics-address=127.0.0.1 --rpc-address=127.0.0.1 --nat=any'" altona
2020-06-30 19:16:09 +02:00
cheatfate d71ce6a60a Fix names 2020-06-30 09:59:48 +03:00
cheatfate a6f359527d Allow to specify rpcAddress and metricsAddress. 2020-06-30 09:59:48 +03:00
tersec b4db2ad693
remove v0.11.3 support; add block_sim to CI (#1253)
* remove v0.11.3 support; add block_sim to CI

* rm stray PERSISTENT_COMMITTEE_PERIOD

* remove TopicFilter.InteropAttestations

* bump two comment-spec-refs to v0.12.1
2020-06-29 18:08:58 +00:00
Eugene Kabanov 7712442236
Allow nat option for connect_to_testnet script. (#1250) 2020-06-29 12:05:00 +02:00
Zahary Karadjov dbb3dafce1
Switch to a new Infura account as a band-aid 2020-06-29 12:43:01 +03:00
Zahary Karadjov 96ac81c89f Better UX; Don't delete testnet data folders 2020-06-29 02:18:48 +03:00
Ștefan Talpalaru d85ae34245
shared_testnet: validator creation [skip ci] 2020-06-28 03:51:00 +02:00
Ștefan Talpalaru 675024a989
new Docker image for shared testnets 2020-06-26 17:26:18 +02:00
Zahary Karadjov b67a506b3f
Fixes for local sim on macOS [skip ci] 2020-06-25 12:42:30 +03:00
Ștefan Talpalaru 33ad3c05fe
connect_to_testnet.nims: add "--buildOnly" and "--runOnly" opts [skip ci]
and "--writeLogFile"
2020-06-24 12:54:17 +02:00
tersec 1b3c19dc21
default to v0.12.1 beacon chain spec, keeping witti working (#1222)
* default to v0.12.1 beacon chain spec, keeping witti working

* a couple of fixes that help for the Onyx testnet
2020-06-23 19:40:44 +00:00
Zahary Karadjov eb48c05b7f Don't assign validator keys to the bootstrap node; Ganache mode for launch_local_testnet 2020-06-23 07:59:35 +00:00
Zahary Karadjov 7211a99086 Smooth out the differences between Ganache and Infura; Working local sim and Altona target 2020-06-22 17:30:04 +03:00
Zahary Karadjov 1def383ad1 Reword the Tmux setup to handle all simulation scenarios in more visible way 2020-06-22 17:30:04 +03:00
Zahary Karadjov e9d68e2f7b Add simulation mode that bootstraps from an Eth1 ganache instance 2020-06-22 17:30:04 +03:00
Zahary Karadjov 4275abbefe Makefile targets for ctail and ntu 2020-06-22 17:30:04 +03:00
Ștefan Talpalaru 3a3d3f9bde
timestamps in log files 2020-06-17 16:44:48 +02:00
Ștefan Talpalaru 673eeb6a65
connect_to_testnet.nims changes [skip ci]
- remove `--dev-build` option
- unconditionally write the "nbc.log" file, but do it after a chdir to
  dataDir because Chronicles doesn't seem to support proper paths for
  "file(...)" in sink definitions
- change base RPC port (9090 -> 9190) because 9090 is the default
  Prometheus daemon listening port
2020-06-17 16:03:34 +02:00
Jacek Sieka 8fbbd59885
metric names (#1188)
* fix metric names to not clash with native libp2p metrics
* run testnet node with rpc enabled by default
2020-06-17 13:04:24 +02:00
Ștefan Talpalaru f5ec1bc5bb
support multiple local Witti nodes [skip ci] 2020-06-11 18:42:58 +02:00
Ștefan Talpalaru 421378b92d
connect_to_testnet.nims: mkDir dataDir 2020-06-11 17:37:27 +02:00
Zahary Karadjov 5807e2a767 Handle the delayed creation of the bootstrap node address file 2020-06-11 17:40:08 +03:00
Zahary Karadjov ff49932bb9 Reduce the number of validators to their older values (before this branch) 2020-06-11 17:40:08 +03:00
Zahary Karadjov bea243ae04 Attempt to fix the CI finalization test
Since I'm not able to reproduce the finalization failure locally
and it does happen only sporadically, one possible explanation is
that the introduction of keystores lead to a slower initialization
of the beacon nodes which somehow interferes with their behavior
during the initial slots.

If increasing the start-up delay fixes the problems, the hypothesis
will be confirmed.
2020-06-11 17:40:08 +03:00
Zahary Karadjov 2acda1c115 Provide a default value for secretsDir (similar to validatorsDir) 2020-06-11 17:40:08 +03:00
Zahary Karadjov a75c632f7a Fixed launch_local_testnet; Renamed validator_keygen to keystore_directories 2020-06-11 17:40:08 +03:00
Zahary Karadjov e3378e52ac Distribute the keystore secret files to the testnet hosts 2020-06-11 17:40:08 +03:00
Zahary Karadjov a8113cf2bc Restore the local sim to a working state 2020-06-11 17:40:08 +03:00
Zahary Karadjov 17343442ea Implement more of the KeyStore spec and integrate it in the beacon node 2020-06-11 17:40:08 +03:00
Ștefan Talpalaru 8e648da399
connect_to_testnet.nims: don't die when GNU getopts is missing 2020-06-11 04:11:30 +02:00
Ștefan Talpalaru e2025c5752
Prometheus & Grafana refactoring
- moved "process_dashboard.nim" in "tools/"
- README: made Witti the documented testnet and added instructions for
  getting metrics out of the local node
- moved Prometheus config file generation in its own script
- the static Grafana dashboard definition now covers all nodes, using
  a variable; only the remote testnet dashboards need to be dynamically
  generated
- "launch_local_testnet.sh" no longer needs a "--grafana" option
2020-06-10 19:55:26 +02:00
Ștefan Talpalaru 7c7b6fb361
Grafana: only show one node per page [skip ci]
by using Grafana variables to get dropdown menus
2020-06-07 01:42:54 +02:00
Zahary Karadjov 0c78fc39e7
Use the latest LibP2P 2020-06-05 19:34:12 +03:00
Ștefan Talpalaru 3a6a9f8135 generate Grafana dashboards for remote testnet nodes 2020-06-03 12:44:16 +03:00
Zahary Karadjov 0bcdabfcdf Detect another invalid SSZ input found through fuzzing
The first offset of an SSZ object should always have a fixed constant
value. Otherwise, some unused bytes may appear between the fixed portion
and the dynamic portion.

Please note that this fix shutds down the minimal forward compatibility
currently supported by the SSZ format (and thus, the expected behavior
must be clarified in the SSZ spec).
2020-05-30 14:47:35 +03:00
Ștefan Talpalaru b0a7cb17eb
connect_to_testnet.nims: enable metrics [skip ci]
you still need to pass NIMFLAGS="-d:insecure" to Make
2020-05-27 22:35:40 +02:00
Zahary Karadjov 52c3ebf631 Fuzzing tests for SSZ 2020-05-20 15:51:45 +03:00
Jacek Sieka 30f84224a4 create dumpdir in beaconnode binary 2020-05-20 11:03:24 +00:00
Ștefan Talpalaru 8b4065bc19
Jenkins: support running 2 parallel jobs on the same node (#1025)
* Jenkins: support running 2 parallel jobs on the same node

* try escaping $
2020-05-14 19:34:47 +02:00
Ștefan Talpalaru 2d78523423
fix Grafana dashboards (#1006)
* fix Grafana dashboards

and refactor a couple of scripts

* start.sh: allow running standalone script
2020-05-11 18:35:53 +00:00
Jacek Sieka fb2e0ddbec
sync fixes (#1005)
* sync fixes

* fix Status message finalized info
* work around sync starting before initial status exchange
* don't fail block on deposit signature check failure (fixes #989)
* print ForkDigest and Version nicely
* dump incoming blocks
* fix crash when libp2p peer connection is closed
* update chunk size to 16 to work around missing blocks when syncing

* bump libp2p

* bump libp2p

* better deposit skip message
2020-05-11 18:08:52 +00:00
Ștefan Talpalaru a7a50824a1
more metrics (#1004) 2020-05-11 06:25:49 +00:00
Ștefan Talpalaru fa663be029
launch_local_testnet.sh: get the metrics out with "--grafana" 2020-05-06 00:02:39 +02:00
Ștefan Talpalaru 1e7a4188d2
launch_local_testnet.sh: dump logs more often 2020-05-02 00:09:04 +02:00
Ștefan Talpalaru 11a36980a7
launch_local_testnet.sh: avoid infinite waits
and tail the logs on failure
2020-05-01 20:57:12 +02:00
Ștefan Talpalaru 784c5e3d73
Jenkins: add timeouts to testnet finalisation tests (#947)
* Jenkins: add timeouts to testnet finalisation tests

- add nicer child process management to "launch_local_testnet.sh"
- correct a comment in "start.sh"

* the Jenkins worker's version of "timeout" doesn't support "-v"
2020-04-30 13:59:57 +00:00
Zahary Karadjov fada11e1e9 Address review comments
* Create a json log named "nbc.log" only when the --dev-build option
  is specified to the connect_to_testnet script.

  This option is specified for the targets "testnet0", "testnet1"
  and "schlesi-dev"
2020-04-28 13:49:30 +03:00
Zahary Karadjov 7237a370ba Implement https://github.com/status-im/nim-beacon-chain/issues/872
Please see the newly added 'schlesi-dev' Makefile target.
It demonstrates how the log level can be specified for individual topics.

Additionally, when connecting to testnets like 'schlesi' there will be
two additional log files produced in the working directory:

* json-log.txt
* text-log.txt (in the textblocks format)
2020-04-28 13:49:30 +03:00
Zahary Karadjov 80b538452e Updated the custom preset loader to the latest spec; Working makefile target for connecting to the Schlesi testnet 2020-04-27 23:22:54 +03:00
Zahary Karadjov f0b1fffefc
Fix the connect_to_testnet script 2020-04-27 00:06:22 +03:00
Zahary Karadjov 740b76d152 Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim
BEWARE! This commit will trigger a stack overflow during local sim
2020-04-26 13:04:53 +03:00
Ștefan Talpalaru 39a893ea90
use the right preset in "make testnet{0,1}" [skip ci] (#932)
...and stop pretending to handle the preset in "connect_to_testnet.nims".
2020-04-24 17:28:03 +02:00
tersec 8a72ae89b9
fix mainnet finalization (#906)
* fix mainnet finalization and swith eth2_network_simulation to a kind of small-mainnet profile

* Fix slot reference in trace logging

* bump a couple of spec refs from v0.11.0 to v0.11.1

* bump another spec ref to v0.11.1, one more try at Jenkins test vector download CI issue

* fix other slot reference in trace logging and skip past single-block/multi-slot gaps to re-approach from ancestry side by state_transitioning, by requiring exact match on both root hash and slot for fast path

* make more precise the fast path condition

* redo logic to make uniform with BeaconChainDB; fix chronos deprecation warning

* revert not-working replacement of deprecated chronos futures `or`

* switch testnet1 to mainnet
2020-04-20 19:27:52 +02:00
Ștefan Talpalaru 8af4ee9dbc connect_to_testnet.nims: don't hide makeDeposits failure [skip ci] 2020-04-18 16:17:18 +03:00
Zahary Karadjov d7c36a677e Remove p2pd from the build recipes 2020-03-23 14:59:33 +02:00
Zahary Karadjov 0c018cb68a Mechanically remove all mentions of the daemon from the code 2020-03-23 14:59:33 +02:00
Zahary Karadjov d2b7ee27de The RPC service listens on localhost by default 2020-03-22 19:00:09 +02:00