Commit Graph

239 Commits

Author SHA1 Message Date
tersec b4f9908d9d
run logtrace in finalization CI (#1512)
* run logtrace in finalization CI

* use specified DATA_DIR rather than assuming default

* convert rest of hardcoded local_testnet_data references

* logtrace asr needs DEBUG-level logs

* clean up some shell (not) quoting issues
2020-08-17 08:36:29 +02:00
tersec bc6eefe31e
add --enable-logtrace argument to launch_local_testnet (#1502)
* add --enable-logtrace argument to launch_local_testnet

* scan for all available logfiles

* remove specific filename references

* update v0.11.3 spec ref to v0.12.2
2020-08-16 11:12:19 +02:00
Mamy Ratsimbazafy 454b9d0724
Bump nim-blscurve (#1491)
* Bump BLSCurve

* Use unified aggregation API

* use new blscurve with unified aggregate API

* bump

* fix toRaw

* replace state_sim combine with AggregateSignature

* Fix 32-bit

* Fix 32-bit for real and test deactivating ccache for fno-tree-lopp-vectorize flag

* change compilation switches to narrow down Linux issue

* Use -fno-tree-vectorize to disable both tree-loop-vectorize and tree-slp-vectorize

* blscurve now disables both Loop and SLP vectorization

* Add tests for the miracl/milagro fallback

* Travis has max log size of 4MB

* Test with Miracl in the finalization test

* fix state_sim log level

* Coment out the slow fallback tests
2020-08-15 19:33:58 +02:00
tersec af3355e0f8
create local testnet mode for eth2_network (#1494) 2020-08-12 14:16:59 +00:00
Zahary Karadjov d58668157a Use consistent naming for the synonyms 'deposits-dir' and 'validators-dir' 2020-08-02 23:00:43 +03:00
Ștefan Talpalaru 6a30c4a67d
launch_local_testnet.sh: add "--reuse-existing-data-dir" (#1396)
- also unify the indentation towards the least readable denominator, due
  to ".editorconfig" making my Vim display tabs as 2 spaces wide.
2020-07-30 02:46:10 +02:00
Ștefan Talpalaru fa9f35e148
Jenkins: run local testnet test on macOS (#1391) 2020-07-29 14:08:27 +02:00
Zahary Karadjov f4c19e303a Non-interactive generation of keystores in the local sim 2020-07-28 07:36:25 +03:00
Zahary Karadjov 40ea9e9886 Integrate the Wallet support with the Deposits creation; Produce Launchpad-compatible deposits 2020-07-28 07:36:25 +03:00
tersec 20a2525390
v0.12.2 beacon chain protocol update (#1378) 2020-07-27 12:59:57 +02:00
Ștefan Talpalaru c47532f2b0
deal with a temporary loss of network connectivity (#1354)
* don't kill the program if not connected to a bootstrap node within 30 seconds

* recover faster from loss of network connectivity

* connectWorker(): sleep 1s between dials

* launch_local_testnet.sh: increase BOOTSTRAP_TIMEOUT

* don't use metric value in program logic

* refactor some ungainly variable names
2020-07-23 22:51:56 +02:00
Zahary Karadjov 5a4f6b7e5e
Restore the SSZ fuzzign tests to working state 2020-07-22 16:20:07 +03:00
Ștefan Talpalaru ffd4bc1dd4
launch_local_testnet.sh: make node 0 the bootstrap node 2020-07-16 19:04:57 +02:00
Zahary Karadjov d16676b294
Bake the metadata for testnet0 and testnet1 as well; Remove the old scripts 2020-07-10 17:39:54 +03:00
Zahary Karadjov 540b2828b2
Adapt the local sim scripts to use the new run-time presets 2020-07-10 01:08:54 +03:00
Viktor Kirilov 1482b0430d - work towards more REST API endpoints being implemented
- testnets can now be launched with a separate validator client - make altona SCRIPT_PARAMS="--separateVC"
- reverted the ctrl+C signal handler code reuse - not necessary for the VC anyway (default is good enough)
- added a bit more logging in the VC
- removed unnecessary code in the VC - connect() just parses the address & port...
- fixed a couple more VC issues - when fetching the duties for an epoch fails on the BN side ==> the VC shouldn't be left in a broken state
- documented the currently supported json-rpc endpoints
- added more checks on the BN side for the API - bounds-checking the requests & also checking if the BN itself is synced
- other cleanup

currently a local sim doesn't finalize, but participation in the altona network with a separate VC is painless and works just as well as with in-process validators in a BN
2020-07-08 13:29:03 +03:00
Ștefan Talpalaru 20ac6d46fe
connect_to_testnet.nims: --extraBeaconNodeOptions
make SCRIPT_PARAMS="--extraBeaconNodeOptions='--metrics-address=127.0.0.1 --rpc-address=127.0.0.1 --nat=any'" altona
2020-06-30 19:16:09 +02:00
cheatfate d71ce6a60a Fix names 2020-06-30 09:59:48 +03:00
cheatfate a6f359527d Allow to specify rpcAddress and metricsAddress. 2020-06-30 09:59:48 +03:00
tersec b4db2ad693
remove v0.11.3 support; add block_sim to CI (#1253)
* remove v0.11.3 support; add block_sim to CI

* rm stray PERSISTENT_COMMITTEE_PERIOD

* remove TopicFilter.InteropAttestations

* bump two comment-spec-refs to v0.12.1
2020-06-29 18:08:58 +00:00
Eugene Kabanov 7712442236
Allow nat option for connect_to_testnet script. (#1250) 2020-06-29 12:05:00 +02:00
Zahary Karadjov dbb3dafce1
Switch to a new Infura account as a band-aid 2020-06-29 12:43:01 +03:00
Zahary Karadjov 96ac81c89f Better UX; Don't delete testnet data folders 2020-06-29 02:18:48 +03:00
Ștefan Talpalaru d85ae34245
shared_testnet: validator creation [skip ci] 2020-06-28 03:51:00 +02:00
Ștefan Talpalaru 675024a989
new Docker image for shared testnets 2020-06-26 17:26:18 +02:00
Zahary Karadjov b67a506b3f
Fixes for local sim on macOS [skip ci] 2020-06-25 12:42:30 +03:00
Ștefan Talpalaru 33ad3c05fe
connect_to_testnet.nims: add "--buildOnly" and "--runOnly" opts [skip ci]
and "--writeLogFile"
2020-06-24 12:54:17 +02:00
tersec 1b3c19dc21
default to v0.12.1 beacon chain spec, keeping witti working (#1222)
* default to v0.12.1 beacon chain spec, keeping witti working

* a couple of fixes that help for the Onyx testnet
2020-06-23 19:40:44 +00:00
Zahary Karadjov eb48c05b7f Don't assign validator keys to the bootstrap node; Ganache mode for launch_local_testnet 2020-06-23 07:59:35 +00:00
Zahary Karadjov 7211a99086 Smooth out the differences between Ganache and Infura; Working local sim and Altona target 2020-06-22 17:30:04 +03:00
Zahary Karadjov 1def383ad1 Reword the Tmux setup to handle all simulation scenarios in more visible way 2020-06-22 17:30:04 +03:00
Zahary Karadjov e9d68e2f7b Add simulation mode that bootstraps from an Eth1 ganache instance 2020-06-22 17:30:04 +03:00
Zahary Karadjov 4275abbefe Makefile targets for ctail and ntu 2020-06-22 17:30:04 +03:00
Ștefan Talpalaru 3a3d3f9bde
timestamps in log files 2020-06-17 16:44:48 +02:00
Ștefan Talpalaru 673eeb6a65
connect_to_testnet.nims changes [skip ci]
- remove `--dev-build` option
- unconditionally write the "nbc.log" file, but do it after a chdir to
  dataDir because Chronicles doesn't seem to support proper paths for
  "file(...)" in sink definitions
- change base RPC port (9090 -> 9190) because 9090 is the default
  Prometheus daemon listening port
2020-06-17 16:03:34 +02:00
Jacek Sieka 8fbbd59885
metric names (#1188)
* fix metric names to not clash with native libp2p metrics
* run testnet node with rpc enabled by default
2020-06-17 13:04:24 +02:00
Ștefan Talpalaru f5ec1bc5bb
support multiple local Witti nodes [skip ci] 2020-06-11 18:42:58 +02:00
Ștefan Talpalaru 421378b92d
connect_to_testnet.nims: mkDir dataDir 2020-06-11 17:37:27 +02:00
Zahary Karadjov 5807e2a767 Handle the delayed creation of the bootstrap node address file 2020-06-11 17:40:08 +03:00
Zahary Karadjov ff49932bb9 Reduce the number of validators to their older values (before this branch) 2020-06-11 17:40:08 +03:00
Zahary Karadjov bea243ae04 Attempt to fix the CI finalization test
Since I'm not able to reproduce the finalization failure locally
and it does happen only sporadically, one possible explanation is
that the introduction of keystores lead to a slower initialization
of the beacon nodes which somehow interferes with their behavior
during the initial slots.

If increasing the start-up delay fixes the problems, the hypothesis
will be confirmed.
2020-06-11 17:40:08 +03:00
Zahary Karadjov 2acda1c115 Provide a default value for secretsDir (similar to validatorsDir) 2020-06-11 17:40:08 +03:00
Zahary Karadjov a75c632f7a Fixed launch_local_testnet; Renamed validator_keygen to keystore_directories 2020-06-11 17:40:08 +03:00
Zahary Karadjov e3378e52ac Distribute the keystore secret files to the testnet hosts 2020-06-11 17:40:08 +03:00
Zahary Karadjov a8113cf2bc Restore the local sim to a working state 2020-06-11 17:40:08 +03:00
Zahary Karadjov 17343442ea Implement more of the KeyStore spec and integrate it in the beacon node 2020-06-11 17:40:08 +03:00
Ștefan Talpalaru 8e648da399
connect_to_testnet.nims: don't die when GNU getopts is missing 2020-06-11 04:11:30 +02:00
Ștefan Talpalaru e2025c5752
Prometheus & Grafana refactoring
- moved "process_dashboard.nim" in "tools/"
- README: made Witti the documented testnet and added instructions for
  getting metrics out of the local node
- moved Prometheus config file generation in its own script
- the static Grafana dashboard definition now covers all nodes, using
  a variable; only the remote testnet dashboards need to be dynamically
  generated
- "launch_local_testnet.sh" no longer needs a "--grafana" option
2020-06-10 19:55:26 +02:00
Ștefan Talpalaru 7c7b6fb361
Grafana: only show one node per page [skip ci]
by using Grafana variables to get dropdown menus
2020-06-07 01:42:54 +02:00
Zahary Karadjov 0c78fc39e7
Use the latest LibP2P 2020-06-05 19:34:12 +03:00
Ștefan Talpalaru 3a6a9f8135 generate Grafana dashboards for remote testnet nodes 2020-06-03 12:44:16 +03:00
Zahary Karadjov 0bcdabfcdf Detect another invalid SSZ input found through fuzzing
The first offset of an SSZ object should always have a fixed constant
value. Otherwise, some unused bytes may appear between the fixed portion
and the dynamic portion.

Please note that this fix shutds down the minimal forward compatibility
currently supported by the SSZ format (and thus, the expected behavior
must be clarified in the SSZ spec).
2020-05-30 14:47:35 +03:00
Ștefan Talpalaru b0a7cb17eb
connect_to_testnet.nims: enable metrics [skip ci]
you still need to pass NIMFLAGS="-d:insecure" to Make
2020-05-27 22:35:40 +02:00
Zahary Karadjov 52c3ebf631 Fuzzing tests for SSZ 2020-05-20 15:51:45 +03:00
Jacek Sieka 30f84224a4 create dumpdir in beaconnode binary 2020-05-20 11:03:24 +00:00
Ștefan Talpalaru 8b4065bc19
Jenkins: support running 2 parallel jobs on the same node (#1025)
* Jenkins: support running 2 parallel jobs on the same node

* try escaping $
2020-05-14 19:34:47 +02:00
Ștefan Talpalaru 2d78523423
fix Grafana dashboards (#1006)
* fix Grafana dashboards

and refactor a couple of scripts

* start.sh: allow running standalone script
2020-05-11 18:35:53 +00:00
Jacek Sieka fb2e0ddbec
sync fixes (#1005)
* sync fixes

* fix Status message finalized info
* work around sync starting before initial status exchange
* don't fail block on deposit signature check failure (fixes #989)
* print ForkDigest and Version nicely
* dump incoming blocks
* fix crash when libp2p peer connection is closed
* update chunk size to 16 to work around missing blocks when syncing

* bump libp2p

* bump libp2p

* better deposit skip message
2020-05-11 18:08:52 +00:00
Ștefan Talpalaru a7a50824a1
more metrics (#1004) 2020-05-11 06:25:49 +00:00
Ștefan Talpalaru fa663be029
launch_local_testnet.sh: get the metrics out with "--grafana" 2020-05-06 00:02:39 +02:00
Ștefan Talpalaru 1e7a4188d2
launch_local_testnet.sh: dump logs more often 2020-05-02 00:09:04 +02:00
Ștefan Talpalaru 11a36980a7
launch_local_testnet.sh: avoid infinite waits
and tail the logs on failure
2020-05-01 20:57:12 +02:00
Ștefan Talpalaru 784c5e3d73
Jenkins: add timeouts to testnet finalisation tests (#947)
* Jenkins: add timeouts to testnet finalisation tests

- add nicer child process management to "launch_local_testnet.sh"
- correct a comment in "start.sh"

* the Jenkins worker's version of "timeout" doesn't support "-v"
2020-04-30 13:59:57 +00:00
Zahary Karadjov fada11e1e9 Address review comments
* Create a json log named "nbc.log" only when the --dev-build option
  is specified to the connect_to_testnet script.

  This option is specified for the targets "testnet0", "testnet1"
  and "schlesi-dev"
2020-04-28 13:49:30 +03:00
Zahary Karadjov 7237a370ba Implement https://github.com/status-im/nim-beacon-chain/issues/872
Please see the newly added 'schlesi-dev' Makefile target.
It demonstrates how the log level can be specified for individual topics.

Additionally, when connecting to testnets like 'schlesi' there will be
two additional log files produced in the working directory:

* json-log.txt
* text-log.txt (in the textblocks format)
2020-04-28 13:49:30 +03:00
Zahary Karadjov 80b538452e Updated the custom preset loader to the latest spec; Working makefile target for connecting to the Schlesi testnet 2020-04-27 23:22:54 +03:00
Zahary Karadjov f0b1fffefc
Fix the connect_to_testnet script 2020-04-27 00:06:22 +03:00
Zahary Karadjov 740b76d152 Spec-compliant implementation of Eth1 monitoring; Eth1-enabled local sim
BEWARE! This commit will trigger a stack overflow during local sim
2020-04-26 13:04:53 +03:00
Ștefan Talpalaru 39a893ea90
use the right preset in "make testnet{0,1}" [skip ci] (#932)
...and stop pretending to handle the preset in "connect_to_testnet.nims".
2020-04-24 17:28:03 +02:00
tersec 8a72ae89b9
fix mainnet finalization (#906)
* fix mainnet finalization and swith eth2_network_simulation to a kind of small-mainnet profile

* Fix slot reference in trace logging

* bump a couple of spec refs from v0.11.0 to v0.11.1

* bump another spec ref to v0.11.1, one more try at Jenkins test vector download CI issue

* fix other slot reference in trace logging and skip past single-block/multi-slot gaps to re-approach from ancestry side by state_transitioning, by requiring exact match on both root hash and slot for fast path

* make more precise the fast path condition

* redo logic to make uniform with BeaconChainDB; fix chronos deprecation warning

* revert not-working replacement of deprecated chronos futures `or`

* switch testnet1 to mainnet
2020-04-20 19:27:52 +02:00
Ștefan Talpalaru 8af4ee9dbc connect_to_testnet.nims: don't hide makeDeposits failure [skip ci] 2020-04-18 16:17:18 +03:00
Zahary Karadjov d7c36a677e Remove p2pd from the build recipes 2020-03-23 14:59:33 +02:00
Zahary Karadjov 0c018cb68a Mechanically remove all mentions of the daemon from the code 2020-03-23 14:59:33 +02:00
Zahary Karadjov d2b7ee27de The RPC service listens on localhost by default 2020-03-22 19:00:09 +02:00
kdeme 10d9042893 Specifically set extip to loopback address for local testnet 2020-03-11 16:05:11 +00:00
Ștefan Talpalaru 929fcf0036
eth2_network_simulation: propagate LOG_LEVEL [skip ci]
also add a "--log-level" option to launch_local_testnet.sh
2020-03-09 01:37:57 +01:00
Ștefan Talpalaru c2c570fcb0
bump submodules
plus a few small modifications
2020-02-26 01:14:20 +01:00
Ștefan Talpalaru 0b05fecedc
make SCRIPT_PARAMS="--skipGoerliKey" testnet1 2020-02-25 02:49:47 +01:00
Ștefan Talpalaru 8b8a99135f
bump vendor/nim-eth
and improve htop processes in launch_local_testnet.sh
2020-02-23 01:56:29 +01:00
Ștefan Talpalaru bd194da3d1
launch_local_testnet.sh: copy validator keys to nodes 2020-02-21 14:19:34 +01:00
Ștefan Talpalaru f146b71197
testnet finalisation testing in CI (disabled) 2020-02-19 03:26:56 +01:00
Mamy André-Ratsimbazafy 37446302b2 Use the new nim-eth2-scenarios path for the test vectors 2020-02-17 15:26:11 +00:00
Ștefan Talpalaru 06322385d9
lightweight stack traces
- plus some light test runner refactoring and some Makefile cosmetic changes
- compile tools with LOG_LEVEL=TRACE in CI
- bump a couple of submodules
2020-02-13 17:54:53 +01:00
Ștefan Talpalaru 82447e22b2
cleanup [skip ci] 2020-02-11 18:43:57 +01:00
Ștefan Talpalaru 8df447d497
Docker: optimise container beacon_node for server CPU [skip ci]
This means that we can no longer do "docker run" locally, so we compile
a local beacon_node with the same flags as the container one.
2020-02-11 18:41:25 +01:00
Ștefan Talpalaru d78a7f4a6a
Merge branch 'runtimeloglevel' into devel 2020-02-09 03:38:14 +01:00
Ștefan Talpalaru 6007b01d47
testnet: don't use -march=native when building the Docker image [skip ci]
(because we run that Docker image both locally and remotely, we can't
use the server's "-march" either)
- reduce the genesis offset to 5 minutes
2020-02-09 03:22:42 +01:00
Stefan Talpalaru 7feaa11372
make testnetX: propagate LOG_LEVEL to beacon_node runtime [skip ci]
This allows running `make LOG_LEVEL=TRACE testnet1` and having that log
level also enabled at runtime, not just at compile time.
2020-02-08 00:57:48 +01:00
Jacek Sieka 521b0ed6ba
Clean 20200205 (#729)
* beacon node code cleanup
* rudimentary error checking on mainnet monitor
* start client even when sending deposit
* work around missing block number exception
* connect to testnet with web3 url
* pretty-print digests in json
2020-02-07 08:13:38 +01:00
Stefan Talpalaru a37aa3b86d `make testnet1`: switch the local node to the Nim libp2p
so we can debug it without having to also consider Go-Nim libp2p
interoperability issues
2020-02-06 13:36:28 +01:00
Zahary Karadjov 0d9503ee49
Allow run-time switching to the TRACE log level 2020-02-03 17:06:35 +01:00
Zahary Karadjov 2cb1cc69ba Fix shell script error triggered by braces in testnet names 2020-01-29 10:10:28 +02:00
Zahary Karadjov b1c247fa4a
Change the testnet servers domain names 2020-01-28 22:28:52 +02:00
Zahary Karadjov 7781536576
Handle the option of not resetting the network properly 2020-01-22 21:11:15 +02:00
Ștefan Talpalaru 7e36ba4f4e launch_local_testnet.sh 2020-01-22 13:44:58 +00:00
Zahary Karadjov c65575d105 Improved logging related to bootstrap nodes 2020-01-21 01:26:57 +02:00
Zahary Karadjov 4848b12050 Detect and use Lighthouse's boot_enr.yaml files 2020-01-21 01:26:57 +02:00
Zahary Karadjov 095b271bd4 Implement a loader for custom const presets
The loader has been tested with the presets published by Lighthouse.
You can try connecting to one of their testnets by running:

cd nim-beacon-chain
./connect-to-testnet lighthouse/testnet0
2020-01-21 01:26:57 +02:00
Ștefan Talpalaru 13b9f61f6f
reset_testnet.sh: `make build` moved earlier [skip ci] 2020-01-13 14:48:19 +01:00
Ștefan Talpalaru 05f0950f41
cosmetic change [skip ci] 2020-01-13 14:40:40 +01:00