Commit Graph

231 Commits

Author SHA1 Message Date
Zahary Karadjov def69e2a06
Revert "More sparse state snapshots in the Gnosis network"
This reverts commit 557717b517.
2022-04-11 13:56:42 +03:00
Zahary Karadjov 557717b517
More sparse state snapshots in the Gnosis network 2022-04-09 18:07:36 +03:00
Jacek Sieka d1a8daec7e
`gnosis-chain` -> `gnosis` (#3577)
* `gnosis-chain` -> `gnosis`

Use same name as LH/Teku throughout

* fixes #3504
* fixes large stack temporary that can cause crashes during genesis
detection
2022-04-08 23:11:37 +03:00
Jacek Sieka 6f4fa32c1d
remove bundled pyrmont support (#3568)
The `pyrmont` testnet has been discontinued.

For experiments, it's still possible to run pyrmont nodes by passing a
genesis/config, but this PR removes the bundled `--network:pyrmont`
option.

* update docs
* remove empty docs
* remove obsolete `eth2-stats` page
2022-04-05 12:59:25 +00:00
Ștefan Talpalaru 725692544e
Makefile: fix "gnosis-chain-build" (#3503) 2022-03-15 14:56:01 +01:00
Etan Kissling 89ac586bd4
serve light client data in CI / dev builds (#3487)
Adjust config for CI / dev builds to serve light client data by default:
`--serve-light-client-data=1 --import-light-client-data=only-new`
2022-03-12 22:12:18 +01:00
Ștefan Talpalaru a130ce1eac
Windows: give up stack traces for 40% better performance (#3466)
See: https://github.com/status-im/nimbus-eth2/issues/3453

TODO: make libbacktrace/libunwind work in MSYS2, to get those stack traces back.
2022-03-05 16:40:08 +02:00
Zahary Karadjov 3795d57d30
PKG package for macOS 2022-03-05 15:53:51 +02:00
Zahary Karadjov ec07d3364b
MSI installer for Windows based on the WIX toolkit 2022-03-05 15:53:51 +02:00
zah c29aa9d846
Support for Gnosis Chain (#3415)
* Support for Gnosis Chain

`make gnosis-chain-build` will build the Nimbus gnosis chain binary,
stored in `build/nimbus_beacon_node_for_gnosis_chain`.

`make gnosis-chain` will connect to the network.

Other changes:

* Restore compilation with -d:has_genesis_detection
* Removed Makefile target related to testnet0 and testnet1
* Added more debug logging for failed peer handshakes
* Report misconfigured builds which try to embed network metadata
that is incompatible with the currently selected const preset.
* Don't bundle network metadata in minimal builds, as they are not compatible
2022-02-25 10:22:44 +02:00
Ștefan Talpalaru 555ce310cc
Jenkins: get rid of macOS warnings (#3412)
* Jenkins: get rid of macOS warnings

We're linking object files with debug info with ones without it. Perfectly justified, from our point of view, but dsymutil complains verbosely about it, with no switch to silence it.
2022-02-18 15:30:43 +01:00
tersec 2160d02977
use REST for make pyrmont/prater; use --rest options for local testnet (#3380) 2022-02-12 12:27:43 +01:00
Jacek Sieka c7e92bfd84
wss_sim: state transition simulator (#3309)
It's sometimes useful to simulate what happens when a chain runs from a
given state with a given set of private keys - `wss_sim` allows running
such a simulation.

One use of such a tool is to simulate a weak subjectivity attack,
creating alternative histories of the same chain:
https://notes.status.im/nimbus-insecura-network#
2022-01-22 10:25:30 +01:00
Zahary Karadjov 47f1f7ff1a More efficient reward data persistance; Address review comments
The new format is based on compressed CSV files in two channels:

* Detailed per-epoch data
* Aggregated "daily" summaries

The use of append-only CSV file speeds up significantly the epoch
processing speed during data generation. The use of compression
results in smaller storage requirements overall. The use of the
aggregated files has a very minor cost in both CPU and storage,
but leads to near interactive speed for report generation.

Other changes:

- Implemented support for graceful shut downs to avoid corrupting
  the saved files.

- Fixed a memory leak caused by lacking `StateCache` clean up on each
  iteration.

- Addressed review comments

- Moved the rewards and penalties calculation code in a separate module

Required invasive changes to existing modules:

- The `data` field of the `KeyedBlockRef` type is made public to be used
  by the validator rewards monitor's Chain DAG update procedure.

- The `getForkedBlock` procedure from the `blockchain_dag.nim` module
  is made public to be used by the validator rewards monitor's Chain DAG
  update procedure.
2022-01-18 01:56:56 +02:00
Jacek Sieka 12429a71c5
remove benchmarking code (#3233)
obsoleted by other benchmarks, no longer compiles
2022-01-03 16:31:35 +01:00
Jacek Sieka 1a87e60f7a
remove unnecessary binaries from builds (#3179)
* `fork_choice` and `proto_array` are already covered by better tests
* `nimbus_signing_node` is not used by local testnet
2021-12-10 11:56:51 +01:00
Jacek Sieka cb3f1fd4fc
remove nbench (#3152)
Used during initial development of the spec, `nbench` has fallen behind
and by and large been superceded by `block_sim`, `state_sim` and
`ncli_db bench`.
2021-12-03 15:46:56 +01:00
Eugene Kabanov e62c7c7c37
Remote signing client/server. (#3077) 2021-11-30 03:20:21 +02:00
Jacek Sieka 0c4bfb1911
test fixture cleanups (#2976)
* "official" -> "scenarios", like the submodule
* fewer test binaries - various compile hacks have been improved over
time, test suite should follow
* remove obsolete bls tests - there are better test vectors in
nim-blscurve
* remove obsolete mentions of `ssz_testing`
* remove obsolete comments about proc vs globals, unittest2 already uses
proc's
2021-10-12 13:36:52 +02:00
Ștefan Talpalaru 1c50cd224e
Grafana: add 3 more panels (#2946)
(and delete old tool)
2021-10-04 12:04:33 +02:00
Ștefan Talpalaru b67e43e9f1
remove unused Docker test (#2945) 2021-10-04 08:32:29 +00:00
Ștefan Talpalaru 9b2b3bf1ab
enable libp2p agent metrics (#2933) 2021-10-01 16:15:08 +02:00
Etan Kissling fa3d7fa086
fix build output color on macOS (#2857)
On macOS, `echo` does not support the `\e` extension from GNU coreutils.
Replacing with the portable `\x1B` to fix Terminal build output colors.
2021-09-09 14:14:19 +02:00
Jacek Sieka a7a65bce42
disentangle eth2 types from the ssz library (#2785)
* reorganize ssz dependencies

This PR continues the work in
https://github.com/status-im/nimbus-eth2/pull/2646,
https://github.com/status-im/nimbus-eth2/pull/2779 as well as past
issues with serialization and type, to disentangle SSZ from eth2 and at
the same time simplify imports and exports with a structured approach.

The principal idea here is that when a library wants to introduce SSZ
support, they do so via 3 files:

* `ssz_codecs` which imports and reexports `codecs` - this covers the
basic byte conversions and ensures no overloads get lost
* `xxx_merkleization` imports and exports `merkleization` to specialize
and get access to `hash_tree_root` and friends
* `xxx_ssz_serialization` imports and exports `ssz_serialization` to
specialize ssz for a specific library

Those that need to interact with SSZ always import the `xxx_` versions
of the modules and never `ssz` itself so as to keep imports simple and
safe.

This is similar to how the REST / JSON-RPC serializers are structured in
that someone wanting to serialize spec types to REST-JSON will import
`eth2_rest_serialization` and nothing else.

* split up ssz into a core library that is independendent of eth2 types
* rename `bytes_reader` to `codec` to highlight that it contains coding
and decoding of bytes and native ssz types
* remove tricky List init overload that causes compile issues
* get rid of top-level ssz import
* reenable merkleization tests
* move some "standard" json serializers to spec
* remove `ValidatorIndex` serialization for now
* remove test_ssz_merkleization
* add tests for over/underlong byte sequences
* fix broken seq[byte] test - seq[byte] is not an SSZ type

There are a few things this PR doesn't solve:

* like #2646 this PR is weak on how to handle root and other
dontSerialize fields that "sometimes" should be computed - the same
problem appears in REST / JSON-RPC etc

* Fix a build problem on macOS

* Another way to fix the macOS builds

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2021-08-18 20:57:58 +02:00
tersec f0321d2eb5
remove inspector (#2737) 2021-07-29 14:00:04 +00:00
Jacek Sieka 3f9c1fdf4e
More RuntimeConfig cleanup (#2716)
* remove from BeaconChainDB (doesn't depend on runtime config)
* eth2-testnets -> eth2-networks
* use `cfg` name throughout
2021-07-13 16:27:10 +02:00
tersec 6b07b79427
re-add minimal tests, now that const sanity checks gone (#2715) 2021-07-13 08:38:35 +02:00
Jacek Sieka 23eea197f6
Implement split preset/config support (#2710)
* Implement split preset/config support

This is the initial bulk refactor to introduce runtime config values in
a number of places, somewhat replacing the existing mechanism of loading
network metadata.

It still needs more work, this is the initial refactor that introduces
runtime configuration in some of the places that need it.

The PR changes the way presets and constants work, to match the spec. In
particular, a "preset" now refers to the compile-time configuration
while a "cfg" or "RuntimeConfig" is the dynamic part.

A single binary can support either mainnet or minimal, but not both.
Support for other presets has been removed completely (can be readded,
in case there's need).

There's a number of outstanding tasks:

* `SECONDS_PER_SLOT` still needs fixing
* loading custom runtime configs needs redoing
* checking constants against YAML file

* yeerongpilly support

`build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG`

* load fork epoch from config

* fix fork digest sent in status
* nicer error string for request failures
* fix tools

* one more

* fixup

* fixup

* fixup

* use "standard" network definition folder in local testnet

Files are loaded from their standard locations, including genesis etc,
to conform to the format used in the `eth2-networks` repo.

* fix launch scripts, allow unknown config values

* fix base config of rest test

* cleanups

* bundle mainnet config using common loader
* fix spec links and names
* only include supported preset in binary

* drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 15:01:38 +02:00
Eugene Kabanov 32702d37a3
Move REST simulation test to Jenkins. (#2688) 2021-06-30 08:57:07 +00:00
tersec ae1abf24af
add Altair support to block quarantine/clearance and block_sim (#2662)
* add Altair support to the block quarantine

* switch some spec/datatypes imports to spec/datatypes/base

* add Altair support to block_clearance

* allow runtime configuration of Altair transition slot

* enable Altair in block_sim, including in CI
2021-06-23 14:43:18 +00:00
Ștefan Talpalaru 201d9d4056
Makefile: the "+" that makes a difference 2021-06-01 15:23:45 +02:00
tersec e8c62553ec
reorganize SSZ-based tests to support phase0 and altair (#2612)
* reorganize SSZ-based tests to support phase0 and altair

* label Altair SSZ consensus object tests as such

* update test summary
2021-05-28 20:32:26 +00:00
Zahary Karadjov b7aa30adfd
Merge stable into unstable 2021-05-20 13:50:40 +03:00
Jacek Sieka ce49da6c0a
Introduce unittest2 and junit reports (#2522)
* Introduce unittest2 and junit reports

* fix XML path

* don't combine multiple CI runs

* fixup

* public combined report also

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-28 18:41:02 +02:00
Eugene Kabanov 0c3302b826
REST API test framework and tests. (#2492)
* REST API test framework and tests.

* Fix ValidatorIndex tests to properly handle int32, but not uint32 values.

* Fix tests to follow latest REST fixes.

* refactor restapi.sh

and add it to the test suite

* Fix issues.
Add delay timeout which is required.

* Fix restapi.sh script for Windows.

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-27 23:46:24 +03:00
Ștefan Talpalaru 5b32f7ca17 Nim-1.2.12 2021-04-19 10:53:58 +03:00
Zahary Karadjov 61669f269d Clarify that the deposit cretion tools are intended for testnets 2021-04-08 19:58:22 +03:00
Ștefan Talpalaru b2c4441834
book: document "logrotate" (#2390)
* book: document "logrotate"

* Update docs/the_nimbus_book/src/log-rotate.md

Co-authored-by: Sacha Saint-Leger <sacha@status.im>

* Update docs/the_nimbus_book/src/log-rotate.md

Co-authored-by: Sacha Saint-Leger <sacha@status.im>

* Update docs/the_nimbus_book/src/log-rotate.md

Co-authored-by: Sacha Saint-Leger <sacha@status.im>

* Update docs/the_nimbus_book/src/log-rotate.md

Co-authored-by: Sacha Saint-Leger <sacha@status.im>

* Update docs/the_nimbus_book/src/log-rotate.md

Co-authored-by: Sacha Saint-Leger <sacha@status.im>

* rewording

* Update docs/the_nimbus_book/src/log-rotate.md

Co-authored-by: Sacha Saint-Leger <sacha@status.im>

Co-authored-by: Sacha Saint-Leger <sacha@status.im>
2021-03-22 16:40:26 +01:00
tersec 9ff6118c28
clear nimcache on make update (#2426)
* test clearing nimcache

* don't be verbose
2021-03-17 17:29:34 +00:00
tersec b059cb42c5
increase block proposal speed with many validators (#2423)
* increase block proposal speed with many validators

* document CookedSig rationale
2021-03-17 13:35:59 +00:00
Zahary Karadjov c519c6012e
Merge branch 'stable' into unstable 2021-03-15 21:52:12 +02:00
Zahary Karadjov f3fc551ea7
Support for the prater testnet 2021-03-15 21:51:24 +02:00
Jacek Sieka f5180e0840
Revert "Nim-1.2.10" (#2411)
This reverts commit 12e15a0b8d.
2021-03-15 16:26:41 +01:00
Ștefan Talpalaru eafe694536 dist: faster builds for ARM and ARM64
We're moving from very slow QEMU-virtualised builds to very fast ones
based on cross toolchains.

- refactor jobs; add checksums to release note
- build and publish Docker images with ARM/ARM64 binaries
2021-03-15 14:38:39 +02:00
Ștefan Talpalaru 12e15a0b8d Nim-1.2.10 2021-03-15 14:29:48 +02:00
Mamy Ratsimbazafy 2f17ac7b64
Move SSZ, deposit_contracts & eth1_monitor [reorg files 3/5] (#2371)
* move deposit_contract

* Move SSZ

* fix ssz import in tests

* move also eth1_monitor

* forgot to delete the original

* fix comma [skip ci]

* Fix "make" & tools imports

* Fix import

* Fix import again

* rename deposit_contract -> eth1

* Revert ssz move to subfolder

* path fixes [skip ci]
2021-03-03 07:23:05 +01:00
Ștefan Talpalaru 786137ffb4 reinstate LOG_LEVEL
and introduce RUNTIME_LOG_LEVEL
2021-02-19 18:11:34 +02:00
Zahary Karadjov 5f924c0038 Bump modules; Compile with DEBUG logs by default 2021-02-19 08:44:13 +02:00
Ștefan Talpalaru f398720fc4
compile_nim_program.sh: bring back Nim warnings with V=0 (#2325) 2021-02-15 18:08:18 +01:00
Ștefan Talpalaru 4bb90351f6
books: sanity checks and docs (#2320) 2021-02-15 08:33:49 +01:00