Commit Graph

100 Commits

Author SHA1 Message Date
tersec c71bddb612
try not requiring brew install gnu-getopt (#4569) 2023-01-31 15:22:57 +02:00
Zahary Karadjov daa99f43c3
Merge branch 'stable' into unstable 2023-01-26 16:47:26 +01:00
tersec 6818120849
drop Nim 1.2 testing from GitHub Actions CI (#4525) 2023-01-19 17:55:39 +00:00
Zahary Karadjov d40da67eef
Merge branch 'stable' into unstable 2023-01-18 19:42:20 +02:00
tersec 9b0868033e
use GitHub Actions cache and upload-artifact v3 (#4516) 2023-01-17 15:45:04 +00:00
tersec 7ad0d3e6c2
attempt to reduce GitHub Actions runner disk usage (#4511) 2023-01-15 08:43:50 +00:00
tersec 6ccf13e762
log Windows GitHub Action runner partition sizes (#4510) 2023-01-14 21:48:01 +00:00
tersec d2e9cdf1a0
show number of cores and total physical memory on Windows GitHub Actions (#4504) 2023-01-13 21:42:08 +00:00
tersec e09904adab
Revert "reduce intermittent Windows GitHub Actions CI build failures (#4495)" (#4502)
This reverts commit 3b60b225b3.
2023-01-13 13:22:35 +00:00
tersec 3b60b225b3
reduce intermittent Windows GitHub Actions CI build failures (#4495) 2023-01-12 14:47:33 +00:00
Zahary Karadjov 5025990d9e
Publish multiarch images for the nimbus-validator-client 2022-12-21 19:37:17 +02:00
Zahary Karadjov 3be3e15d85
Temporarily disable the macOS-AMD64 packaging 2022-12-21 10:53:17 +02:00
zah 0be57eec6e
Don't package the nimbus_validator_client in the nimbus-eth2 docker images; Document the new dedicated image (#4441) 2022-12-20 23:32:31 +02:00
zah bf50e5af54
Publish official docker images for the Nimbus validator client (#4439)
Fixes https://github.com/status-im/nimbus-eth2/issues/4426
2022-12-19 16:49:25 +00:00
Jacek Sieka f3305189f1
bump & ci fixes (#4409) 2022-12-09 17:06:29 +01:00
Miran 77d28f43f6
remove i386 windows from daily CI (#4150)
This was failing *forever* (yes, literally).
As discussed on Discord, i386 Windows is EOL,
there's no reason to run it.
2022-09-19 21:33:19 +00:00
Miran 5a4c2d0a2b
move i386 testing to nightlies + remove testing of Nim 1.4 (#4103)
* remove i386 target from ci.yml

* add i386 build to nightlies.yml

* don't continue on error for Nim 1.6

* do not test Nim 1.4
2022-09-17 08:38:18 +03:00
Jacek Sieka ff823f4aae
avoid building main binaries twice (#4092)
we can just build everything with trace logging
2022-09-08 11:13:54 +02:00
tersec b32ab88191
switch GitHub Actions CI images from Ubuntu 18.04 to 20.04 (#3988) 2022-08-18 21:07:57 +03:00
Jacek Sieka 255be39e69
packaging updates (#3974)
* packaging updates

* one package per binary (nimbus_beacon_node, nimbus_validator_client)
* use `-` in package name (`_` is separating the version)
* don't include (un)installation scripts in package
* default metrics port 8108 for vc
* fix several upgrade/install errors in scripts
* add JWT option to service files
* don't attempt to remove user on purge
2022-08-17 13:26:31 +03:00
Nikolay Mitev d07d7b1c48 Include auxiliary tools to nightly builds
Create stable link (withouth hash) so that scripts can always get the latest nightly build from the same url
2022-08-11 17:57:44 +03:00
Zahary Karadjov 9b081e524c
Merge branch 'stable' into unstable 2022-07-29 11:28:43 +03:00
Zahary Karadjov 75ccfcb26f
Ship the Validator Client (Currently in BETA) 2022-07-27 20:16:02 +03:00
tersec 615f366dcc
update to supported macOS GitHub CI image (#3898) 2022-07-22 07:08:21 +00:00
zah c01341851d
Backticks in Github Actions YAML need escaping 2022-06-28 23:56:17 +03:00
Jacek Sieka 0152f4ade5
Make deb/rpm packages when creating release (#3796)
* Make deb/rpm packages when creating release

A step towards #3782

* fix arch for arm
2022-06-28 23:34:18 +03:00
Zahary Karadjov f5ded5590e
Add the urgency guidelines automatically to each release 2022-06-21 00:08:42 +03:00
Jacek Sieka 099709872e
Merge branch 'stable' into unstable 2022-06-14 13:35:38 +02:00
Jacek Sieka 38b6f7263d
ci: install cmake on windows in daily job
it's been failing since forever
2022-06-14 13:35:17 +02:00
tersec 6516ad43eb
remove obsolete beta altair github workflow (#3731) 2022-06-10 23:43:37 +00:00
tersec 09663e00d8
don't hide Nim 1.6 build failures (#3685) 2022-06-01 00:57:42 +00:00
Jacek Sieka 29b69129dc
enable hard failures for nim 1.6 in CI (#3642)
makes errors more visible, now that it's sort of working again
2022-05-19 05:07:33 +03:00
tersec 287a9c5f51
bump nim-stew to remove pragma disabling checks (#3555)
* bump nim-stew to remove pragma disabling checks

* use V=1 to try to capture the build VM overflow stack trace
2022-04-12 12:07:06 +02:00
Jacek Sieka 4207b127f9
era: load blocks and states (#3394)
* era: load blocks and states

Era files contain finalized history and can be thought of as an
alternative source for block and state data that allows clients to avoid
syncing this information from the P2P network - the P2P network is then
used to "top up" the client with the most recent data. They can be
freely shared in the community via whatever means (http, torrent, etc)
and serve as a permanent cold store of consensus data (and, after the
merge, execution data) for history buffs and bean counters alike.

This PR gently introduces support for loading blocks and states in two
cases: block requests from rest/p2p and frontfilling when doing
checkpoint sync.

The era files are used as a secondary source if the information is not
found in the database - compared to the database, there are a few key
differences:

* the database stores the block indexed by block root while the era file
indexes by slot - the former is used only in rest, while the latter is
used both by p2p and rest.
* when loading blocks from era files, the root is no longer trivially
available - if it is needed, it must either be computed (slow) or cached
(messy) - the good news is that for p2p requests, it is not needed
* in era files, "framed" snappy encoding is used while in the database
we store unframed snappy - for p2p2 requests, the latter requires
recompression while the former could avoid it
* front-filling is the process of using era files to replace backfilling
- in theory this front-filling could happen from any block and
front-fills with gaps could also be entertained, but our backfilling
algorithm cannot take advantage of this because there's no (simple) way
to tell it to "skip" a range.
* front-filling, as implemented, is a bit slow (10s to load mainnet): we
load the full BeaconState for every era to grab the roots of the blocks
- it would be better to partially load the state - as such, it would
also be good to be able to partially decompress snappy blobs
* lookups from REST via root are served by first looking up a block
summary in the database, then using the slot to load the block data from
the era file - however, there needs to be an option to create the
summary table from era files to fully support historical queries

To test this, `ncli_db` has an era file exporter: the files it creates
should be placed in an `era` folder next to `db` in the data directory.
What's interesting in particular about this setup is that `db` remains
as the source of truth for security purposes - it stores the latest
synced head root which in turn determines where a node "starts" its
consensus participation - the era directory however can be freely shared
between nodes / people without any (significant) security implications,
assuming the era files are consistent / not broken.

There's lots of future improvements to be had:

* we can drop the in-memory `BlockRef` index almost entirely - at this
point, resident memory usage of Nimbus should drop to a cool 500-600 mb
* we could serve era files via REST trivially: this would drop backfill
times to whatever time it takes to download the files - unlike the
current implementation that downloads block by block, downloading an era
at a time almost entirely cuts out request overhead
* we can "reasonably" recreate detailed state history from almost any
point in time, turning an O(slot) process into O(1) effectively - we'll
still need caches and indices to do this with sufficient efficiency for
the rest api, but at least it cuts the whole process down to minutes
instead of hours, for arbitrary points in time

* CI: ignore failures with Nim-1.6 (temporary)

* test fixes

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2022-03-23 09:58:17 +01:00
Ștefan Talpalaru a130ce1eac
Windows: give up stack traces for 40% better performance (#3466)
See: https://github.com/status-im/nimbus-eth2/issues/3453

TODO: make libbacktrace/libunwind work in MSYS2, to get those stack traces back.
2022-03-05 16:40:08 +02:00
Ștefan Talpalaru ebba093362
Nim-1.6 compatibility (#3434) 2022-02-25 10:19:12 +02:00
Jacek Sieka 87e98b9e54
Revert "bump submodules (#3366)" (#3406)
This reverts commit 6e1ad080e8.
2022-02-17 12:50:37 +01:00
Ștefan Talpalaru 6e1ad080e8
bump submodules (#3366)
and add Nim-1.6 compatibility
2022-02-16 13:41:50 +02:00
Ștefan Talpalaru 1306aca1ff
cron.yml: show Nim version 2021-12-15 20:43:17 +01:00
Mamy Ratsimbazafy 97da6e1365
Fork choice EF consensus tests (#3041)
* add EF fork choice tests to CI

* checkpoints

* compilation fixes and add test to preset dependent suite

* support longpaths on Windows CI

* skip minimal tests (long paths issue + impl detals tested)

* fix stackoverflow on some platforms

* rebase on top of https://github.com/status-im/nimbus-eth2/pull/3054

* fix stack usage
2021-11-25 19:41:39 +01:00
Ștefan Talpalaru 74f03ddadb
CI: statically check stack size on Linux (#3055) 2021-11-05 18:11:25 +01:00
Jacek Sieka 233d756518
Logging and startup improvements (#3038)
* Logging and startup improvements

Color support for released binaries!

* startup scripts no longer log to file by default - this only affects
source builds - released binaries don't support file logging
* add --log-stdout option to control logging to stdout (colors, json)
* detect tty:s vs redirected logs and log accordingly
* add option to disable log colors at runtime
* simplify several "common" logs, showing the most important information
earlier and more clearly
* remove line numbers / file information / tid - these take up space and
are of little use to end users
  * still enabled in debug builds and tools
* remove `testnet_servers_image` compile-time option
* server images, released binaries and compile-from-source now offer
the same behaviour and features
* fixes https://github.com/status-im/nimbus-eth2/issues/2326
* fixes https://github.com/status-im/nimbus-eth2/issues/1794
* remove instanteneous block speed from sync message, keeping only
average

before:

```
INF 2021-10-28 16:45:59.000+02:00 Slot start                                 topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:884 lastSlot=2384027 wallSlot=2384028 delay=461us84ns peers=0 head=75a10ee5:3348 headEpoch=104 finalized=cd6804ba:3264 finalizedEpoch=102 sync="wwwwwwwwww:0:0.0000:0.0000:00h00m (3348)"
INF 2021-10-28 16:45:59.046+02:00 Slot end                                   topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:821 slot=2384028 nextSlot=2384029 head=75a10ee5:3348 headEpoch=104 finalizedHead=cd6804ba:3264 finalizedEpoch=102 nextAttestationSlot=-1 nextProposalSlot=-1 nextActionWait=n/a
```

after:

```
INF 2021-10-28 22:43:23.033+02:00 Slot start                                 topics="beacnde" slot=2385815 epoch=74556 sync="DDPDDPUDDD:10:5.2258:01h19m (2361088)" peers=37 head=eacd2dae:2361096 finalized=73782:a4751487 delay=33ms687us715ns
INF 2021-10-28 22:43:23.291+02:00 Slot end                                   topics="beacnde" slot=2385815 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 head=eacd2dae:2361096
```

* fix comment

* documentation updates

* mention `--log-file` may be deprecated in the future
* update various docs
2021-11-02 18:06:36 +01:00
Etan Kissling fdbfe19b3e
fix CI for contributions from forked repos (#3028)
This updates the publish-unit-test-result integration with support for
pull requests coming from forked repositories. The update changes the
behaviour so that (a) unit test results are displayed as a comment, and
(b) macOS and Windows builds no longer time out during the CI action.
The official guide for the publish-unit-test-result action has been
followed for this purpose. See the official readme at:
https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.23/README.md#support-fork-repositories-and-dependabot-branches
2021-10-27 00:49:35 +02:00
Zahary Karadjov 9e8081e405 Merge branch 'stable' into unstable 2021-10-21 04:13:23 +03:00
Ștefan Talpalaru 8f815074a1
CI cron.yml: test Nim "version-1-6" branch (#2850) 2021-10-19 19:46:49 +02:00
Jacek Sieka 0c4bfb1911
test fixture cleanups (#2976)
* "official" -> "scenarios", like the submodule
* fewer test binaries - various compile hacks have been improved over
time, test suite should follow
* remove obsolete bls tests - there are better test vectors in
nim-blscurve
* remove obsolete mentions of `ssz_testing`
* remove obsolete comments about proc vs globals, unittest2 already uses
proc's
2021-10-12 13:36:52 +02:00
Jacek Sieka eb706f89a5
Merge remote-tracking branch 'origin/stable' into unstable 2021-10-01 16:55:50 +02:00
Ștefan Talpalaru 12ba76dd05
CI: Altair build fix 2021-08-30 13:30:58 +02:00
Zahary Karadjov a48e8d4683
Build the Altair beta from the unstable branch 2021-08-30 11:12:23 +03:00
Ștefan Talpalaru 9caf852dd4
cron CI: various fixes (#2594) 2021-08-18 14:40:03 +02:00