* Logging and startup improvements
Color support for released binaries!
* startup scripts no longer log to file by default - this only affects
source builds - released binaries don't support file logging
* add --log-stdout option to control logging to stdout (colors, json)
* detect tty:s vs redirected logs and log accordingly
* add option to disable log colors at runtime
* simplify several "common" logs, showing the most important information
earlier and more clearly
* remove line numbers / file information / tid - these take up space and
are of little use to end users
* still enabled in debug builds and tools
* remove `testnet_servers_image` compile-time option
* server images, released binaries and compile-from-source now offer
the same behaviour and features
* fixes https://github.com/status-im/nimbus-eth2/issues/2326
* fixes https://github.com/status-im/nimbus-eth2/issues/1794
* remove instanteneous block speed from sync message, keeping only
average
before:
```
INF 2021-10-28 16:45:59.000+02:00 Slot start topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:884 lastSlot=2384027 wallSlot=2384028 delay=461us84ns peers=0 head=75a10ee5:3348 headEpoch=104 finalized=cd6804ba:3264 finalizedEpoch=102 sync="wwwwwwwwww:0:0.0000:0.0000:00h00m (3348)"
INF 2021-10-28 16:45:59.046+02:00 Slot end topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:821 slot=2384028 nextSlot=2384029 head=75a10ee5:3348 headEpoch=104 finalizedHead=cd6804ba:3264 finalizedEpoch=102 nextAttestationSlot=-1 nextProposalSlot=-1 nextActionWait=n/a
```
after:
```
INF 2021-10-28 22:43:23.033+02:00 Slot start topics="beacnde" slot=2385815 epoch=74556 sync="DDPDDPUDDD:10:5.2258:01h19m (2361088)" peers=37 head=eacd2dae:2361096 finalized=73782:a4751487 delay=33ms687us715ns
INF 2021-10-28 22:43:23.291+02:00 Slot end topics="beacnde" slot=2385815 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 head=eacd2dae:2361096
```
* fix comment
* documentation updates
* mention `--log-file` may be deprecated in the future
* update various docs
Tests have been moved to the Makefile and the test binaries are built in
parallel.
The Nim compilation scheme has been moved to a script and adapted to
work with different binaries being built in parallel from the same main
source file (think minimal/mainnet tests).
* use LTO in release builds
This significantly (40%) speeds up block replay and hashing - for example replaying first 1000
blocks, without/with LTO:
```
[arnetheduck@tempus ncli]$ ../env.sh nim c -d:release ncli_db
[arnetheduck@tempus ncli]$ ./ncli_db bench --db:db --network:medalla --slots:1000
Loaded 215006 blocks, head slot 307400
All time are ms
Average, StdDev, Min, Max, Samples, Test
Validation is turned off meaning that no BLS operations are performed
25468.481, 0.000, 25468.481, 25468.481, 1, Initialize DB
0.297, 0.516, 0.053, 13.645, 721, Load block from database
26.458, 0.000, 26.458, 26.458, 1, Load state from database
20.737, 8.288, 11.096, 199.325, 690, Apply block
333.069, 62.798, 45.225, 429.452, 31, Apply epoch block
0.000, 0.000, 0.000, 0.000, 0, Database block store
```
```
[arnetheduck@tempus ncli]$ ../env.sh nim c -d:release --passc:-flto --passl:-flto --stacktrace:off ncli_db
[arnetheduck@tempus ncli]$ ./ncli_db bench --db:db --network:medalla --slots:1000
Loaded 215006 blocks, head slot 307400
All time are ms
Average, StdDev, Min, Max, Samples, Test
Validation is turned off meaning that no BLS operations are performed
23903.006, 0.000, 23903.006, 23903.006, 1, Initialize DB
0.253, 0.122, 0.047, 0.731, 721, Load block from database
24.455, 0.000, 24.455, 24.455, 1, Load state from database
18.734, 7.062, 10.346, 167.397, 690, Apply block
194.869, 33.175, 29.311, 226.981, 31, Apply epoch block
```
Epoch processing is heavy on both arithmetics and hash caching, both of which get a
significant boost here.
This makes sense: nim creates lots of small functions spread out over many C files. A much
worse solution is to try to annotate code with `inline` - it copies functions to multiple
C files but still doesn't do intermodule optimizations significantly limiting the
compilers' ability to reason about the code, causing bloat and misrepresenting the usefulness
of a function to the call frequency analysis that drives actual (C-compiler) inlining and many
other optimizations.
In particular, many nim functions are part of `system` or the `C` backend - stack tracing,
memory allocation etc - nim's inlining system is pretty incomplete in that it does not deal
with these and many other cases.
* windows workaround
* skip LTO on windows for now
* Clean up PR + bump nimbus-build-system
* pcre on 32-bit + Improve env variable handling + cache mingw
* Add badge + fix setting env variable
* Auto cancel if commit becomes outdated
* fix shell for deriving env variable
* Add more cancellation points
* Add finalization tests to Github Action
* Fix case
* change cancel actions + fixes for windows and finalization
* have to use matrix variable for cache path/key
* ARCH_OVERRIDE=LATFORM issue rebuild cache
* Update scripts - deactivate workflows with identified issues + reactivate caching
* workaround mac getopt
* Disable all aAVX512f extensions (Error: invalid register for .seh_savexmm in Cygwin)
* Fix cross compile of libminiupmp #1723
* Cache fetch-dlls to avoid being a drag on nim-lang.org
* Fix windows downloading DLLs twice and set CFLAGS env variable for Linux32
* fix silly yaml mistake
* .
* reactivate win32 after https://github.com/status-im/nim-beacon-chain/pull/1726
* Comment out minimal tests for now
* Bump BLSCurve
* Use unified aggregation API
* use new blscurve with unified aggregate API
* bump
* fix toRaw
* replace state_sim combine with AggregateSignature
* Fix 32-bit
* Fix 32-bit for real and test deactivating ccache for fno-tree-lopp-vectorize flag
* change compilation switches to narrow down Linux issue
* Use -fno-tree-vectorize to disable both tree-loop-vectorize and tree-slp-vectorize
* blscurve now disables both Loop and SLP vectorization
* Add tests for the miracl/milagro fallback
* Travis has max log size of 4MB
* Test with Miracl in the finalization test
* fix state_sim log level
* Coment out the slow fallback tests
Ștefan Talpalaru
Jacek Sieka
Zahary Karadjov
tersec
Mamy Ratsimbazafy
Eugene Kabanov