Commit Graph

63 Commits

Author SHA1 Message Date
tersec 3e2c0a64df
always use --mm:refc, even under Nim 2.0 (#5273) 2023-08-08 02:04:41 +02:00
Jacek Sieka b8a32419b8
async batch verification (+40% sig verification throughput) (#5176)
* async batch verification

When batch verification is done, the main thread is blocked reducing
concurrency.

With this PR, the new thread signalling primitive in chronos is used to
offload the full batch verification process to a separate thread
allowing the main threads to continue async operations while the other
threads verify signatures.

Similar to previous behavior, the number of ongoing batch verifications
is capped to prevent runaway resource usage.

In addition to the asynchronous processing, 3 addition changes help
drive throughput:

* A loop is used for batch accumulation: this prevents a stampede of
small batches in eager mode where both the eager and the scheduled batch
runner would pick batches off the queue, prematurely picking "fresh"
batches off the queue
* An additional small wait is introduced for small batches - this helps
create slightly larger batches which make better used of the increased
concurrency
* Up to 2 batches are scheduled to the threadpool during high pressure,
reducing startup latency for the threads

Together, these changes increase attestation verification throughput
under load up to 30%.

* fixup

* Update submodules

* fix blst build issues (and a PIC warning)

* bump

---------

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2023-08-03 11:36:45 +03:00
tersec 8686991890
add optimized Linux amd64 flavor (#4914)
* add optimized Linux amd64 flavor

* copyright year
2023-05-11 13:05:52 +03:00
tersec 6458133f5c
disable conditionality of warning disabling (#4908)
* disable conditionality of warning disabling

* add copyright notice with current year

* express version constraint in nimble file

* also needs copyright year
2023-05-09 09:30:29 +00:00
henridf 974b1657b4
use nim-blscurve fork with upgraded blst (#4788) 2023-04-14 12:36:54 +00:00
henridf 635a924e8c
Add nim-kzg4844 and use it in validate_blobs (#4732) 2023-03-23 09:47:04 +00:00
tersec d454fcaa90
Use clang optimization flag for actual macOS target CPU, not iOS phone CPU (#4706) 2023-03-11 09:33:53 +02:00
tersec 2dd2a782d9
disable BareExcept warnings (#4662)
* disable BareExcept warnings

* BareExcept only exists in >= 1.6.11
2023-02-24 19:50:46 +00:00
tersec 1c62a5eb24
capella VC support (#4586) 2023-02-03 16:12:11 +01:00
Jacek Sieka 04392c236b
bumps (#4352)
* build&style fixes
* use multithreading for LTO outside of make too
* blst 0.3.10 with no significant changes
2022-11-24 20:56:02 +00:00
zah 2536db1b6a
Work-around a Nim 1.6 issue in the handling of the NBS paths file (#4253) 2022-10-21 21:48:43 +02:00
tersec 655e2fa808
only support macOS hardware supported by non-EOL macOS software (#4246) 2022-10-20 00:55:01 +03:00
zah 3f90fe8a7d
Switch to the new style of importing vendor packages (#4218) 2022-10-04 23:39:12 +00:00
tersec 52b32c13e1
update macOS baseline x86 CPU architecture to Ivy Bridge (#3929) 2022-08-02 19:02:56 +00:00
Etan Kissling 2a2bcea70d
group justified and finalized `Checkpoint` (#3841)
The justified and finalized `Checkpoint` are frequently passed around
together. This introduces a new `FinalityCheckpoint` data structure that
combines them into one.

Due to the large usage of this structure in fork choice, also took this
opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec.
Many additional tests enabled, some need more work, e.g. EL mock blocks.
Also implemented `discard_equivocations` which was skipped in #3661,
and improved code reuse across fork choice logic while at it.
2022-07-06 13:33:02 +03:00
zah f08f9e2bd7
Bump nim-json-rpc and nim-websock (use news by default) (#3820) 2022-06-29 17:35:09 +03:00
tersec 5145c04be2
Revert "Switch back to nim-websock (#3665)" (#3812)
This reverts commit 8d3afd8285.
2022-06-27 22:21:30 +00:00
Tanguy 8d3afd8285
Switch back to nim-websock (#3665) 2022-06-21 17:43:55 +02:00
Jacek Sieka c6915ed3e0
remove faststreams backend for eth2_network (#3617)
unmaintained, incompatible with altair+
2022-05-08 09:08:13 +02:00
tersec 4a372410a4
use MAX_CHUNK_SIZE_BELLATRIX for signed Bellatrix blocks (#3613)
* use MAX_CHUNK_SIZE_BELLATRIX for signed Bellatrix blocks

* Update beacon_chain/networking/eth2_network.nim

Co-authored-by: Etan Kissling <etan@status.im>

* localPassC to localPassc

* check against maxChunkSize rather than constant

Co-authored-by: Etan Kissling <etan@status.im>
2022-05-05 05:45:35 +00:00
Ștefan Talpalaru a130ce1eac
Windows: give up stack traces for 40% better performance (#3466)
See: https://github.com/status-im/nimbus-eth2/issues/3453

TODO: make libbacktrace/libunwind work in MSYS2, to get those stack traces back.
2022-03-05 16:40:08 +02:00
tersec 702d9e8c55
for x86 macOS, require >= Nehalem (#3353) 2022-02-02 15:24:41 +00:00
Ștefan Talpalaru 72658d42d2 Apple M1: proper "-march=native" substitute 2021-12-17 13:27:53 +02:00
Ștefan Talpalaru 7eecbedf3e
Windows: define "nimRawSetjmp" (#3204)
Avoids some rare crashes. Fixes https://github.com/status-im/nimbus-eth2/issues/3121
2021-12-17 04:05:47 +01:00
tersec 07e15a26fb
non-march=native builds should use SSSE3 instead of SSE3 (#3162) 2021-12-05 18:02:58 +00:00
zah bc6a6a8b8f
Switch back to the news package in nim-json-rpc (#3136) 2021-11-29 21:47:31 +01:00
Ștefan Talpalaru 74f03ddadb
CI: statically check stack size on Linux (#3055) 2021-11-05 18:11:25 +01:00
Ștefan Talpalaru b09cd71348 build: show stderr by default
Also added: `-d:cwarnings` and `-d:limitStackUsage`
2021-11-05 16:06:48 +02:00
Jacek Sieka 233d756518
Logging and startup improvements (#3038)
* Logging and startup improvements

Color support for released binaries!

* startup scripts no longer log to file by default - this only affects
source builds - released binaries don't support file logging
* add --log-stdout option to control logging to stdout (colors, json)
* detect tty:s vs redirected logs and log accordingly
* add option to disable log colors at runtime
* simplify several "common" logs, showing the most important information
earlier and more clearly
* remove line numbers / file information / tid - these take up space and
are of little use to end users
  * still enabled in debug builds and tools
* remove `testnet_servers_image` compile-time option
* server images, released binaries and compile-from-source now offer
the same behaviour and features
* fixes https://github.com/status-im/nimbus-eth2/issues/2326
* fixes https://github.com/status-im/nimbus-eth2/issues/1794
* remove instanteneous block speed from sync message, keeping only
average

before:

```
INF 2021-10-28 16:45:59.000+02:00 Slot start                                 topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:884 lastSlot=2384027 wallSlot=2384028 delay=461us84ns peers=0 head=75a10ee5:3348 headEpoch=104 finalized=cd6804ba:3264 finalizedEpoch=102 sync="wwwwwwwwww:0:0.0000:0.0000:00h00m (3348)"
INF 2021-10-28 16:45:59.046+02:00 Slot end                                   topics="beacnde" tid=386429 file=nimbus_beacon_node.nim:821 slot=2384028 nextSlot=2384029 head=75a10ee5:3348 headEpoch=104 finalizedHead=cd6804ba:3264 finalizedEpoch=102 nextAttestationSlot=-1 nextProposalSlot=-1 nextActionWait=n/a
```

after:

```
INF 2021-10-28 22:43:23.033+02:00 Slot start                                 topics="beacnde" slot=2385815 epoch=74556 sync="DDPDDPUDDD:10:5.2258:01h19m (2361088)" peers=37 head=eacd2dae:2361096 finalized=73782:a4751487 delay=33ms687us715ns
INF 2021-10-28 22:43:23.291+02:00 Slot end                                   topics="beacnde" slot=2385815 nextActionWait=n/a nextAttestationSlot=-1 nextProposalSlot=-1 head=eacd2dae:2361096
```

* fix comment

* documentation updates

* mention `--log-file` may be deprecated in the future
* update various docs
2021-11-02 18:06:36 +01:00
Ștefan Talpalaru c4bf4f8fff
Apple M1: disable -march=native (#2759) 2021-08-04 15:22:23 +02:00
Zahary Karadjov b7aa30adfd
Merge stable into unstable 2021-05-20 13:50:40 +03:00
Ștefan Talpalaru e4f4cb1513
remove "beacon_chain.nimble" (#2316)
Tests have been moved to the Makefile and the test binaries are built in
parallel.

The Nim compilation scheme has been moved to a script and adapted to
work with different binaries being built in parallel from the same main
source file (think minimal/mainnet tests).
2021-02-14 20:04:54 +00:00
Ștefan Talpalaru e58a355a1c
CI release: ARM and ARM64 builds (#2213) 2021-01-07 10:19:29 +01:00
Jacek Sieka b367db1a2a
disable frame pointer omission optimization (#2189)
* disable frame pointer omission optimization

it breaks the GC

* oops
2020-12-16 14:07:48 +01:00
Ștefan Talpalaru 07ea1cd177
Revert "disableMarchNative: also disable ADX in BLST (#2175)"
This reverts commit 6cfe7de6b0.

It's useless, since "-madx" will not be enabled without "-march=native".
2020-12-11 16:51:09 +01:00
Ștefan Talpalaru 6cfe7de6b0
disableMarchNative: also disable ADX in BLST (#2175) 2020-12-11 08:16:17 +01:00
Ștefan Talpalaru 92750d5313 reduce peak memory usage during build 2020-12-10 17:17:12 +02:00
tersec 1d7fb2ed0c
remove {.inline.} pragmas (#2033)
* remove {.inline.} pragmas

* re-add inline on bitseqs functions and tweak inlining threshold

* remove macOS/LLVM inlining setting; revert non-init/module-local/tests inline pragma removal
2020-11-20 11:00:22 +01:00
Jacek Sieka ab8f8ccaba
clean up dependencies (#2008)
* clean up dependencies

* use non-prce-mode for metrics
* clean up obsolete snappy and gflags deps from manuals

* conditional pcre
2020-11-13 16:00:45 +01:00
Ștefan Talpalaru 2396417581
bump nim-libbacktrace (#1799)
so stack traces work with LTO
2020-10-27 13:09:03 +01:00
Ștefan Talpalaru f1acc0532c
merge "nim.cfg" into "config.nims" (#1820)
to reduce the number of configuration files
2020-10-07 12:02:00 +02:00
Mamy Ratsimbazafy 6f455fb8e4
Revert "ARM64 BLST hotfix - disable BLST on ARM64 (#1753)" (#1775)
This reverts commit feece1382d.
2020-09-29 14:43:32 +02:00
Mamy Ratsimbazafy 98661914f1
Revert "bump nim-libbacktrace (#1769)" (#1773)
This reverts commit de1b3315e8.
2020-09-28 22:37:18 +02:00
Ștefan Talpalaru de1b3315e8
bump nim-libbacktrace (#1769)
* bump nim-libbacktrace

so it works with LTO

* macOS fix
2020-09-28 21:23:53 +02:00
Ștefan Talpalaru ef47782e2e
config.nims: fix the fix 2020-09-26 13:19:01 +02:00
Ștefan Talpalaru fffecbc8fd
Nim define for disabling LTO (#1751) 2020-09-25 18:15:02 +02:00
Mamy Ratsimbazafy feece1382d
ARM64 BLST hotfix - disable BLST on ARM64 (#1753) 2020-09-25 18:14:07 +02:00
Jacek Sieka 1a9e9fc398
use LTO in release builds (#1661)
* use LTO in release builds

This significantly (40%) speeds up block replay and hashing - for example replaying first 1000
blocks, without/with LTO:

```
[arnetheduck@tempus ncli]$ ../env.sh nim c -d:release ncli_db
[arnetheduck@tempus ncli]$ ./ncli_db bench --db:db --network:medalla --slots:1000
Loaded 215006 blocks, head slot 307400
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
   25468.481,        0.000,    25468.481,    25468.481,            1, Initialize DB
       0.297,        0.516,        0.053,       13.645,          721, Load block from database
      26.458,        0.000,       26.458,       26.458,            1, Load state from database
      20.737,        8.288,       11.096,      199.325,          690, Apply block
     333.069,       62.798,       45.225,      429.452,           31, Apply epoch block
       0.000,        0.000,        0.000,        0.000,            0, Database block store
```

```
[arnetheduck@tempus ncli]$ ../env.sh nim c -d:release --passc:-flto --passl:-flto --stacktrace:off ncli_db
[arnetheduck@tempus ncli]$ ./ncli_db bench --db:db --network:medalla --slots:1000
Loaded 215006 blocks, head slot 307400
All time are ms
     Average,       StdDev,          Min,          Max,      Samples,         Test
Validation is turned off meaning that no BLS operations are performed
   23903.006,        0.000,    23903.006,    23903.006,            1, Initialize DB
       0.253,        0.122,        0.047,        0.731,          721, Load block from database
      24.455,        0.000,       24.455,       24.455,            1, Load state from database
      18.734,        7.062,       10.346,      167.397,          690, Apply block
     194.869,       33.175,       29.311,      226.981,           31, Apply epoch block
```

Epoch processing is heavy on both arithmetics and hash caching, both of which get a
significant boost here.

This makes sense: nim creates lots of small functions spread out over many C files. A much
worse solution is to try to annotate code with `inline` - it copies functions to multiple
C files but still doesn't do intermodule optimizations significantly limiting the
compilers' ability to reason about the code, causing bloat and misrepresenting the usefulness
of a function to the call frequency analysis that drives actual (C-compiler) inlining and many
other optimizations.

In particular, many nim functions are part of `system` or the `C` backend - stack tracing,
memory allocation etc - nim's inlining system is pretty incomplete in that it does not deal
with these and many other cases.

* windows workaround

* skip LTO on windows for now
2020-09-24 18:40:28 +02:00
Mamy Ratsimbazafy 1f5b487fc6
Github Actions CI (#1693)
* Clean up PR + bump nimbus-build-system

* pcre on 32-bit + Improve env variable handling + cache mingw

* Add badge + fix setting env variable

* Auto cancel if commit becomes outdated

* fix shell for deriving env variable

* Add more cancellation points

* Add finalization tests to Github Action

* Fix case

* change cancel actions + fixes for windows and finalization

* have to use matrix variable for cache path/key

* ARCH_OVERRIDE=LATFORM issue rebuild cache

* Update scripts - deactivate workflows with identified issues + reactivate caching

* workaround mac getopt

* Disable all aAVX512f extensions (Error: invalid register for .seh_savexmm in Cygwin)

* Fix cross compile of libminiupmp #1723

* Cache fetch-dlls to avoid being a drag on nim-lang.org

* Fix windows downloading DLLs twice and set CFLAGS env variable for Linux32

* fix silly yaml mistake

* .

* reactivate win32 after https://github.com/status-im/nim-beacon-chain/pull/1726

* Comment out minimal tests for now
2020-09-24 13:45:34 +02:00
Jacek Sieka eaea3dbc94
remove chronos future tracking from default build (#1628)
uses a lot of memory, specially during sync
2020-09-11 17:07:16 +02:00