8211db1ea8
There's been a lot of talk about the Nimbus EVM "stack problem". I think we
assumed changing it would require big changes to the interpreter code, touching
a lot of functions.
It turned out to be a low hanging fruit.
This patch solves the stack problem, but hardly touches anything. The change
in EVM stack memory is from 13 MB worst case to just 48 kB, a 250x reduction.
I've been doing work on the database/storage/trie code. While looking at the
API between the EVM and the database/storage/trie, this stack patch stood out
and made itself obvious. As it's tiny, rather than more talk, here it is.
Note: This patch is intentionally small, non-invasive, and hopefully easy to
understand, so that it doesn't conflict with other work done on the EVM, and
can easily be grafted into any other EVM structure.
Motivation
==========
- We run out of space and crash on some targets, unless the stack limit is
raised above its default. Surprise segmentation faults are unhelpful.
- Some CI targets have been disabled for months due to this.
- Because usage borders on the system limits, when working on
database/storage/trie/sync code (called from the EVM), segmentation faults
occur and are misleading. They cause lost time due to thinking there's a
crash bug in the code being worked on, when there's nothing wrong with it.
- Sometimes unrelated, trivial code changes elsewhere trigger CI test failures.
It looks like abrupt termination. A simple, recent patch was crashing in
`make test` even though it was a trivial refactor. Turns out it pushed the
stack over the edge.
- A large stack has to be scanned by the Nim garbage collector sometimes.
Larger stack means slower GC and memory allocation.
- The structure of this small patch suggests how to weave async into the EVM
with almost no changes to the EVM, and no async transformation overhead.
- The patch seemed obvious when working on the API between EVM and storage.
Measurements before
===================
All these tests were run on Ubuntu 20.04 server, x86-64. This is one of the
targets that has been disabled for a while in CI in EVMC mode due to crashing,
and excessive stack usage is the cause.
Testing commit 0c34a8e3 `2021-04-08 17:46:00 +0200 CI: use MSYS2 on Windows`.
$ rm -f build/all_tests && make ENABLE_EVMC=1 test
$ ulimit -S -s 16384 # Requires larger stack than default to avoid crash.
$ ./build/all_tests 9 | tee tlog
[Suite] persist block json tests
...
Stack range 38496 depthHigh 3
...
Stack range 13140272 depthHigh 1024
[OK] tests/fixtures/PersistBlockTests/block1431916.json
These tests use 13.14 MB of stack to run, and so crash with the default stack
limit on Ubuntu Server 20.04 (8MB). Exactly 12832 bytes per EVM call stack
frame. It's interesting to see some stack frames take a bit more.
$ rm -f build/all_tests && make ENABLE_EVMC=1 test
$ ulimit -S -s 16384 # Requires larger stack than default.
$ ./build/all_tests 7 | tee tlog
[Suite] new generalstate json tests
...
Stack range 15488 depthHigh 2
...
Stack range 3539312 depthHigh 457
[OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json
...
Stack range 3756144 depthHigh 485
[OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json
...
Stack range 7929968 depthHigh 1024
[OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json
These tests use 7.92MB of stack to run. About 7264 bytes per EVM call stack
frame. It _only just_ avoids crashing with the default Ubuntu Server stack
limit of 8 MB. However, it still crashes on Windows x86-64, which is why the
CI target is currently disabled.
On Linux where this passes, this is so borderline that it affects work and
testing of storage and sync code, because that's called from the EVM. Which
was a motivation for dealing with the stack instead of letting this linger.
Also, this stack greatly exceeds the default thread stack size.
$ rm -f build/all_tests && make ENABLE_EVMC=0 test
$ ulimit -S -s 16384 # Requires larger stack than default to avoid crash.
$ ./build/all_tests 9 | tee tlog
[Suite] persist block json tests
...
Stack range 33216 depthHigh 3
...
Stack range 11338032 depthHigh 1024
[OK] tests/fixtures/PersistBlockTests/block1431916.json
These tests use 11.33 MB stack to run, and so crash with a default stack limit
of 8MB. Exactly 11072 bytes per EVM call stack frame. It's interesting to see
some stack frames take a bit more.
$ rm -f build/all_tests && make ENABLE_EVMC=0 test
$ ulimit -S -s 16384 # Requires larger stack than default.
$ ./build/all_tests 7 | tee tlog
[Suite] new generalstate json tests
...
Stack range 10224 depthHigh 2
...
Stack range 2471760 depthHigh 457
[OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json
...
Stack range 2623184 depthHigh 485
[OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json
...
Stack range 5537824 depthHigh 1024
[OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json
These tests use 5.54 MB of stack to run, and avoid crashing on with a default
stack limit of 8 MB. About 5408 bytes per EVM call stack frame.
However, this is uncomfortably close to the limit, as the stack frame size is
sensitive to changes in the code.
Also, this stack greatly exceeds the default thread stack size.
Measurements after
==================
(This patch doesn't address EVMC mode, which is not our default. EVMC stack
usage remains about the same. EVMC mode is addressed in another tiny patch.)
$ rm -f build/all_tests && make ENABLE_EVMC=0 test
$ ulimit -S -s 80 # Because we can! 80k stack.
$ ./build/all_tests 9 | tee tlog
[Suite] persist block json tests
...
Stack range 496 depthHigh 3
...
Stack range 49504 depthHigh 1024
[OK] tests/fixtures/PersistBlockTests/block1431916.json
$ rm -f build/all_tests && make ENABLE_EVMC=0 test
$ ulimit -S -s 72 # Because we can! 72k stack.
$ ./build/all_tests 7 | tee tlog
[Suite] new generalstate json tests
...
Stack range 448 depthHigh 2
...
Stack range 22288 depthHigh 457
[OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json
...
Stack range 23632 depthHigh 485
[OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json
...
Stack range 49504 depthHigh 1024
[OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json
For both tests, a satisfying *48 bytes* per EVM call stack frame, and EVM takes
not much more than 48 kB. With other overheads, both tests run in 80 kB stack
total at maximum EVM depth.
We must add some headroom on this for database activity called from the EVM,
and different compile targets. But it means the EVM itself is no longer a
stack burden.
This is much smaller than the default thread stack size on Linux (2MB), with
plenty of margin. It's even smaller than Linux from a long time ago (128kB),
and some small embedded C targets. (Just fyi, though, some JVM environments
allocated just 32 kB to thread stacks.)
This size is also well suited to running EVMs in threads, if that's useful.
Subtle exception handling and `dispose`
=======================================
It is important that each `snapshot` has a corresponding `dispose` in the event
of an exception being raised. This code does do that, but in a subtle way.
The pair of functions `execCallOrCreate` and `execCallOrCreateAux` are
equivalent to the following code, where you can see `dispose` more clearly:
proc execCallOrCreate*(c: Computation) =
defer: c.dispose()
if c.beforeExec():
return
c.executeOpcodes()
while not c.continuation.isNil:
c.child.execCallOrCreate()
c.child = nil
(c.continuation)()
c.executeOpcodes()
c.afterExec()
That works fine, but only reduces the stack used to 300-700 kB instead of 48 kB.
To get lower we split the above into separate `execCallOrCreate` and
`execCallOrCreateAux`. Only the outermost has `defer`, and instead of handling
one level, it walks the entire `c.parent` chain calling `dispose` if needed.
The inner one avoids `defer`, which greatly reduces the size of its stackframe.
`c` is a `var` parameter, at each level of recursion. So the outermost proc
sees the temporary changes made by all inner calls. This is why `c` is updated
and the `c.parent` chain is maintained at each step.
Signed-off-by: Jamie Lokier <jamie@shareable.org>
Nimbus: an Ethereum 2.0 Sharding Client for Resource-Restricted Devices
Rationale
Nimbus: an Ethereum 2.0 Sharding Client. The code in this repository is currently focusing on Ethereum 1.0 feature parity, while all 2.0 research and development is happening in parallel in nimbus-eth2.
Development Updates
To keep up to date with changes and development progress, follow the Nimbus blog.
Building & Testing
We currently do not guarantee that Nimbus will work on Windows.
Prerequisites
- RocksDB
- PCRE
- GNU Make, Bash and the usual POSIX utilities. Git 2.9.4 or newer.
On Windows, a precompiled DLL collection download is available through the fetch-dlls Makefile target: (Windows instructions).
# MacOS with Homebrew
brew install rocksdb pcre
# Fedora
dnf install rocksdb-devel pcre pcre-devel
# Debian and Ubuntu
sudo apt-get install librocksdb-dev libpcre3-dev
# Arch (AUR)
pakku -S rocksdb pcre-static
rocksdb can also be installed by following their instructions.
Obtaining the prerequisites through the Nix package manager
Experimental
Users of the Nix package manager can install all prerequisites simply by running:
nix-shell default.nix
Build & Develop
POSIX-compatible OS
# The first `make` invocation will update all Git submodules.
# You'll run `make update` after each `git pull`, in the future, to keep those submodules up to date.
make nimbus
# See available command line options
build/nimbus -- help
# Start syncing with mainnet
build/nimbus
# Update to latest version
git pull
make update
# Run tests
make test
To run a command that might use binaries from the Status Nim fork:
./env.sh bash # start a new interactive shell with the right env vars set
which nim
nim --version
# or without starting a new interactive shell:
./env.sh which nim
./env.sh nim --version
Our Wiki provides additional helpful information for debugging individual test cases and for pairing Nimbus with a locally running copy of Geth.
Windows
(Experimental support!)
Install Mingw-w64 for your architecture using the "MinGW-W64 Online Installer" (first link under the directory listing). Run it and select your architecture in the setup menu ("i686" on 32-bit, "x86_64" on 64-bit), set the threads to "win32" and the exceptions to "dwarf" on 32-bit and "seh" on 64-bit. Change the installation directory to "C:\mingw-w64" and add it to your system PATH in "My Computer"/"This PC" -> Properties -> Advanced system settings -> Environment Variables -> Path -> Edit -> New -> C:\mingw-w64\mingw64\bin (it's "C:\mingw-w64\mingw32\bin" on 32-bit)
Install Git for Windows and use a "Git Bash" shell to clone and build Nimbus.
If you don't want to compile RocksDB and SQLite separately, you can fetch pre-compiled DLLs with:
mingw32-make fetch-dlls # this will place the right DLLs for your architecture in the "build/" directory
You can now follow those instructions in the previous section by replacing make with mingw32-make (regardless of your 32-bit or 64-bit architecture):
mingw32-make nimbus # build the Nimbus binary
mingw32-make test # run the test suite
# etc.
Raspberry PI
Experimental The code can be compiled on a Raspberry PI:
- Raspberry PI 3b+
- 64gb SD Card (less might work too, but the default recommended 4-8GB will probably be too small)
- Rasbian Buster Lite - Lite version is enough to get going and will save some disk space!
Assuming you're working with a freshly written image:
# Start by increasing swap size to 2gb:
sudo vi /etc/dphys-swapfile
# Set CONF_SWAPSIZE=2048
# :wq
sudo reboot
# Install prerequisites
sudo apt-get install git libgflags-dev libsnappy-dev
mkdir status
cd status
# Install rocksdb
git clone https://github.com/facebook/rocksdb.git
cd rocksdb
make shared_lib
sudo make install
cd..
# Raspberry pi doesn't include /usr/local/lib in library search path - need to add
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
git clone https://github.com/status-im/nimbus.git
cd nimbus
# Follow instructions above!
Android
Experimental Code can be compiled and run on Android devices
Environment setup
- Install the Termux app from FDroid or the Google Play store
- Install a PRoot of your choice following the instructions for your preferred distribution. Note, the Ubuntu PRoot is known to contain all Nimbus prerequisites compiled on Arm64 architecture (common architecture for Android devices). Depending on the distribution, it may require effort beyond the scope of this guide to get all prerequisites.
Assuming Ubuntu PRoot is used
# Install prerequisites
apt install librocksdb-dev libpcre3-dev
# Clone repo and build Nimbus just like above
git clone https://github.com/status-im/nimbus.git
cd nimbus
make
make nimbus
build/nimbus
Development tips
Interesting Make variables and targets are documented in the nimbus-build-system repo.
-
you can switch the DB backend with a Nim compiler define:
-d:nimbus_db_backend=...where the (case-insensitive) value is one of "rocksdb" (the default), "sqlite", "lmdb" -
the Premix debugging tools are documented separately
-
you can control the Makefile's verbosity with the V variable (defaults to 0):
make V=1 # verbose
make V=2 test # even more verbose
- same for the Chronicles log level:
make LOG_LEVEL=DEBUG nimbus # this is the default
make LOG_LEVEL=TRACE nimbus # log everything
- pass arbitrary parameters to the Nim compiler:
make NIMFLAGS="-d:release"
- if you want to use SSH keys with GitHub (also handles submodules):
make github-ssh
- force a Nim compiler rebuild:
rm vendor/Nim/bin/nim
make -j8 build-nim
Git submodule workflow
Working on a dependency:
cd vendor/nim-chronicles
git checkout -b mybranch
# make some changes
git status
git commit -a
git push origin mybranch
# create a GitHub PR and wait for it to be approved and merged
git checkout master
git pull
git branch -d mybranch
# realise that the merge was done without "--no-ff"
git branch -D mybranch
# update the submodule's commit in the superproject
cd ../..
git status
git add vendor/nim-chronicles
git commit
It's important that you only update the submodule commit after it's available upstream.
You might want to do this on a new branch of the superproject, so you can make a GitHub PR for it and see the CI test results.
Don't update all Git submodules at once, just because you found the relevant
Git command or make target. You risk updating submodules to other people's
latest commits when they are not ready to be used in the superproject.
Adding the submodule "https://github.com/status-im/foo" to "vendor/foo":
vendor/nimbus-build-system/scripts/add_submodule.sh status-im/foo
# or
./env.sh add_submodule status-im/foo
# want to place it in "vendor/bar" instead?
./env.sh add_submodule status-im/foo vendor/bar
Removing the submodule "vendor/bar":
git submodule deinit -f -- vendor/bar
git rm -f vendor/bar
Checking out older commits, either to bisect something or to reproduce an older build:
git checkout <commit hash here>
make clean
make -j8 update
Running a dependency's test suite using nim instead of nimble (which cannot be
convinced not to run a dependency check, thus clashing with our jury-rigged
"vendor/.nimble/pkgs"):
cd vendor/nim-rocksdb
../nimbus-build-system/scripts/nimble.sh test
# or
../../env.sh nimble test
Metric visualisation
Install Prometheus and Grafana. On Gentoo, it's emerge prometheus grafana-bin.
# build Nimbus
make nimbus
# the Prometheus daemon will create its data dir in the current dir, so give it its own directory
mkdir ../my_metrics
# copy the basic config file over there
cp -a examples/prometheus.yml ../my_metrics/
# start Prometheus in a separate terminal
cd ../my_metrics
prometheus --config.file=prometheus.yml # loads ./prometheus.yml, writes metric data to ./data
# start a fresh Nimbus sync and export metrics
rm -rf ~/.cache/nimbus/db; ./build/nimbus --prune:archive --metricsServer
Start the Grafana server. On Gentoo it's /etc/init.d/grafana start. Go to
http://localhost:3000, log in with admin:admin and change the password.
Add Prometheus as a data source. The default address of http://localhost:9090 is OK, but Grafana 6.3.5 will not apply that semitransparent default you see in the form field, unless you click on it.
Create a new dashboard. Click on its default title in the upper left corner ("New Dashboard"). In the new page, click "Import dashboard" in the right column and upload "examples/Nimbus-Grafana-dashboard.json".
In the main panel, there's a hidden button used to assign metrics to the left or right Y-axis - it's the coloured line on the left of the metric name, in the graph legend.
To see a single metric, click on its name in the legend. Click it again to go back to the combined view. To edit a panel, click on its title and select "Edit".
Troubleshooting
Report any errors you encounter, please, if not already documented!
- Turn it off and on again:
make clean
make update
License
Licensed and distributed under either of
- MIT license: LICENSE-MIT or http://opensource.org/licenses/MIT
or
- Apache License, Version 2.0, (LICENSE-APACHEv2 or http://www.apache.org/licenses/LICENSE-2.0)
at your option. These files may not be copied, modified, or distributed except according to those terms.