Nimbus: an Ethereum Execution Client for Resource-Restricted Devices https://status-im.github.io/nimbus-eth1/
Go to file
Jamie Lokier 6ef9bfd21b
EVMC: Byte-endian conversions for 256-bit numeric values
Perform byte-endian conversion for 256-bit numeric values, but not 256-bit
hashes.  These conversions are necessary for EVMC binary compatibility.

In new EVMC, all host-side conversions are explicit, calling `flip256`.

These conversions are performed in the EVMC "glue" code, which deals with the
binary interface, so the host services aren't aware of conversions.

We intend to skip these conversions when Nimbus host calls Nimbus EVM, even
when it's a shared library, using a negotiated EVMC extension.  But for now
we're focused on correctness and cross-validation with third party EVMs.

The overhead of endian conversion is not too high because most EVMC host calls
access the database anyway.  `getTxContext` does not, so the conversions from
that are cached here.  Also, well-optimised EVMs don't call it often.

It is arguable whether endian conversion should occur for storage slots (`key`).

In favour of no conversion: Slot keys are 32-byte blobs, and this is clear in
the EVMC definition where slot keys are `evmc_bytes32` (not `evmc_uint256be`),
meaning treating as a number is _not_ expected by EVMC.  Although they are
often small numbers, sometimes they are a hash from the contract code plus a
number.  Slot keys are hashed on the host side with Keccak256 before any
database calls, so the host side does not look at them numerically.

In favour of conversion: They are often small numbers and it is helpful to log
them as such, rather than a long string of zero digits with 1-2 non-zero.  The
representation in JSON has leading zeros removed, like a number rather than a
32-byte blob.  There is also an interesting space optimisation when the keys
are used unhashed in storage.

Nimbus currently treats slot keys on the host side as numbers, and the tests
pass when endian conversion is done.  So to remain consistent with other parts
of Nimbus we convert slot keys.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-12-10 16:23:27 +00:00
.github/workflows Add local testnet script and required json-rpc calls (#891) 2021-11-24 08:45:55 +01:00
.vscode vscode: add default build task 2018-08-22 21:43:22 -06:00
doc Update Nimbus - An Ethereum 2.0 Sharding Client_xt.md 2018-05-28 10:44:12 -07:00
docker first step into hive integration, fixes Dockerfile 2021-02-14 18:26:46 +07:00
examples Nimbus-Grafana-dashboard.json: gauge panel fix 2019-10-02 19:02:06 +02:00
fluffy Bit of cleanup (#903) 2021-12-08 11:54:22 +01:00
hive_integration Hive: Add Arrow Glacier fork to Hive integration + related fixes 2021-12-10 13:40:54 +00:00
nimbus EVMC: Byte-endian conversions for 256-bit numeric values 2021-12-10 16:23:27 +00:00
nix Remove -linkshared & update Nix file to be consistent 2019-11-27 11:06:58 +01:00
premix fixes related to nim-json-rpc bump 2021-11-30 14:13:20 +07:00
stateless Bugfix: Off by 1 in EIP-170 code size checks in `stateless` 2021-10-19 10:30:53 +01:00
tests Arrow Glacier fork 2021-12-10 13:40:51 +00:00
vendor Use json rpc client to run tests on portal testnet (#899) 2021-12-03 09:51:25 +01:00
.appveyor.yml Remove wakunode and waku rpc code from repository 2020-05-07 20:49:14 +03:00
.gitignore CI: use MSYS2 on Windows 2021-04-09 17:19:40 +07:00
.gitmodules Use SSZ code from nim-ssz-serialization module (#875) 2021-10-23 14:28:12 +02:00
.travis.yml Remove wakunode and waku rpc code from repository 2020-05-07 20:49:14 +03:00
BlockchainTests.md fixes EIP2929 CALL opCode 2021-01-14 23:22:28 +07:00
GeneralStateTests.md fixes EIP2929 CALL opCode 2021-01-14 23:22:28 +07:00
LICENSE-APACHEv2 Update README badges and add dual-license header 2018-04-06 16:52:10 +02:00
LICENSE-MIT Update README badges and add dual-license header 2018-04-06 16:52:10 +02:00
Makefile Use json rpc client to run tests on portal testnet (#899) 2021-12-03 09:51:25 +01:00
PersistBlockTests.md Add PersistBlockTest for block 1352922, fixes #363 2019-08-14 20:56:57 +02:00
PrecompileTests.md update tests logs 2021-05-17 11:14:33 +07:00
README.md Fix typo in README.md (#850) 2021-09-29 13:04:51 +02:00
TracerTests.md update test logs 2019-01-06 14:27:06 +01:00
TransactionTests.md update tx tests json fixtures 2020-02-20 09:02:20 +02:00
azure-pipelines.yml Nim-1.2.6-RC1 (#524) 2020-07-30 14:10:22 +02:00
config.nims Windows: support more Mingw-w64 versions 2021-05-19 00:42:56 +02:00
default.nix Add nix recipe to build Nimbus wrappers 2019-11-19 11:02:00 +00:00
env.sh bump submodules 2019-08-24 21:12:43 +02:00
newBlockchainTests.md update test logs 2021-09-29 10:55:32 +07:00
newGeneralStateTests.md update test logs 2021-09-29 10:55:32 +07:00
nimbus.nimble Use json rpc client to run tests on portal testnet (#899) 2021-12-03 09:51:25 +01:00
run-nimbus-sync Sync: Use stable nodekey/enode to accelerate sync testing 2021-04-29 21:25:26 +01:00
test_macro.nim [FEATURE] Add support for handling experimental api call (#746) 2021-07-07 11:04:18 +02:00
witnessBuilderBC.md update tests logs 2021-05-17 11:14:33 +07:00
witnessBuilderGST.md update tests logs 2021-05-17 11:14:33 +07:00

README.md

Nimbus: ultra-light Ethereum execution layer client

License: Apache License: MIT Stability: experimental GH action-nimbus-eth1 GH action-fluffy

Discord: Nimbus Gitter: #status-im/nimbus Status: #nimbus-general

Introduction

This repository contains our development work on our execution-layer client to pair with our consensus-layer client. This client focuses on efficiency and security and strives to be as light-weight as possible in terms of resources used.

This repository is also home to fluffy, a Portal Network light client.

All consensus-layer client development is happening in parallel in the nimbus-eth2 repository.

Development Updates

Monthly development updates are shared here.

Some recent highlights include:

  • Renewed funding from the EF to accelerate development
  • Completed Berlin and London fork compatibility (EIP-1559). It now passes nearly all the EF Hive testsuite, and 100% of contract execution tests (47,951 tests)
  • New GraphQL and WebSocket APIs, complementing JSON-RPC
  • EVMC compatibility, supporting third-party optimised EVM plugins
  • Up to 100x memory saving during contract executions
  • Asynchronous EVM to execute many contracts in parallel, while they wait for data from the network
  • Proof-of-authority validation for the Goerli test network
  • Updated network protocols, to work with the latest eth/65-66 and snap/1 protocols
  • A prototype new mechanism for state sync which combines what have been called Fast sync, Snap sync and Beam sync in a self-tuning way, and allows the user to participate in the network (read accounts, run transactions etc.) while sync is still in progress
  • A significant redesign of the storage database to use less disk space and run faster.

For more detailed write-ups on the development progress, follow the Nimbus blog.

Building & Testing

We currently do not guarantee that Nimbus will work on Windows.

Prerequisites

  • RocksDB
  • GNU Make, Bash and the usual POSIX utilities. Git 2.9.4 or newer.

On Windows, a precompiled DLL collection download is available through the fetch-dlls Makefile target: (Windows instructions).

# MacOS with Homebrew
brew install rocksdb

# Fedora
dnf install rocksdb-devel

# Debian and Ubuntu
sudo apt-get install librocksdb-dev

# Arch (AUR)
pakku -S rocksdb

rocksdb can also be installed by following their instructions.

Obtaining the prerequisites through the Nix package manager

Experimental

Users of the Nix package manager can install all prerequisites simply by running:

nix-shell default.nix

Build & Develop

POSIX-compatible OS

# The first `make` invocation will update all Git submodules.
# You'll run `make update` after each `git pull`, in the future, to keep those submodules up to date.
make nimbus

# See available command line options
build/nimbus --help

# Start syncing with mainnet
build/nimbus

# Update to latest version
git pull
make update

# Run tests
make test

To run a command that might use binaries from the Status Nim fork:

./env.sh bash # start a new interactive shell with the right env vars set
which nim
nim --version

# or without starting a new interactive shell:
./env.sh which nim
./env.sh nim --version

Our Wiki provides additional helpful information for debugging individual test cases and for pairing Nimbus with a locally running copy of Geth.

Windows

(Experimental support!)

Install Mingw-w64 for your architecture using the "MinGW-W64 Online Installer" (first link under the directory listing). Run it and select your architecture in the setup menu ("i686" on 32-bit, "x86_64" on 64-bit), set the threads to "win32" and the exceptions to "dwarf" on 32-bit and "seh" on 64-bit. Change the installation directory to "C:\mingw-w64" and add it to your system PATH in "My Computer"/"This PC" -> Properties -> Advanced system settings -> Environment Variables -> Path -> Edit -> New -> C:\mingw-w64\mingw64\bin (it's "C:\mingw-w64\mingw32\bin" on 32-bit)

Install Git for Windows and use a "Git Bash" shell to clone and build Nimbus.

If you don't want to compile RocksDB and SQLite separately, you can fetch pre-compiled DLLs with:

mingw32-make fetch-dlls # this will place the right DLLs for your architecture in the "build/" directory

You can now follow those instructions in the previous section by replacing make with mingw32-make (regardless of your 32-bit or 64-bit architecture):

mingw32-make nimbus # build the Nimbus binary
mingw32-make test # run the test suite
# etc.

Raspberry PI

Experimental The code can be compiled on a Raspberry PI:

  • Raspberry PI 3b+
  • 64gb SD Card (less might work too, but the default recommended 4-8GB will probably be too small)
  • Rasbian Buster Lite - Lite version is enough to get going and will save some disk space!

Assuming you're working with a freshly written image:


# Start by increasing swap size to 2gb:
sudo vi /etc/dphys-swapfile
# Set CONF_SWAPSIZE=2048
# :wq
sudo reboot

# Install prerequisites
sudo apt-get install git libgflags-dev libsnappy-dev

mkdir status
cd status

# Install rocksdb
git clone https://github.com/facebook/rocksdb.git
cd rocksdb
make shared_lib
sudo make install
cd..

# Raspberry pi doesn't include /usr/local/lib in library search path - need to add
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

git clone https://github.com/status-im/nimbus.git

cd nimbus

# Follow instructions above!

Android

Experimental Code can be compiled and run on Android devices

Environment setup
  • Install the Termux app from FDroid or the Google Play store
  • Install a PRoot of your choice following the instructions for your preferred distribution. Note, the Ubuntu PRoot is known to contain all Nimbus prerequisites compiled on Arm64 architecture (common architecture for Android devices). Depending on the distribution, it may require effort beyond the scope of this guide to get all prerequisites.

Assuming Ubuntu PRoot is used

# Install prerequisites
apt install librocksdb-dev

# Clone repo and build Nimbus just like above
git clone https://github.com/status-im/nimbus.git

cd nimbus

make

make nimbus

build/nimbus

Experimental make variables

Apart from standard make flags (see link in the next chapter), the following make variables can be set to control which version of a virtual engine is compiled. The variables are listed with decreasing priority (in case of doubt, the lower prioritised variable is ignored when the higher on is available.)

  • ENABLE_EVMC=1
    Enable mostly EVMC compliant wrapper around the native nim VM

  • ENABLE_VM2LOWMEM=1
    Enable new re-factored version of the native nim VM. This version is not optimised and coded in a way so that low memory compilers can handle it (observed on 32 bit windows 7.)

  • ENABLE_VM2=1
    Enable new re-factored version of the native nim VM.

For these variables, using <variable>=0 is ignored and <variable>=2 has the same effect as <variable>=1 (ditto for other numbers.)

Development tips

Interesting Make variables and targets are documented in the nimbus-build-system repo.

  • you can switch the DB backend with a Nim compiler define: -d:nimbus_db_backend=... where the (case-insensitive) value is one of "rocksdb" (the default), "sqlite", "lmdb"

  • the Premix debugging tools are documented separately

  • you can control the Makefile's verbosity with the V variable (defaults to 0):

make V=1 # verbose
make V=2 test # even more verbose
make LOG_LEVEL=DEBUG nimbus # this is the default
make LOG_LEVEL=TRACE nimbus # log everything
  • pass arbitrary parameters to the Nim compiler:
make NIMFLAGS="-d:release"
  • if you want to use SSH keys with GitHub (also handles submodules):
make github-ssh
  • force a Nim compiler rebuild:
rm vendor/Nim/bin/nim
make -j8 build-nim

Git submodule workflow

Working on a dependency:

cd vendor/nim-chronicles
git checkout -b mybranch
# make some changes
git status
git commit -a
git push origin mybranch
# create a GitHub PR and wait for it to be approved and merged
git checkout master
git pull
git branch -d mybranch
# realise that the merge was done without "--no-ff"
git branch -D mybranch
# update the submodule's commit in the superproject
cd ../..
git status
git add vendor/nim-chronicles
git commit

It's important that you only update the submodule commit after it's available upstream.

You might want to do this on a new branch of the superproject, so you can make a GitHub PR for it and see the CI test results.

Don't update all Git submodules at once, just because you found the relevant Git command or make target. You risk updating submodules to other people's latest commits when they are not ready to be used in the superproject.

Adding the submodule "https://github.com/status-im/foo" to "vendor/foo":

vendor/nimbus-build-system/scripts/add_submodule.sh status-im/foo
# or
./env.sh add_submodule status-im/foo
# want to place it in "vendor/bar" instead?
./env.sh add_submodule status-im/foo vendor/bar

Removing the submodule "vendor/bar":

git submodule deinit -f -- vendor/bar
git rm -f vendor/bar

Checking out older commits, either to bisect something or to reproduce an older build:

git checkout <commit hash here>
make clean
make -j8 update

Running a dependency's test suite using nim instead of nimble (which cannot be convinced not to run a dependency check, thus clashing with our jury-rigged "vendor/.nimble/pkgs"):

cd vendor/nim-rocksdb
../nimbus-build-system/scripts/nimble.sh test
# or
../../env.sh nimble test

Metric visualisation

Install Prometheus and Grafana. On Gentoo, it's emerge prometheus grafana-bin.

# build Nimbus
make nimbus
# the Prometheus daemon will create its data dir in the current dir, so give it its own directory
mkdir ../my_metrics
# copy the basic config file over there
cp -a examples/prometheus.yml ../my_metrics/
# start Prometheus in a separate terminal
cd ../my_metrics
prometheus --config.file=prometheus.yml # loads ./prometheus.yml, writes metric data to ./data
# start a fresh Nimbus sync and export metrics
rm -rf ~/.cache/nimbus/db; ./build/nimbus --prune:archive --metricsServer

Start the Grafana server. On Gentoo it's /etc/init.d/grafana start. Go to http://localhost:3000, log in with admin:admin and change the password.

Add Prometheus as a data source. The default address of http://localhost:9090 is OK, but Grafana 6.3.5 will not apply that semitransparent default you see in the form field, unless you click on it.

Create a new dashboard. Click on its default title in the upper left corner ("New Dashboard"). In the new page, click "Import dashboard" in the right column and upload "examples/Nimbus-Grafana-dashboard.json".

In the main panel, there's a hidden button used to assign metrics to the left or right Y-axis - it's the coloured line on the left of the metric name, in the graph legend.

To see a single metric, click on its name in the legend. Click it again to go back to the combined view. To edit a panel, click on its title and select "Edit".

Obligatory screenshot.

Troubleshooting

Report any errors you encounter, please, if not already documented!

  • Turn it off and on again:
make clean
make update

License

Licensed and distributed under either of

or

at your option. These files may not be copied, modified, or distributed except according to those terms.