Commit Graph

103 Commits

Author SHA1 Message Date
Kim De Mey 06512409ea
Fix, rework and re-enable Portal beacon light client (#1481) 2023-02-26 19:18:03 +01:00
Jordan Hrycaj 7490f23124
Silence some compiler gossip -- part 6, evm (#1462)
* Silence some compiler gossip -- part 6, evm

details:
  Adding some missing exception annotation

* Update evmc cases

why:
  were previously missing

* Increase Windows stack needed to run EVMC unit tests

why:
  After annotating functions to trace exceptions some unit tests started
  to fail on Windows without clear error report.

  EVMC works recursively and now there seems to be a stack problem
  reported by the nim compiler. Increasing the NIM stack ass sugessted by
  NIM (using -d:nimCallDepthLimit=###) had some effect but no clear
  solution.

Note that this patch set unrolls some NIM compiler settings
2023-02-14 14:37:21 +00:00
Kim De Mey 8523d8b40e
Bump submodules and fix related issues (#1448)
* Updated to the latest nim-eth, nim-rocksdb, nim-web3

* Bump nimbus-eth2 module and fix related issues

Temporarily disabling Portal beacon light client network as it is
a lot of copy pasted code that did not yet take into account
forks. This will require a bigger rework and was not yet tested
in an actual network anyhow.

* More nimbus fixes after module bumps

---------

Co-authored-by: Adam Spitz <adamspitz@status.im>
Co-authored-by: jangko <jangko128@gmail.com>
2023-01-27 15:57:48 +01:00
Kim De Mey 3a079bf30e
Add first iteration of a fluffy beacon bridge for block data (#1437) 2023-01-19 09:33:11 +01:00
KonradStaniec 270ce41d5c
Add light client bridge (#1386)
* Add light client bridge binary
2022-12-27 15:25:20 +01:00
Kim De Mey 591a1e7062
Add content verifier tool check data availability on the network (#1388) 2022-12-16 21:11:09 +01:00
Kim De Mey 99769cc0e1
Rename lc_proxy to nimbus_verified_proxy (#1321)
* Change Light client proxy naming to verified proxy naming in code

* Rename the lc_proxy files to the verified proxy naming

* Update to the verified proxy name in the docs
2022-11-23 17:11:38 +01:00
Kim De Mey 36d430aaa2
Add headers with proof content type and use it for verification (#1281)
* Add headers with proof content type and use it for verification

- Add BlockHeaderWithProof content type & content
- Use BlockHeaderWithProof content to verify if chain data is
part of the canonical chain
- Adjust parser & seeder code to be able to seed these headers
with proof
- Adjust eth_data_exporter to be able to export custom header
ranges for which to build proofs (mostly for testing)

There is currently quite some ugliness & clean-up needed for which
a big part is due tos upporting both BlockHeader and
BlockHeaderWithProof on the network.

* Change accumulator proof to array / SSZ vector type

- Change accumulator proof to SSZ vector instead of SSZ list.
- Add and use general buildProof and buildHeaderWithProof func.

* Make the BlockHeaderWithProof an SSZ Union with None option

* Update portal-spec-tests to master commit
2022-11-04 09:27:01 +01:00
Adam Spitz e040e2671a
Added basic async capabilities for vm2. (#1260)
* Added basic async capabilities for vm2.

This is a whole new Git branch, not the same one as last time
(https://github.com/status-im/nimbus-eth1/pull/1250) - there wasn't
much worth salvaging. Main differences:

I didn't do the "each opcode has to specify an async handler" junk
that I put in last time. Instead, in oph_memory.nim you can see
sloadOp calling asyncChainTo and passing in an async operation.
That async operation is then run by the execCallOrCreate (or
asyncExecCallOrCreate) code in interpreter_dispatch.nim.

In the test code, the (previously existing) macro called "assembler"
now allows you to add a section called "initialStorage", specifying
fake data to be used by the EVM computation run by that test. (In
the long run we'll obviously want to write tests that for-real use
the JSON-RPC API to asynchronously fetch data; for now, this was
just an expedient way to write a basic unit test that exercises the
async-EVM code pathway.)

There's also a new macro called "concurrentAssemblers" that allows
you to write a test that runs multiple assemblers concurrently (and
then waits for them all to finish). There's one example test using
this, in test_op_memory_lazy.nim, though you can't actually see it
doing so unless you uncomment some echo statements in
async_operations.nim (in which case you can see the two concurrently
running EVM computations each printing out what they're doing, and
you'll see that they interleave).

A question: is it possible to make EVMC work asynchronously? (For
now, this code compiles and "make test" passes even if ENABLE_EVMC
is turned on, but it doesn't actually work asynchronously, it just
falls back on doing the usual synchronous EVMC thing. See
FIXME-asyncAndEvmc.)

* Moved the AsyncOperationFactory to the BaseVMState object.

* Made the AsyncOperationFactory into a table of fn pointers.

Also ditched the plain-data Vm2AsyncOperation type; it wasn't
really serving much purpose. Instead, the pendingAsyncOperation
field directly contains the Future.

* Removed the hasStorage idea.

It's not the right solution to the "how do we know whether we
still need to fetch the storage value or not?" problem. I
haven't implemented the right solution yet, but at least
we're better off not putting in a wrong one.

* Added/modified/removed some comments.

(Based on feedback on the PR.)

* Removed the waitFor from execCallOrCreate.

There was some back-and-forth in the PR regarding whether nested
waitFor calls are acceptable:

https://github.com/status-im/nimbus-eth1/pull/1260#discussion_r998587449

The eventual decision was to just change the waitFor to a doAssert
(since we probably won't want this extra functionality when running
synchronously anyway) to make sure that the Future is already
finished.
2022-11-01 11:35:46 -04:00
KonradStaniec 1c1c6f56cc
Add initial implementation of portal powered light client (#1280)
* Add initial implementation of portal powered light client
2022-10-28 09:49:18 +02:00
Kim De Mey dd1748fd49
Make nimble task names more consistent (#1257) 2022-10-10 15:49:51 +02:00
Kim De Mey 78f7de1344
Make accumulator finite at merge block (#1256)
- Let accumulator finish its last pre merge epoch (hash_tree_root
on incomplete epoch).
- Adjust code to use isPreMerge and remove isCurrentEpoch
- Split up tests to a set that runs with a mainnet merge block
number and a set that runs with a testing value.
2022-10-10 12:59:55 +02:00
Kim De Mey 621c6a31a7
Fix bug in inCurrentEpoch and improve accumulator related tests (#1217)
* Fix bug in inCurrentEpoch and improve accumulator related tests

- Fix negative wraparound / underflow in inCurrentEpoch
- Add tests in accumulator tests to verify the above
- Add header offer tests with accumulator that does and doesn't
contain historical epochs
- Additional clean-up of history tests
- enable canonicalVerify in the tests
2022-09-09 13:12:09 +02:00
KonradStaniec 9d10f8fbae
Add mpt proof verification to proxy (#1213)
* Add mpt proof verification to proxy
2022-09-06 18:14:50 +02:00
KonradStaniec caa5e009ff
Initial Binary for light client proxy (#1202)
* Initial Binary for light client proxy
2022-08-26 13:54:10 +02:00
Kim De Mey 55dcbe9e58
Make the default network for portalcli the history network (#1195) 2022-08-23 16:28:10 +02:00
Kim De Mey 50af402f59
Add verification of block data by use of accumulator (#1178) 2022-08-01 21:00:21 +02:00
jangko e3cf999033
fix .nimble dependency 2022-07-22 09:05:42 +07:00
Kim De Mey 33015ea4a3
Adjust to the newly specified SSZ types for Block Body and Receipts (#1147) 2022-07-01 21:51:51 +02:00
KonradStaniec 730013bde3
Change test data and propagation format (#1115)
* Change test data and propagation format

* Add tool to download data from local node
2022-06-08 15:14:01 +02:00
Kim De Mey 6e05c7588e
Add Header Accumulator for the Header Gossip Network (#1094) 2022-05-18 22:49:35 +02:00
Jacek Sieka 1d6a9951d6
move rocksdb support to eth1 (#927)
* move rocksdb support to eth1

only used here / causes unnecessary build deps
2022-04-06 07:28:19 +02:00
Zahary Karadjov 6c94291eee
Fix the build under Nim 1.2 2022-03-30 23:27:10 +03:00
Zahary Karadjov 69cc7c0c13
Specify some binaries in the nimble file 2022-03-30 18:59:09 +03:00
Kim De Mey 8c3f210526
Add blockwalk tool to test range of blocks on Portal network (#1020) 2022-03-30 08:55:38 +02:00
Ștefan Talpalaru 51bc1cf87f
dist: precompiled binaries and Docker images (#1015)
* dist: precompiled binaries and Docker images

The builds are reproducible, the binaries are portable and statically link librocksdb.

This took some patching. Upstream PR: https://github.com/facebook/rocksdb/pull/9752

32-bit ARM is missing as a target because two different GCC versions
fail with an ICE when trying to cross-compile RocksDB. Using Clang
instead is too much trouble for a platform that nobody should be using
anyway.

(Clang doesn't come with its own target headers and libraries, can't be
easily convinced to use the ones from GCC, so it needs an fs image from
a 32-bit ARM distro - at which point I stopped caring).

* CI: disable reproducibility test
2022-03-27 13:21:15 +02:00
KonradStaniec cccd1e4640
Add tooling necessary to improve utp-testing (#931)
* Add tooling necessary to improve utp-testing
2022-01-19 16:06:23 +01:00
Kim De Mey 63b0945b52
Use json rpc client to run tests on portal testnet (#899)
Tests have also been added & adjusted. Original ones were actually
failing but unnoticed due to mistake in port passed in curl command.
2021-12-03 09:51:25 +01:00
Kim De Mey 02afda1b45
Run everything fluffy with chronosStrictException (#889) 2021-11-18 17:52:44 +01:00
jangko 7066b48303
add -d:unittest2DisableParamFiltering to .nimble
fix #840
2021-11-14 16:04:19 +07:00
Kim De Mey 8e26acc4eb
Move tools and add common types (#846)
* Move some duplicated types to common_types

* Move portalcli to tools/ and update the READMEs
2021-09-24 17:18:54 +02:00
Kim De Mey e37bafd47e
Add a first, simple, content network test (#760)
* Add a first, simple, content network test

* Use sqlite as nimbus_db_backend for fluffy tests

* Fix backend selection for sqlite
2021-07-15 15:12:33 +02:00
Kim De Mey 424fd405e9
Move Portal wire protocol from nim-eth to fluffy (#749)
* Move Portal wire protocol from nim-eth to fluffy

* Rename fluffy make targets

* Move all_fluffy_tests a folder down
2021-07-09 13:34:16 +02:00
KonradStaniec a59a2c61b6
[FEATURE] Add support for handling experimental api call (#746)
* [FEATURE] Add support for handling experimental api call
2021-07-07 11:04:18 +02:00
Kim De Mey f8b3922eb5
Change name from nlpn to fluffy (#734) 2021-06-28 17:53:13 +02:00
jangko 2269d16c4c
reintegrate test_rpc back into all_tests
for a long time, test_rpc will crash all_tests
if they are combined together. but now it works fine
2021-06-22 08:32:32 +07:00
Kim De Mey aef7a25174
Add nlpn (#714)
* Add start of nlpn client

* Fix some error handling in nlpn

* Clean-up GOARCH from nlpn.yml and add vendor to include paths
2021-06-16 21:18:45 +02:00
Jamie Lokier bfa7bd0488
Whisper: Stop testing Whisper
Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-01 18:12:47 +01:00
Jamie Lokier 8161de1a8f
Limit stack size on Windows like other targets
We can't use `ulimit -s` to limit stack size on Windows.  Even though Bash
accepts `ulimit -s` and the numbers change it has no effect and is not passed
to child processes.

(See https://public-inbox.org/git/alpine.DEB.2.21.1.1709131448390.4132@virtualbox/)

Instead, set it when building the test executable, following, a suggestion from
@stefantalpalaru.
https://github.com/status-im/nimbus-eth1/pull/598#discussion_r621107128

To ensure no conflict with `config.nims`, `-d:windowsNoSetStack` is used.  This
proved unnecessary in practice because the command-line option is passed to the
linker after the config file option.  But given we don't have an automated test
to verify linker behaviour, it's best not to rely on the option order, neither
how the linker treats it, or whether Nim will always send them in that order.

Testing:

This has been verified by using a smaller limit.  At 200k, all `ENABLE_EVMC=0`
OS targets passed as expected, and all `ENABLE_EVMC=1` OS targets failed with
expected kinds of errors due to stack overflow, including Windows.
(400k wasn't small enough; 32-bit x86 Windows passed on that).

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-30 01:00:31 +01:00
Jamie Lokier a3c8a5c3f3
EVMC: Small stacks when using EVMC, closes #575 (segfaults)
This patch reduces stack space used with EVM in ENABLE_EVMC=1 mode, from 13 MB
worst case to 550 kB, a 24x reduction.

This completes fixing the "stack problem" and closes #575 (`EVM: Different
segmentation faults when running the test suite with EVMC`).

It also closes #256 (`recursive EVM call trigger unrecoverable stack overflow`).

After this patch, it is possible to re-enable the CI targets which had to be
disabled due to #575.

This change is also a required precursor for switching over to "nearly EVMC" as
the clean and focused Nimbus-internal API between EVM and sync/database
processes, and is also key to the use of Chronos `async` in those processes
when calling the EVM.

(The motivation is the internal interface has to be substantially changed
_anyway_ for the parallel sync and database processes, and EVMC turns out to be
well-designed and well-suited for this.  It provides good separation between
modules, and suits our needs better than our other current interface.  Might as
well use a good one designed by someone else.  EVMC is 98% done in Nimbus
thanks to great work done before by @jangko, and we can use Nimbus-specific
extensions where we need flexibility, including for performance.  Being aligned
with the ecosystem is a useful bonus feature.)

All tests below were run on Ubuntu 20.04 LTS server, x86-64.  This matches one
of the targets that has been disabled for a while in CI in EVMC mode due to
stack overflow crashing the tests, so it's a good choice.

Measurements before
===================

Testing commit `e76e0144 2021-04-22 11:29:42 +0700 add submodules: graphql and
toml-serialization`.

    $ rm -f build/all_tests && make ENABLE_EVMC=1 test
    $ ulimit -S -s 16384 # Requires larger stack than default to avoid crash.
    $ ./build/all_tests 9 | tee tlog
    [Suite] persist block json tests
    ...
    Stack range 38416 depthHigh 3
    ...
    Stack range 13074720 depthHigh 1024
    [OK] tests/fixtures/PersistBlockTests/block1431916.json

These tests use 13.07 MB of stack to run, and so crash with the default stack
limit on Ubuntu Server 20.04 (8MB).  Exactly 12768 bytes per EVM call stack
frame.

    $ rm -f build/all_tests && make ENABLE_EVMC=1 test
    $ ulimit -S -s 16384 # Requires larger stack than default.
    $ ./build/all_tests 7 | tee tlog
    [Suite] new generalstate json tests
        ...
    Stack range 14384 depthHigh 2
        ...
    Stack range 3495456 depthHigh 457
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json
    ...
    Stack range 3709600 depthHigh 485
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json
        ...
    Stack range 7831600 depthHigh 1024
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json

These tests use 7.83MB of stack to run.  About 7648 bytes per EVM call stack
frame.  It _only just_ avoids crashing with the default Ubuntu Server stack
limit of 8 MB.  However, it still crashes on Windows x86-64, which is why the
Windows CI EVMC target is currently disabled.

On Linux where this passes, this is so borderline that it affects work and
testing of the complex storage code, because that's called from the EVM.

Also, this greatly exceeds the default thread stack size.

Measurements after
==================

    $ rm -f build/all_tests && make ENABLE_EVMC=1 test
    $ ulimit -S -s 600 # Because we can!  600k stack.
    $ ./build/all_tests 9 | tee tlog
    [Suite] persist block json tests
    ...
    Stack range 1936 depthHigh 3
    ...
        Stack range 556272 depthHigh 1022
        Stack range 556512 depthHigh 1023
        Stack range 556816 depthHigh 1023
        Stack range 557056 depthHigh 1024
        Stack range 557360 depthHigh 1024
        [OK] tests/fixtures/PersistBlockTests/block1431916.json

    $ rm -f build/all_tests && make ENABLE_EVMC=1 test
    $ ulimit -S -s 600 # Because we can!  600k stack.
    $ ./build/all_tests 7 | tee tlog
    [Suite] new generalstate json tests
        ...
    Stack range 1392 depthHigh 2
        ...
    Stack range 248912 depthHigh 457
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json
    ...
    Stack range 264144 depthHigh 485
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json
        ...
        Stack range 557360 depthHigh 1024
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stStaticCall/static_CallRecursiveBombPreCall.json

For both tests, a satisfying *544 bytes* per EVM call stack frame, and EVM
takes less than 600 kB total.  With other overheads, both tests run in 600 kB
stack total at maximum EVM depth.

We must add some headroom on this for database activity called from the EVM,
and different compile targets.  But it means the EVM itself is no longer a
stack burden.

This is much smaller than the default thread stack size on Linux (2MB), with
plenty of margin.  (Just fyi, it isn't smaller than a _small_ thread stack on
Linux from a long time ago (128kB), and some small embedded C targets.)

This size is well suited to running EVMs in threads.

Further reduction
=================

This patch solves the stack problem.  Windows and Linux 64-bit EVMC CI targets
can be re-enabled, and there is no longer a problem with stack usage.

We can reduce further to ~340 bytes per frame and 350 kB total, while still
complying with EVMC.  But as this involves changing how errors are handled to
comply fully with EVMC, and removing `dispose` calls, it's not worth doing now
while there are other EVMC changes in progress that will have the same effect.

A Nimbus-specific extension will allow us to avoid recursion with EVMC anyway,
bringing bytes per frame to zero.  We need the extension anyway, to support
Chronos `async` which parallel transaction processing is built around.

Interop with non-Nimbus over EVMC won't let us avoid recursion, but then we
can't control the stack frame size either.  To prevent stack overflow in
interop I anticipate using (this method in Aleth)
[6e96ce34e3/libethereum/ExtVM.cpp (L61)].

Smoke test other versions of GCC and Clang/LLVM
===============================================

As all builds including Windows use GCC or Apple's Clang/LLVM, this is just to
verify we're in the right ballpark on all targets.  I've only checked `x86_64`
though, not 32-bit, and not ARM.

It's interesting to see GCC 10 uses less stack.  This is because it optimises
`struct` returns better, sometimes skipping an intermediate copy.  Here it
benefits the EVMC API, but I found GCC 10 also improves the larger stack usage
of the rest of `nimbus-eth1` as well.

Apple clang 12.0.0 (clang-1200.0.26.2) on MacOS 10.15:

- 544 bytes per EVM call stack frame

GCC 10.3.0 (Ubuntu 10.3.0-1ubuntu1) on Ubuntu 21.04:

- 464 bytes per EVM call stack frame

GCC 10.2.0 (Ubuntu 10.2.0-5ubuntu1~20.04) on Ubuntu 20.04 LTS:

- 464 bytes per EVM call stack frame

GCC 11.0.1 20210417 (experimental; Ubuntu 11-20210417-1ubuntu1) on Ubuntu 21.04:

- 8 bytes per EVM call stack frame

GCC 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04) on Ubuntu 20.04 LTS:

- 544 bytes per EVM call stack frame

GCC 8.4.0 (Ubuntu 8.4.0-3ubuntu2) on Ubuntu 20.04 LTS:

- 544 bytes per EVM call stack frame

GCC 7.5.0 (Ubuntu 7.5.0-6ubuntu2) on Ubuntu 20.04 LTS:

- 544 bytes per EVM call stack frame

GCC 9.2.1 20191008 (Ubuntu 9.2.1-9ubuntu2) on Ubuntu 19.10:

- 528 bytes per EVM call stack frame

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-27 05:53:32 +01:00
kdeme 769228418e Remove wakunode and waku rpc code from repository 2020-05-07 20:49:14 +03:00
Jacek Sieka d441687488 bump 2020-04-27 18:40:45 +03:00
Ștefan Talpalaru 1694f7097f
add azure-pipelines.yml
- remove redundant flags from nimbus.nimble
- clean up .appveyor.yml comments
2020-04-16 00:34:07 +02:00
Ștefan Talpalaru 8568ec35ea
enable -march=native by default
plus some drive-by fixes and cosmetic changes
2020-02-13 20:18:27 +01:00
Ștefan Talpalaru 06e0eb80e6
lightweight stack traces
Also converted the top-level nim.cfg into a config.nims, in the process.
2020-02-11 04:33:21 +01:00
kdeme a03d0be31f Add start_network.nim to start different types of networks 2020-01-23 12:39:36 +02:00
kdeme 574c5ec0af Add lightNode and logMetrics options 2020-01-23 12:39:36 +02:00
kdeme bcaf6d03ee Add simple example and readme 2020-01-23 12:39:36 +02:00
kdeme 7b80b313e4 Start implementation of waku node 2020-01-23 12:39:36 +02:00
Ștefan Talpalaru fc50c596d1
tests: reduce RAM usage 2019-09-05 19:26:16 +02:00