Commit Graph

2102 Commits

Author SHA1 Message Date
Jamie Lokier 44744596fd
Hive: Attach current nimbus-eth1 test results (summary)
Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-26 18:30:05 +01:00
Jamie Lokier 480e2d4009
Hive: Improvements to documentation
- Explain some of the prerequisites in more detail (Go version).

- Practical issues around Go version and Docker configuration to make it work.

- Make the first example be one that works with nimbus-eth1.
  (Instead of giving an obscure error message that looks like the user's Docker
  setup is broken).

- Rename to `README.md` per convention.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-26 18:27:29 +01:00
jangko 639674d341
add disableParamFiltering() to both test_generalstate_json and test_blockchain_json
disableParamFiltering() is called in test_all, so its ok when
called via test_all. but when test_generalstate_json or
test_blockchain_json is called independently, and we are using
command line switch, the default param handler from std/unittest
will interfere. hence need to disable it.
2021-04-25 12:20:03 +07:00
jangko a95cec3aa8
add graphql test suite 2021-04-24 11:01:56 +07:00
jangko 103996f6b2
integrate graphql service into nimbus-eth1 2021-04-24 11:01:09 +07:00
jangko 5367b2ddf5
add graphql service files to nimbus-eth1 2021-04-24 10:59:55 +07:00
jangko 34536b0d25
add --graphql and --graphqlbind to cli parser 2021-04-24 10:58:05 +07:00
jangko 3dfc1501f0
modify estimateGas in rpc_utils.nim and it can be reusable for graphql too 2021-04-24 10:56:22 +07:00
jangko 82d4f6a2d4
fixes getReceipts proc signature in db_chain.nim
instead of using header as input param, now getReceipts using
receiptRoot hash, the intention is clearer and less data passed around
when we only using receiptRoot instead of whole block header.
2021-04-24 10:51:05 +07:00
jangko 1728dd8d54
fix getTransactionCount bug in db_chain.nim
getTransactionCount will always return 0 because the txCount
variable is not returned
2021-04-24 10:47:37 +07:00
jangko 1ca4b63c3a
bump chronicles, serialization and json-serialization
recent nim-serialization API changes already made it's way into
toml-serialization. and with the addition of newest toml-serialization
along with it's bugfixes, those submodules mentioned above
become out of date because toml-serialization need newer version of
nim-serialization.
2021-04-24 10:36:48 +07:00
Jordan Hrycaj e02c6d4c3d renamed computation.nim, memory.nim, utils_numeric.nim, interpreter.nim => v2*.nim
why:
  these files provide part of the externally accessible interface
  provided by vm_cpmputation.nim, vm_internals.nim. so the
  new filename indicates that the source code belongs to vm2 (rather
  than vm).
2021-04-23 14:04:06 +03:00
Jordan Hrycaj 2ca9621799 renamed message.nim, precompiles.nim, gas_costs.nim => v2*.nim
why:
  these files provide part of the externally accessible interface
  provided by vm_message.nim, vm_precompile.nim, vm_gas_cost.nim. so the
  new filename indicates that the source code belongs to vm2 (rather
  than vm).
2021-04-23 14:04:06 +03:00
Jordan Hrycaj f159e67bba renamed states*.nim => v2states*.nim
why:
  these files provide part of the externally accessible interface
  provided by vm_state*.nim. so the new filename indicates that the
  source code belongs to vm2 (rather than vm).
2021-04-23 14:04:06 +03:00
Jordan Hrycaj 7b6767c4a3 renamed types.nim, vm_fork.nim, opcode_values.nim => v2*.nim
why:
  these files provide part of the externally accessible interface
  provided by vm_types*.nim. so the new filename indicates that the
  source code belongs to vm2 (rather than vm).
2021-04-23 14:04:06 +03:00
Jordan Hrycaj cf2d771c4d remove evmc code from vm2
why:
  handled by original vm
2021-04-23 14:04:06 +03:00
Jordan Hrycaj b7bf84a71f added compiler flag sentinels to vm2 headers
why:
  making sure that deep links into vm2 sources are configured properly. it
  is intended that only the vm_*.nim interface headers are allowed to
  source files in vm2. the sentinels just protect from coding errors.
2021-04-23 14:04:06 +03:00
Jordan Hrycaj b4f8450968 provide identical copy of vm folder => vm2, activated by make flag ENABLE_VM2=1
why:
  vm2 enabled by ENABLE_VM2=1 behaves as vm without ENABLE_EVMC=1 until
  it doesn't in some future fatch set. this leaves some wiggle room
  to work on a vm copy without degrading the original implementation.

details:
  + additional make flag ENABLE_VM2=1 (or ENABLE_VM2=0 to explicitely disable)
  + when both flags ENABLE_EVMC=1 and ENABLE_VM2=1 are present, the former
    flag ENABLE_EVMC=1 takes precedence, this is implemented at the NIM
    compiler level for -d:evmc_enabled and -d:vm2_enabled
2021-04-23 14:04:06 +03:00
jangko e76e0144d0
add submodules: graphql and toml-serialization 2021-04-22 11:29:42 +07:00
Jamie Lokier 085661c24f
EVM: Eliminate recursion entirely
This patch eliminates recursion entirely from the EVM when ENABLE_EVMC=0.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-20 16:12:45 +01:00
Jamie Lokier e0f5a5e7f0 Sync: Script to assist with sync testing
Script to run Nimbus-eth1 syncing to any of the networks that Geth supports
by name, connecting to a single dedicated peer for that network.

All protocols are disabled other than the minimum we need for block sync.

This is very helpful for debugging and improving pipelined sync, and for
improving the database and other processing, even though p2p sync is the
endgame.

- Discovery protocols are turned off
- NAT hole punching protocols are turned off
- Whisper protocol is turned off (`--protocols:eth`)
- The only connection is to a single, active Geth for that network.
- Each network's data is stored in a different location to avoid conflicts.
- Each network is accessed by binding to a different port locally,
  so we can run several at the same time.
- Log level is set to `TRACE` because we can read them when we're not
  swamped with discovery messages.  Sync isn't fast enough yet.
2021-04-20 14:56:53 +01:00
Ștefan Talpalaru 295c1409f6
also use "-fno-omit-frame-pointer" on Windows (#584)
now that we no longer use the buggy gcc-8.1.0 in CI
2021-04-14 15:03:13 +02:00
Jamie Lokier 8211db1ea8
EVM: Small patch that reduces EVM stack usage to almost nothing
There's been a lot of talk about the Nimbus EVM "stack problem".  I think we
assumed changing it would require big changes to the interpreter code, touching
a lot of functions.

It turned out to be a low hanging fruit.

This patch solves the stack problem, but hardly touches anything.  The change
in EVM stack memory is from 13 MB worst case to just 48 kB, a 250x reduction.

I've been doing work on the database/storage/trie code.  While looking at the
API between the EVM and the database/storage/trie, this stack patch stood out
and made itself obvious.  As it's tiny, rather than more talk, here it is.

Note: This patch is intentionally small, non-invasive, and hopefully easy to
understand, so that it doesn't conflict with other work done on the EVM, and
can easily be grafted into any other EVM structure.

Motivation
==========

- We run out of space and crash on some targets, unless the stack limit is
  raised above its default.  Surprise segmentation faults are unhelpful.

- Some CI targets have been disabled for months due to this.

- Because usage borders on the system limits, when working on
  database/storage/trie/sync code (called from the EVM), segmentation faults
  occur and are misleading.  They cause lost time due to thinking there's a
  crash bug in the code being worked on, when there's nothing wrong with it.

- Sometimes unrelated, trivial code changes elsewhere trigger CI test failures.
  It looks like abrupt termination.  A simple, recent patch was crashing in
  `make test` even though it was a trivial refactor.  Turns out it pushed the
  stack over the edge.

- A large stack has to be scanned by the Nim garbage collector sometimes.
  Larger stack means slower GC and memory allocation.

- The structure of this small patch suggests how to weave async into the EVM
  with almost no changes to the EVM, and no async transformation overhead.

- The patch seemed obvious when working on the API between EVM and storage.

Measurements before
===================

All these tests were run on Ubuntu 20.04 server, x86-64.  This is one of the
targets that has been disabled for a while in CI in EVMC mode due to crashing,
and excessive stack usage is the cause.

Testing commit 0c34a8e3 `2021-04-08 17:46:00 +0200 CI: use MSYS2 on Windows`.

    $ rm -f build/all_tests && make ENABLE_EVMC=1 test
    $ ulimit -S -s 16384 # Requires larger stack than default to avoid crash.
    $ ./build/all_tests 9 | tee tlog
    [Suite] persist block json tests
    ...
	Stack range 38496 depthHigh 3
    ...
    Stack range 13140272 depthHigh 1024
    [OK] tests/fixtures/PersistBlockTests/block1431916.json

These tests use 13.14 MB of stack to run, and so crash with the default stack
limit on Ubuntu Server 20.04 (8MB).  Exactly 12832 bytes per EVM call stack
frame.  It's interesting to see some stack frames take a bit more.

    $ rm -f build/all_tests && make ENABLE_EVMC=1 test
    $ ulimit -S -s 16384 # Requires larger stack than default.
    $ ./build/all_tests 7 | tee tlog
    [Suite] new generalstate json tests
	...
	Stack range 15488 depthHigh 2
	...
	Stack range 3539312 depthHigh 457
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json
    ...
	Stack range 3756144 depthHigh 485
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json
	...
	Stack range 7929968 depthHigh 1024
     [OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json

These tests use 7.92MB of stack to run.  About 7264 bytes per EVM call stack
frame.  It _only just_ avoids crashing with the default Ubuntu Server stack
limit of 8 MB.  However, it still crashes on Windows x86-64, which is why the
CI target is currently disabled.

On Linux where this passes, this is so borderline that it affects work and
testing of storage and sync code, because that's called from the EVM.  Which
was a motivation for dealing with the stack instead of letting this linger.

Also, this stack greatly exceeds the default thread stack size.

    $ rm -f build/all_tests && make ENABLE_EVMC=0 test
    $ ulimit -S -s 16384 # Requires larger stack than default to avoid crash.
    $ ./build/all_tests 9 | tee tlog
    [Suite] persist block json tests
    ...
    Stack range 33216 depthHigh 3
    ...
    Stack range 11338032 depthHigh 1024
    [OK] tests/fixtures/PersistBlockTests/block1431916.json

These tests use 11.33 MB stack to run, and so crash with a default stack limit
of 8MB.  Exactly 11072 bytes per EVM call stack frame.  It's interesting to see
some stack frames take a bit more.

    $ rm -f build/all_tests && make ENABLE_EVMC=0 test
    $ ulimit -S -s 16384 # Requires larger stack than default.
    $ ./build/all_tests 7 | tee tlog
    [Suite] new generalstate json tests
	...
    Stack range 10224 depthHigh 2
	...
    Stack range 2471760 depthHigh 457
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json
    ...
    Stack range 2623184 depthHigh 485
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json
	...
    Stack range 5537824 depthHigh 1024
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json

These tests use 5.54 MB of stack to run, and avoid crashing on with a default
stack limit of 8 MB.  About 5408 bytes per EVM call stack frame.

However, this is uncomfortably close to the limit, as the stack frame size is
sensitive to changes in the code.

Also, this stack greatly exceeds the default thread stack size.

Measurements after
==================

(This patch doesn't address EVMC mode, which is not our default.  EVMC stack
usage remains about the same.  EVMC mode is addressed in another tiny patch.)

    $ rm -f build/all_tests && make ENABLE_EVMC=0 test
    $ ulimit -S -s 80 # Because we can!  80k stack.
    $ ./build/all_tests 9 | tee tlog
    [Suite] persist block json tests
    ...
    Stack range 496 depthHigh 3
    ...
    Stack range 49504 depthHigh 1024
    [OK] tests/fixtures/PersistBlockTests/block1431916.json

    $ rm -f build/all_tests && make ENABLE_EVMC=0 test
    $ ulimit -S -s 72 # Because we can!  72k stack.
    $ ./build/all_tests 7 | tee tlog
    [Suite] new generalstate json tests
	...
    Stack range 448 depthHigh 2
	...
    Stack range 22288 depthHigh 457
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json
    ...
    Stack range 23632 depthHigh 485
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json
	...
    Stack range 49504 depthHigh 1024
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json

For both tests, a satisfying *48 bytes* per EVM call stack frame, and EVM takes
not much more than 48 kB.  With other overheads, both tests run in 80 kB stack
total at maximum EVM depth.

We must add some headroom on this for database activity called from the EVM,
and different compile targets.  But it means the EVM itself is no longer a
stack burden.

This is much smaller than the default thread stack size on Linux (2MB), with
plenty of margin.  It's even smaller than Linux from a long time ago (128kB),
and some small embedded C targets.  (Just fyi, though, some JVM environments
allocated just 32 kB to thread stacks.)

This size is also well suited to running EVMs in threads, if that's useful.

Subtle exception handling and `dispose`
=======================================

It is important that each `snapshot` has a corresponding `dispose` in the event
of an exception being raised.  This code does do that, but in a subtle way.

The pair of functions `execCallOrCreate` and `execCallOrCreateAux` are
equivalent to the following code, where you can see `dispose` more clearly:

    proc execCallOrCreate*(c: Computation) =
      defer: c.dispose()
      if c.beforeExec():
        return
      c.executeOpcodes()
      while not c.continuation.isNil:
        c.child.execCallOrCreate()
        c.child = nil
        (c.continuation)()
        c.executeOpcodes()
      c.afterExec()

That works fine, but only reduces the stack used to 300-700 kB instead of 48 kB.

To get lower we split the above into separate `execCallOrCreate` and
`execCallOrCreateAux`.  Only the outermost has `defer`, and instead of handling
one level, it walks the entire `c.parent` chain calling `dispose` if needed.
The inner one avoids `defer`, which greatly reduces the size of its stackframe.

`c` is a `var` parameter, at each level of recursion.  So the outermost proc
sees the temporary changes made by all inner calls.  This is why `c` is updated
and the `c.parent` chain is maintained at each step.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-13 23:35:26 +01:00
Ștefan Talpalaru 0c34a8e365
CI: use MSYS2 on Windows 2021-04-09 17:19:40 +07:00
Jacek Sieka 3147df0dcd
switch to chronos metrics, remove insecure (#580)
* switch to chronos metrics, remove insecure

See https://github.com/status-im/nimbus-eth2/pull/2468

also fixes pcre linking for real, and adds some random build flags that
help nimbus-eth2 stay afloat

* fix help

* don't omit frame pointers on windows
2021-04-09 09:26:06 +02:00
Jamie Lokier 8a806da600
Bugfix: Fix blockchain tests: It was quietly skipping 4654 of them
The "new block chain json tests" were being skipped on Linux, but silently so
that CI didn't notice.  These are a significant part of the Ethereum test suite.

See the missing output from `make test`, also visible in CI logs for Linux
targets (prior to this commit):

      [OK] tests/fixtures/eth_tests/TransactionTests/ttGasPrice/TransactionWithGasPriceOverflow.json
      [OK] tests/fixtures/eth_tests/TransactionTests/ttGasPrice/TransactionWithHighGasPrice.json

    [Suite] new block chain json tests
            <-- nothing here

    [Suite] Fork ID tests
      [OK] MainNet
      [OK] RopstenNet

Commit 3d468a7 (`fixes path pointing to eth_tests`) renamed the JSON fixture
source directoryf in the witness-builder tests but not the regular blockchain
tests.  As a result, searching for JSON test files yielded zero results.  To
make this less likely in future, zero results is now an error.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-08 16:40:53 +01:00
Jamie Lokier e4bd5532c5
Tests: Rename incorrect case `newBlockChainTests.md` file in repo
Commit 3d468a7 (`fixes path pointing to eth_tests`) renamed a test output
file subtly, changing just the capitalisation:

    - jsonTest(newFolder, "newBlockChainTests", testFixture, skipNewBCTests)
    + jsonTest(newFolder, "newBlockchainTests", testFixture, skipNewBCTests)

However, the file checked into the repo continued to be the old name, uppercase
`C`.  Of course, Mac and Windows have a sort of case insensitivity and Linux
does not.

This meant, on Linux, both files were created, test differences didn't make
themselves visible in `git diff`, and the repo would not get updates with
changed test output.  On Mac and Windows it worked.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-08 10:27:11 +01:00
Jordan Hrycaj 2eb46ca221
Merge pull request #573 from status-im/feature/isolate-evms
Feature/isolate evms
2021-04-08 08:00:34 +01:00
jangko bbf5790b5e
fixes docker address and port binding in hive entrypoint script 2021-04-08 08:52:49 +07:00
andri lim 9823b3d132
fixes nimbus hive client code 2021-04-08 08:52:41 +07:00
jangko a923016a12
allow missing chainId in chain config 2021-04-08 08:52:41 +07:00
jangko 1801317208
dealing with missing 'code' keys in customNetPrealloc 2021-04-08 08:52:40 +07:00
jangko fb7d5b5319
fixes customNetPrealloc in genesis.nim: dealing with missing keys in genesis.json 2021-04-08 08:52:40 +07:00
jangko 2befeceeac
add instructions how to run hive and nimbus in hive 2021-04-08 08:52:40 +07:00
jangko e237e21372
add hive client files 2021-04-08 08:52:40 +07:00
jangko a7344b13d1
add debugging tool related to hive: extract_consensus_data 2021-04-08 08:52:30 +07:00
Jamie Lokier 6d30b349b3
Bump nim-eth for: Kademlia crash when trying to sync with testnet or mainnet (#489)
Prior to this commit, when using discv4 (Kademlia) to find peers, there is a
crash after a few minutes.  It occurs for most of us on Eth1 mainnet, and
everyone on Ropsten.

Tested for about 4 days constant operation on Ropsten.  The crash which occured
every few minutes no longer occurs, and discv4 keeps working.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-07 18:09:22 +01:00
Jordan Hrycaj dfc93a74ad
moved validateTransaction() to executor
why:
  not part of VM (see andri's requested change at #573)
2021-04-07 15:13:28 +01:00
Jordan Hrycaj a72f6ed6e0
merging back into original HEAD
why:
  temporary github problem caused macos crash on ci
2021-04-06 15:27:03 +01:00
kdeme f34431e2d4
Bump nim-eth and nim-chronos and fix exception effects 2021-04-02 19:55:21 +02:00
Jordan Hrycaj 827b8c9c81
reset explicit import paths for local modules
why:
  it was convenient to have relocatable source modules when writing the
  vm interface wrappers. this patch moves it back to the standard.

also:
  there are no deep links into the vm folder anymore which leaves some
  room for manoeuvring inside
2021-04-01 12:53:22 +01:00
Jacek Sieka ebb65e9c20
Compile without pcre (#574) 2021-04-01 09:36:13 +02:00
Jordan Hrycaj 05bd635da2
fix formatting 2021-03-31 18:18:00 +01:00
Jordan Hrycaj c8582583ef
merge vm_memory, vm_interpreter, and vm_utils_numeric => vm_internals
why:
  currently used for tests only
2021-03-31 18:15:27 +01:00
Jordan Hrycaj 00ba7a2718
merge vm_forks and vm_opcode_values => vm_type2
why:
  all types, but they cannot be merged int vm_types because of a circular
  dependency.
2021-03-31 17:53:15 +01:00
Jordan Hrycaj 9e365734e6
renamed nvm_ prefixed modules to its original names
why:
  the nvm_ prefix was used inside the vm folder to hide them temporarily
  from the outside world while writing export wrappers. now all
  functionality is accessed via vm_*, rather than vm/* imports.

todo:
  at a later stage the import headers of the vm modules need to get fixed
  to meet style guide standards (as jacek kindly pointed out.)
2021-03-31 17:19:54 +01:00
Jordan Hrycaj 474bd9e910
expanded nvm_interpreter
details:
  explicit symbol exports rather than wholesale module names
2021-03-31 16:49:11 +01:00
Jordan Hrycaj 7c28d5d362
provide vm_utils_numeric as import/export wrapper
details:
  moved original vm/interpreter/utils/utils_numeric.nim => vm/interpreter/utils/utils_numeric.nim
2021-03-31 16:49:07 +01:00
Jordan Hrycaj 99568c9b46
provide vm_opcode_values as import/export wrapper
details:
  moved original vm/interpreter/opcode_values.nim => vm/interpreter/nvm_opcode_values.nim
2021-03-31 16:49:03 +01:00
Jordan Hrycaj cf63b9b03f
provide vm_memory as import/export wrapper
details:
  moved original vm/memory.nim => vm/nvm_memory.nim
2021-03-31 16:48:44 +01:00