Commit Graph

440 Commits

Author SHA1 Message Date
bmoo b09ad5cacb
code cleanup removed unused imports 2021-08-18 10:35:36 +07:00
Jamie Lokier a7b40b0762
EVM: Use the EVMC calls for EIP-2929 access-list and refactor in EVM
Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-08-11 19:47:38 +07:00
Jamie Lokier 74f53c7761
EVMC: Add missing EIP-2929 (Berlin) functions to EVMC host
The update for London (EIP-1559) in 1cdb30df ("bump nim-emvc with evmc revision
8.0.0 to 9.0.0") really bumped EVMC ABI version from 7.5 up to 9.

In other words, it skipped Berlin, going direct from Istanbul to London.

That was accompanied by EVMC changes in 05e9b891 ("EIP-3198: add baseFee op
code in nim-evm"), which added the API changes needed for London.

But the missing Berlin functions weren't added in the move to London.

As a result, our EVMC host became incompatible with Berlin, London, and really
all revisions of the ABI, and if a third party EVM was loaded, it crashed.

This commit adds the missing Berlin host support, and makes our ABI
binary-compatible with real EVMC again.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-08-11 19:47:34 +07:00
Jamie Lokier 11f03a1846
Transaction: EVMC fix, `CREATE2` salt is a 256-bit blob not a number
This changes fixes a bug in `CREATE2` ops when used with EVMC.
Because it changes the salt type, it affects non-EVMC code as well.

The salt was passed through EVMC with the wrong byte order, although this went
unnoticed as the Nimbus host flipped the byte order before using it.

This was found when running Nimbus with third-party EVM,
["evmone"](https://github.com/ethereum/evmone).

There are different ways to remedy this.

If treated as a number, Nimbus EVM would byte-flip the value when calling EVMC,
then Nimbus host would flip the received value.  Finally, it would be flipped a
third time when generating the address in `generateSafeAddress`.  The first two
flips can be eliminated by negotiation (like other numbers), but there would
always be one flip.

As a bit pattern, Nimbus EVM would flip the same way it does when dealing with
hashes on the stack (e.g. with `getBlockHash`).  Nimbus host wouldn't flip at
all - and when using third-party EVMs there would be no flips in Nimbus.

Because this value is not for arithmetic, any bit pattern is valid, and there
shouldn't be any flips when using a third-party EVM, the bit-pattern
interpretation is favoured.  The only flip is done in Nimbus EVM (and might be
eliminated in an optimised version).

As suggested, we'll define a new "opaque 256 bits" type to hold this value.
(Similar to `Hash256`, but the salt isn't necessarily a hash.)

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-08-05 10:35:52 +01:00
Jordan Hrycaj dc8ef09727
fix CI failing 2021-08-05 12:27:14 +07:00
Jordan Hrycaj 4713bd4cf4
#768 Moved/re-implemented ecRecover() from Clique sources to utils/ec_recover
why:
  The same functionality was differently implemented in one or the
  other form.

details:
  Caching and non-caching variants available
2021-08-05 12:27:10 +07:00
Jamie Lokier ab9067133c
Tracing: Remove some trace messages that occur a lot during sync
Disable some trace messages which appeared a lot in the output and probably
aren't so useful any more, when block processing is functioning well at high
speed.

Turning on the trace level globally is useful to get a feel for what's
happening, but only if each category is kept to a reasonable amount.

As well as overwhelming the output so that it's hard to see general activity,
some of these messages happen so much they severely slow down processing.  Ones
called every time an EVM opcode uses some gas are particularly extreme.

These messages have all been chosen as things which are probably not useful any
more (the relevant functionality has been debugged and is tested plenty).

These have been commented out rather than removed.  It may be that turning
trace topics on/off, or other selection, is a better longer term solution, but
that will require better command line options and good defaults for sure.
(I think higher levels `tracev` and `tracevv` levels (extra verbose) would be
more useful for this sort of deep tracing on request.)

For now, enabling `--log-level:TRACE` on the command line is quite useful as
long as we keep each category reasonable, and this patch tries to keep that
balance.

- Don't show "has transactions" on virtually every block imported.
- Don't show "Sender" and "txHash" lines on every transaction processed.
- Don't show "GAS CONSUMPTION" on every opcode executed", this is way too much.
- Don't show "GAS RETURNED" and "GAS REFUND" on each contract call.
- Don't show "op: Stop" on every Stop opcode, which means every transaction.
- Don't show "Insufficient funds" whenever a contract can't call another.
- Don't show "ECRecover", "SHA256 precompile", "RIPEMD160", "Identity"
  or even "Call precompile" every time a precompile is called.  These are
  very well tested now.
- Don't show "executeOpcodes error" whenever a contract returns an error.
  (This is changed to `trace` too, it's a normal event that is well tested.)

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-27 14:12:55 +01:00
jangko 8482cb3ed3
EIP-3541: Fixes typo, 0xFE -> 0xEF 2021-06-30 20:44:34 +07:00
jangko db8988fe64
EIP-1559: Fee market change for ETH 1.0 chain
Transaction and BlockHeader already updated in nim-eth repo
to support EIP-1559

EIP-1559 header validation and gasLimit validation
already implemented in previous commit

This commit deals with block validation:
- Effective gasPrice per EIP-1559
- new miner reward based on priorityFee
2021-06-30 20:30:39 +07:00
jangko 05d905b136
EIP-3529: Replace SSTORE_CLEARS_SCHEDULE
SSTORE_CLEARS_SCHEDULE or FeeSchedule[RefundsClear] in evm
have initial value of 15_000 when introduced by EIP-2200.

EIP-2200 also set new value for SSTORE_RESET_GAS
from 5000 to to 5000 - COLD_SLOAD_COST

Now with EIP-3529, SSTORE_CLEARS_SCHEDULE beecome
SSTORE_RESET_GAS + ACCESS_LIST_STORAGE_KEY_COST

or 5000 - COLD_SLOAD_COST + ACCESS_LIST_STORAGE_KEY_COST
of 5000 - 2100 + 1900 = 4800
2021-06-29 07:37:17 +07:00
jangko 8982e6c649
EIP-3529: Remove the SELFDESTRUCT refund.
- remove it from both nim-evm and nim-evm2
2021-06-29 07:37:17 +07:00
jangko e08c9ef2d9
EIP-3541: Reject new contracts starting with the 0xEF byte 2021-06-29 07:36:56 +07:00
jangko 05e9b891f0
EIP-3198: add baseFee op code in nim-evm 2021-06-29 07:35:16 +07:00
jangko 5159ad7aac
preparation for London hard fork
This preparation is needed for subsequent
EIPs included in London.

- Add London to Fork enum
- Block number to fork
- Parsing London fork in chain config
- Prepare gas costs table for London
- Prepare EVM opcode dispatcher for London
- Block rewards for London
- Prepare hive script for London
2021-06-29 07:34:45 +07:00
Jamie Lokier df71c8bec9
EVMC: Disable byte-endian conversion of 256-bit values on EVM side
We'll re-enable endian conversions based on a negotiated run-time option later,
but for now let's remove one complication to testing the new EVMC paths, and
also gain a little performance.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 18:29:39 +01:00
Jamie Lokier 7c90d8de70
EVM: Remove `vm_forks` everywhere, use common forks list instead
The common forks list was already used, redirected via `vm_forks` for
historical compatibility.  Remove the old `vm_forks` now and divert all imports
to the common forks list outside the EVM.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 15:36:31 +01:00
Jamie Lokier 05bc174bef
Forks: Use a common fork list outside the EVMs
Many places outside the EVM use `Fork` and the fork list, and in general we
want progressively fewer dependencies on EVM internal types and files.

This may prove to be a temporary location, especially when we implement
issue #640.  But it's a fine temporary location if so.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 15:36:31 +01:00
Jamie Lokier b3a788c7ce
Transaction: Move contract address generation outside the EVM
The current EVM generates its own new contract addresses, and this is why there
are separate `msg.contractAddress` and `msg.codeAddress` fields in the
computation start message.

In EVMC, account updates are only allowed on the host side, including contract
generation, and the start message has one destination field, `msg.destination`.
The EVM cannot select addresses, only use them.  It's a sensible design.

The difference makes the current EVM incompatible with EVMC and its message
format, so this patch corrects the difference.  It moves contract address
generation to the host side.  This simplifies the EVM and its API a little.

(As an API change, this is incompatible with vm2, so it's guarded under
`evmc_enabled` to allow vm2 to continue to build and run at this time.  This is
also why there are fewer deletions than would otherwise be expected.)

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 15:36:30 +01:00
Jamie Lokier 775231eef1
EVM: Apply EIP-6 in the code (affects both vm and vm2)
The rationale in EIP-6[1] for changing names to `selfDestruct` applies to code
as much as it does to specs.  Also, Ethereum uses the new names consistently,
so it's useful for our code to match the terms used in later EIP specs and
testsuite entries.

This change is straightforward, and is a prerequisite for patches to come that
do things with the `selfDestruct` fields.

[1] https://eips.ethereum.org/EIPS/eip-6
Hudson Jameson, "EIP-6: Renaming SUICIDE opcode," Ethereum Improvement
Proposals, no. 6, November 2015.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 15:36:30 +01:00
Jamie Lokier 537cac1bf5
EVM: Move where `continuation` is cleared to fix a potential stall
This fixes a bug spotted by @mjfh that was introduced by commit 2a7ccceb:

    try:
      if not c.continuation.isNil:
        (c.continuation)()
        c.continuation = nil
      c.selectVM(fork)
    except CatchableError as e:
      ...

The call to `(c.continuation)()` was moved by 2a7ccceb inside the `try` so
that, like all the Op functions do already, if the continuation raises, the
interpreter's general catch turns the exception into a an error status result.

But if the continuation raises an exception, `continuation` is not cleared in
the next line, and at the next resumption the continuation is called again.
It may loop doing this.

This doesn't currently happen because the continuations don't really raise, but
it's still a correctness issue.

This fix also allows a continuation to spawn a second continuation, if it
encounters a second suspension point.  This also doesn't happen currently,
but the pattern will become useful with async EVM.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-05-27 12:16:37 +01:00
jangko a0d10f5728
drop PublicNetwork enum usage and replace it with NetworkId
we cannot limit the `--networkid` switch to values available in
`PublicNetwork` enum. it should able to accept very wide range of
custom NetworkId.
2021-05-20 14:04:16 +07:00
jangko 76543da456
disable EIP-2537: Precompile for BLS12-381 curve operations
reason: not included in berlin hard fork

but we keep the code around, for future inclusion
2021-05-17 01:29:03 +07:00
jangko 3ccc4642f2
disable EIP-2315: Simple Subroutines for the EVM
reason: not included in berlin hard fork
2021-05-17 01:29:03 +07:00
jangko 6fc3df637c
reenable EIP-2565: modExp gas cost
now it's officially included in berlin hard fork
2021-05-17 01:28:31 +07:00
jangko 79044f1e92
eip2718: test_blockchain_json pass test 2021-05-15 18:09:35 +07:00
jangko f6a0e4bcbd
fixes wrong usage of `chainId` in places where it should be networkId
fixes #643
2021-05-12 09:45:09 +07:00
Jamie Lokier 4187eb1959
Transaction: Prepare txRefundGas to support txCallEvm
There's only one call left to `refundGas(Transaction, ...)`, and the
similarity to the tail of `rpcEstimateGas` is obvious.

Gather this into `call_evm`: `refundGas` -> `txRefundGas`.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-05-03 19:51:20 +01:00
Jamie Lokier 52fd8b8129
Transaction: Prepare txSetupComputation to support txCallEvm
After recent changes, there's only one call left to `setupComputation`, and
it's just a variant like `rpcSetupComputation` but for transaction processing.
The similarity to `rpcSetupComputation` is obvious.

Gather this into `call_evm`: `setupComputation` -> `txSetupComputation`.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-05-03 19:51:20 +01:00
Jamie Lokier 2a7ccceb3e
EVM: Make continuation exceptions behave as they did before
The account database code is not supposed to raise exceptions in the EVM, and
the behaviour is not well defined if it does.  It isn't compliant with EVMC
spec either.  But that will be dealt with properly when the account state-cache
is dealt with, as there is some work to be done on it.

Meanwhile, if it raises in code under `chainTo` and then `(continuation)()`,
the behaviour was changed slightly by the stack-shrink patches.

Before those patches, an exception after the recursion-point was converted to
`c.setError` "Opcode Dispatch Error" in `executeOpcodes.  After, it would
propagate out, a different behaviour.  (It still correctly walked the chain of
`c.dispose()` calls to clean up.)

It's easy to restore the original behaviour just by moving the continuation
call, so let's do that.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-30 11:32:42 +07:00
Jamie Lokier a3c8a5c3f3
EVMC: Small stacks when using EVMC, closes #575 (segfaults)
This patch reduces stack space used with EVM in ENABLE_EVMC=1 mode, from 13 MB
worst case to 550 kB, a 24x reduction.

This completes fixing the "stack problem" and closes #575 (`EVM: Different
segmentation faults when running the test suite with EVMC`).

It also closes #256 (`recursive EVM call trigger unrecoverable stack overflow`).

After this patch, it is possible to re-enable the CI targets which had to be
disabled due to #575.

This change is also a required precursor for switching over to "nearly EVMC" as
the clean and focused Nimbus-internal API between EVM and sync/database
processes, and is also key to the use of Chronos `async` in those processes
when calling the EVM.

(The motivation is the internal interface has to be substantially changed
_anyway_ for the parallel sync and database processes, and EVMC turns out to be
well-designed and well-suited for this.  It provides good separation between
modules, and suits our needs better than our other current interface.  Might as
well use a good one designed by someone else.  EVMC is 98% done in Nimbus
thanks to great work done before by @jangko, and we can use Nimbus-specific
extensions where we need flexibility, including for performance.  Being aligned
with the ecosystem is a useful bonus feature.)

All tests below were run on Ubuntu 20.04 LTS server, x86-64.  This matches one
of the targets that has been disabled for a while in CI in EVMC mode due to
stack overflow crashing the tests, so it's a good choice.

Measurements before
===================

Testing commit `e76e0144 2021-04-22 11:29:42 +0700 add submodules: graphql and
toml-serialization`.

    $ rm -f build/all_tests && make ENABLE_EVMC=1 test
    $ ulimit -S -s 16384 # Requires larger stack than default to avoid crash.
    $ ./build/all_tests 9 | tee tlog
    [Suite] persist block json tests
    ...
    Stack range 38416 depthHigh 3
    ...
    Stack range 13074720 depthHigh 1024
    [OK] tests/fixtures/PersistBlockTests/block1431916.json

These tests use 13.07 MB of stack to run, and so crash with the default stack
limit on Ubuntu Server 20.04 (8MB).  Exactly 12768 bytes per EVM call stack
frame.

    $ rm -f build/all_tests && make ENABLE_EVMC=1 test
    $ ulimit -S -s 16384 # Requires larger stack than default.
    $ ./build/all_tests 7 | tee tlog
    [Suite] new generalstate json tests
        ...
    Stack range 14384 depthHigh 2
        ...
    Stack range 3495456 depthHigh 457
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json
    ...
    Stack range 3709600 depthHigh 485
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json
        ...
    Stack range 7831600 depthHigh 1024
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json

These tests use 7.83MB of stack to run.  About 7648 bytes per EVM call stack
frame.  It _only just_ avoids crashing with the default Ubuntu Server stack
limit of 8 MB.  However, it still crashes on Windows x86-64, which is why the
Windows CI EVMC target is currently disabled.

On Linux where this passes, this is so borderline that it affects work and
testing of the complex storage code, because that's called from the EVM.

Also, this greatly exceeds the default thread stack size.

Measurements after
==================

    $ rm -f build/all_tests && make ENABLE_EVMC=1 test
    $ ulimit -S -s 600 # Because we can!  600k stack.
    $ ./build/all_tests 9 | tee tlog
    [Suite] persist block json tests
    ...
    Stack range 1936 depthHigh 3
    ...
        Stack range 556272 depthHigh 1022
        Stack range 556512 depthHigh 1023
        Stack range 556816 depthHigh 1023
        Stack range 557056 depthHigh 1024
        Stack range 557360 depthHigh 1024
        [OK] tests/fixtures/PersistBlockTests/block1431916.json

    $ rm -f build/all_tests && make ENABLE_EVMC=1 test
    $ ulimit -S -s 600 # Because we can!  600k stack.
    $ ./build/all_tests 7 | tee tlog
    [Suite] new generalstate json tests
        ...
    Stack range 1392 depthHigh 2
        ...
    Stack range 248912 depthHigh 457
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json
    ...
    Stack range 264144 depthHigh 485
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json
        ...
        Stack range 557360 depthHigh 1024
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stStaticCall/static_CallRecursiveBombPreCall.json

For both tests, a satisfying *544 bytes* per EVM call stack frame, and EVM
takes less than 600 kB total.  With other overheads, both tests run in 600 kB
stack total at maximum EVM depth.

We must add some headroom on this for database activity called from the EVM,
and different compile targets.  But it means the EVM itself is no longer a
stack burden.

This is much smaller than the default thread stack size on Linux (2MB), with
plenty of margin.  (Just fyi, it isn't smaller than a _small_ thread stack on
Linux from a long time ago (128kB), and some small embedded C targets.)

This size is well suited to running EVMs in threads.

Further reduction
=================

This patch solves the stack problem.  Windows and Linux 64-bit EVMC CI targets
can be re-enabled, and there is no longer a problem with stack usage.

We can reduce further to ~340 bytes per frame and 350 kB total, while still
complying with EVMC.  But as this involves changing how errors are handled to
comply fully with EVMC, and removing `dispose` calls, it's not worth doing now
while there are other EVMC changes in progress that will have the same effect.

A Nimbus-specific extension will allow us to avoid recursion with EVMC anyway,
bringing bytes per frame to zero.  We need the extension anyway, to support
Chronos `async` which parallel transaction processing is built around.

Interop with non-Nimbus over EVMC won't let us avoid recursion, but then we
can't control the stack frame size either.  To prevent stack overflow in
interop I anticipate using (this method in Aleth)
[6e96ce34e3/libethereum/ExtVM.cpp (L61)].

Smoke test other versions of GCC and Clang/LLVM
===============================================

As all builds including Windows use GCC or Apple's Clang/LLVM, this is just to
verify we're in the right ballpark on all targets.  I've only checked `x86_64`
though, not 32-bit, and not ARM.

It's interesting to see GCC 10 uses less stack.  This is because it optimises
`struct` returns better, sometimes skipping an intermediate copy.  Here it
benefits the EVMC API, but I found GCC 10 also improves the larger stack usage
of the rest of `nimbus-eth1` as well.

Apple clang 12.0.0 (clang-1200.0.26.2) on MacOS 10.15:

- 544 bytes per EVM call stack frame

GCC 10.3.0 (Ubuntu 10.3.0-1ubuntu1) on Ubuntu 21.04:

- 464 bytes per EVM call stack frame

GCC 10.2.0 (Ubuntu 10.2.0-5ubuntu1~20.04) on Ubuntu 20.04 LTS:

- 464 bytes per EVM call stack frame

GCC 11.0.1 20210417 (experimental; Ubuntu 11-20210417-1ubuntu1) on Ubuntu 21.04:

- 8 bytes per EVM call stack frame

GCC 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04) on Ubuntu 20.04 LTS:

- 544 bytes per EVM call stack frame

GCC 8.4.0 (Ubuntu 8.4.0-3ubuntu2) on Ubuntu 20.04 LTS:

- 544 bytes per EVM call stack frame

GCC 7.5.0 (Ubuntu 7.5.0-6ubuntu2) on Ubuntu 20.04 LTS:

- 544 bytes per EVM call stack frame

GCC 9.2.1 20191008 (Ubuntu 9.2.1-9ubuntu2) on Ubuntu 19.10:

- 528 bytes per EVM call stack frame

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-27 05:53:32 +01:00
Jamie Lokier 085661c24f
EVM: Eliminate recursion entirely
This patch eliminates recursion entirely from the EVM when ENABLE_EVMC=0.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-20 16:12:45 +01:00
Jamie Lokier 8211db1ea8
EVM: Small patch that reduces EVM stack usage to almost nothing
There's been a lot of talk about the Nimbus EVM "stack problem".  I think we
assumed changing it would require big changes to the interpreter code, touching
a lot of functions.

It turned out to be a low hanging fruit.

This patch solves the stack problem, but hardly touches anything.  The change
in EVM stack memory is from 13 MB worst case to just 48 kB, a 250x reduction.

I've been doing work on the database/storage/trie code.  While looking at the
API between the EVM and the database/storage/trie, this stack patch stood out
and made itself obvious.  As it's tiny, rather than more talk, here it is.

Note: This patch is intentionally small, non-invasive, and hopefully easy to
understand, so that it doesn't conflict with other work done on the EVM, and
can easily be grafted into any other EVM structure.

Motivation
==========

- We run out of space and crash on some targets, unless the stack limit is
  raised above its default.  Surprise segmentation faults are unhelpful.

- Some CI targets have been disabled for months due to this.

- Because usage borders on the system limits, when working on
  database/storage/trie/sync code (called from the EVM), segmentation faults
  occur and are misleading.  They cause lost time due to thinking there's a
  crash bug in the code being worked on, when there's nothing wrong with it.

- Sometimes unrelated, trivial code changes elsewhere trigger CI test failures.
  It looks like abrupt termination.  A simple, recent patch was crashing in
  `make test` even though it was a trivial refactor.  Turns out it pushed the
  stack over the edge.

- A large stack has to be scanned by the Nim garbage collector sometimes.
  Larger stack means slower GC and memory allocation.

- The structure of this small patch suggests how to weave async into the EVM
  with almost no changes to the EVM, and no async transformation overhead.

- The patch seemed obvious when working on the API between EVM and storage.

Measurements before
===================

All these tests were run on Ubuntu 20.04 server, x86-64.  This is one of the
targets that has been disabled for a while in CI in EVMC mode due to crashing,
and excessive stack usage is the cause.

Testing commit 0c34a8e3 `2021-04-08 17:46:00 +0200 CI: use MSYS2 on Windows`.

    $ rm -f build/all_tests && make ENABLE_EVMC=1 test
    $ ulimit -S -s 16384 # Requires larger stack than default to avoid crash.
    $ ./build/all_tests 9 | tee tlog
    [Suite] persist block json tests
    ...
	Stack range 38496 depthHigh 3
    ...
    Stack range 13140272 depthHigh 1024
    [OK] tests/fixtures/PersistBlockTests/block1431916.json

These tests use 13.14 MB of stack to run, and so crash with the default stack
limit on Ubuntu Server 20.04 (8MB).  Exactly 12832 bytes per EVM call stack
frame.  It's interesting to see some stack frames take a bit more.

    $ rm -f build/all_tests && make ENABLE_EVMC=1 test
    $ ulimit -S -s 16384 # Requires larger stack than default.
    $ ./build/all_tests 7 | tee tlog
    [Suite] new generalstate json tests
	...
	Stack range 15488 depthHigh 2
	...
	Stack range 3539312 depthHigh 457
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json
    ...
	Stack range 3756144 depthHigh 485
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json
	...
	Stack range 7929968 depthHigh 1024
     [OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json

These tests use 7.92MB of stack to run.  About 7264 bytes per EVM call stack
frame.  It _only just_ avoids crashing with the default Ubuntu Server stack
limit of 8 MB.  However, it still crashes on Windows x86-64, which is why the
CI target is currently disabled.

On Linux where this passes, this is so borderline that it affects work and
testing of storage and sync code, because that's called from the EVM.  Which
was a motivation for dealing with the stack instead of letting this linger.

Also, this stack greatly exceeds the default thread stack size.

    $ rm -f build/all_tests && make ENABLE_EVMC=0 test
    $ ulimit -S -s 16384 # Requires larger stack than default to avoid crash.
    $ ./build/all_tests 9 | tee tlog
    [Suite] persist block json tests
    ...
    Stack range 33216 depthHigh 3
    ...
    Stack range 11338032 depthHigh 1024
    [OK] tests/fixtures/PersistBlockTests/block1431916.json

These tests use 11.33 MB stack to run, and so crash with a default stack limit
of 8MB.  Exactly 11072 bytes per EVM call stack frame.  It's interesting to see
some stack frames take a bit more.

    $ rm -f build/all_tests && make ENABLE_EVMC=0 test
    $ ulimit -S -s 16384 # Requires larger stack than default.
    $ ./build/all_tests 7 | tee tlog
    [Suite] new generalstate json tests
	...
    Stack range 10224 depthHigh 2
	...
    Stack range 2471760 depthHigh 457
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json
    ...
    Stack range 2623184 depthHigh 485
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json
	...
    Stack range 5537824 depthHigh 1024
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json

These tests use 5.54 MB of stack to run, and avoid crashing on with a default
stack limit of 8 MB.  About 5408 bytes per EVM call stack frame.

However, this is uncomfortably close to the limit, as the stack frame size is
sensitive to changes in the code.

Also, this stack greatly exceeds the default thread stack size.

Measurements after
==================

(This patch doesn't address EVMC mode, which is not our default.  EVMC stack
usage remains about the same.  EVMC mode is addressed in another tiny patch.)

    $ rm -f build/all_tests && make ENABLE_EVMC=0 test
    $ ulimit -S -s 80 # Because we can!  80k stack.
    $ ./build/all_tests 9 | tee tlog
    [Suite] persist block json tests
    ...
    Stack range 496 depthHigh 3
    ...
    Stack range 49504 depthHigh 1024
    [OK] tests/fixtures/PersistBlockTests/block1431916.json

    $ rm -f build/all_tests && make ENABLE_EVMC=0 test
    $ ulimit -S -s 72 # Because we can!  72k stack.
    $ ./build/all_tests 7 | tee tlog
    [Suite] new generalstate json tests
	...
    Stack range 448 depthHigh 2
	...
    Stack range 22288 depthHigh 457
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json
    ...
    Stack range 23632 depthHigh 485
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json
	...
    Stack range 49504 depthHigh 1024
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json

For both tests, a satisfying *48 bytes* per EVM call stack frame, and EVM takes
not much more than 48 kB.  With other overheads, both tests run in 80 kB stack
total at maximum EVM depth.

We must add some headroom on this for database activity called from the EVM,
and different compile targets.  But it means the EVM itself is no longer a
stack burden.

This is much smaller than the default thread stack size on Linux (2MB), with
plenty of margin.  It's even smaller than Linux from a long time ago (128kB),
and some small embedded C targets.  (Just fyi, though, some JVM environments
allocated just 32 kB to thread stacks.)

This size is also well suited to running EVMs in threads, if that's useful.

Subtle exception handling and `dispose`
=======================================

It is important that each `snapshot` has a corresponding `dispose` in the event
of an exception being raised.  This code does do that, but in a subtle way.

The pair of functions `execCallOrCreate` and `execCallOrCreateAux` are
equivalent to the following code, where you can see `dispose` more clearly:

    proc execCallOrCreate*(c: Computation) =
      defer: c.dispose()
      if c.beforeExec():
        return
      c.executeOpcodes()
      while not c.continuation.isNil:
        c.child.execCallOrCreate()
        c.child = nil
        (c.continuation)()
        c.executeOpcodes()
      c.afterExec()

That works fine, but only reduces the stack used to 300-700 kB instead of 48 kB.

To get lower we split the above into separate `execCallOrCreate` and
`execCallOrCreateAux`.  Only the outermost has `defer`, and instead of handling
one level, it walks the entire `c.parent` chain calling `dispose` if needed.
The inner one avoids `defer`, which greatly reduces the size of its stackframe.

`c` is a `var` parameter, at each level of recursion.  So the outermost proc
sees the temporary changes made by all inner calls.  This is why `c` is updated
and the `c.parent` chain is maintained at each step.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-13 23:35:26 +01:00
Jordan Hrycaj dfc93a74ad
moved validateTransaction() to executor
why:
  not part of VM (see andri's requested change at #573)
2021-04-07 15:13:28 +01:00
Jordan Hrycaj 827b8c9c81
reset explicit import paths for local modules
why:
  it was convenient to have relocatable source modules when writing the
  vm interface wrappers. this patch moves it back to the standard.

also:
  there are no deep links into the vm folder anymore which leaves some
  room for manoeuvring inside
2021-04-01 12:53:22 +01:00
Jordan Hrycaj 00ba7a2718
merge vm_forks and vm_opcode_values => vm_type2
why:
  all types, but they cannot be merged int vm_types because of a circular
  dependency.
2021-03-31 17:53:15 +01:00
Jordan Hrycaj 9e365734e6
renamed nvm_ prefixed modules to its original names
why:
  the nvm_ prefix was used inside the vm folder to hide them temporarily
  from the outside world while writing export wrappers. now all
  functionality is accessed via vm_*, rather than vm/* imports.

todo:
  at a later stage the import headers of the vm modules need to get fixed
  to meet style guide standards (as jacek kindly pointed out.)
2021-03-31 17:19:54 +01:00
Jordan Hrycaj 474bd9e910
expanded nvm_interpreter
details:
  explicit symbol exports rather than wholesale module names
2021-03-31 16:49:11 +01:00
Jordan Hrycaj 7c28d5d362
provide vm_utils_numeric as import/export wrapper
details:
  moved original vm/interpreter/utils/utils_numeric.nim => vm/interpreter/utils/utils_numeric.nim
2021-03-31 16:49:07 +01:00
Jordan Hrycaj 99568c9b46
provide vm_opcode_values as import/export wrapper
details:
  moved original vm/interpreter/opcode_values.nim => vm/interpreter/nvm_opcode_values.nim
2021-03-31 16:49:03 +01:00
Jordan Hrycaj cf63b9b03f
provide vm_memory as import/export wrapper
details:
  moved original vm/memory.nim => vm/nvm_memory.nim
2021-03-31 16:48:44 +01:00
Jordan Hrycaj 7b5d00307c
provide vm_precompiles as import/export wrapper
details:
  moved original vm/precompiles.nim => vm/nvm_precompiles.nim
2021-03-31 16:47:15 +01:00
Jordan Hrycaj 5ce7ca6b32
provide vm_interpreter as import/export wrapper
details:
  moved original vm/interpreter.nim => vm/nvm_interpreter.nim
2021-03-31 16:47:08 +01:00
Jordan Hrycaj eee24de450
provide vm_message as import/export wrapper
details:
  moved original vm/message.nim => vm/nvm_message.nim
2021-03-31 16:47:02 +01:00
Jordan Hrycaj cf3a356d76
provide vm_computation as import/export wrapper
details:
  moved original vm/computation.nim => vm/nvm_computation.nim
2021-03-31 16:38:10 +01:00
Jordan Hrycaj 689458a346
provide vm_gas_costs as import/export wrapper
details:
  moved original vm/interpreter/vm_gas_costs.nim => vm/interpreter/nvm_gas_costs.nim
2021-03-31 16:03:51 +01:00
Jordan Hrycaj 3a3e4d5707
provide vm_forks as import/export wrapper
details:
  moved original vm/interpreter/vm_forks.nim => vm/interpreter/nvm_forks.nim
2021-03-31 16:03:34 +01:00
Jordan Hrycaj 907465300f
isolate vm_state_transactions as import/export wrapper
details:
  moved original vm_state_transactions.nim => vm/nvm_state_transactions.nim
2021-03-31 10:05:47 +01:00
Jordan Hrycaj eda1290c25
isolate vm_state as import/export wrapper
details:
  moved original vm_state.nim => vm/nvm_state.nim
2021-03-31 09:58:26 +01:00
Jordan Hrycaj ed59f602d5
isolate vm_types as import/export wrapper
details:
  moved original vm_types.nim => vm/nvm_types.nim
2021-03-31 09:48:50 +01:00
Jordan Hrycaj a3db0f41d8
remove relative paths ./ and ../ from import section
why:
  relative paths make sources inherently non-relocatable

details:
  import base is set to the nimbus directoy, so importing ./stack
  from file interpreter.nim becomes vm/stack etc.

caveat:
   a file named nimbus/strformat.nim would clash with strformat (but
   not with std/strformat)
2021-03-30 17:20:43 +01:00