status-im/nimbus-eth1
2
0
Fork 0
mirror of https://github.com/status-im/nimbus-eth1.git synced 2025-01-29 13:35:38 +00:00
Code Issues Projects Releases Wiki Activity
nimbus-eth1/nimbus/vm/interpreter_dispatch.nim
Jamie Lokier a3c8a5c3f3
EVMC: Small stacks when using EVMC, closes #575 (segfaults) 
This patch reduces stack space used with EVM in ENABLE_EVMC=1 mode, from 13 MB
worst case to 550 kB, a 24x reduction.

This completes fixing the "stack problem" and closes #575 (`EVM: Different
segmentation faults when running the test suite with EVMC`).

It also closes #256 (`recursive EVM call trigger unrecoverable stack overflow`).

After this patch, it is possible to re-enable the CI targets which had to be
disabled due to #575.

This change is also a required precursor for switching over to "nearly EVMC" as
the clean and focused Nimbus-internal API between EVM and sync/database
processes, and is also key to the use of Chronos `async` in those processes
when calling the EVM.

(The motivation is the internal interface has to be substantially changed
_anyway_ for the parallel sync and database processes, and EVMC turns out to be
well-designed and well-suited for this.  It provides good separation between
modules, and suits our needs better than our other current interface.  Might as
well use a good one designed by someone else.  EVMC is 98% done in Nimbus
thanks to great work done before by @jangko, and we can use Nimbus-specific
extensions where we need flexibility, including for performance.  Being aligned
with the ecosystem is a useful bonus feature.)

All tests below were run on Ubuntu 20.04 LTS server, x86-64.  This matches one
of the targets that has been disabled for a while in CI in EVMC mode due to
stack overflow crashing the tests, so it's a good choice.

Measurements before
===================

Testing commit `e76e0144 2021-04-22 11:29:42 +0700 add submodules: graphql and
toml-serialization`.

    $ rm -f build/all_tests && make ENABLE_EVMC=1 test
    $ ulimit -S -s 16384 # Requires larger stack than default to avoid crash.
    $ ./build/all_tests 9 | tee tlog
    [Suite] persist block json tests
    ...
    Stack range 38416 depthHigh 3
    ...
    Stack range 13074720 depthHigh 1024
    [OK] tests/fixtures/PersistBlockTests/block1431916.json

These tests use 13.07 MB of stack to run, and so crash with the default stack
limit on Ubuntu Server 20.04 (8MB).  Exactly 12768 bytes per EVM call stack
frame.

    $ rm -f build/all_tests && make ENABLE_EVMC=1 test
    $ ulimit -S -s 16384 # Requires larger stack than default.
    $ ./build/all_tests 7 | tee tlog
    [Suite] new generalstate json tests
        ...
    Stack range 14384 depthHigh 2
        ...
    Stack range 3495456 depthHigh 457
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json
    ...
    Stack range 3709600 depthHigh 485
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json
        ...
    Stack range 7831600 depthHigh 1024
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json

These tests use 7.83MB of stack to run.  About 7648 bytes per EVM call stack
frame.  It _only just_ avoids crashing with the default Ubuntu Server stack
limit of 8 MB.  However, it still crashes on Windows x86-64, which is why the
Windows CI EVMC target is currently disabled.

On Linux where this passes, this is so borderline that it affects work and
testing of the complex storage code, because that's called from the EVM.

Also, this greatly exceeds the default thread stack size.

Measurements after
==================

    $ rm -f build/all_tests && make ENABLE_EVMC=1 test
    $ ulimit -S -s 600 # Because we can!  600k stack.
    $ ./build/all_tests 9 | tee tlog
    [Suite] persist block json tests
    ...
    Stack range 1936 depthHigh 3
    ...
        Stack range 556272 depthHigh 1022
        Stack range 556512 depthHigh 1023
        Stack range 556816 depthHigh 1023
        Stack range 557056 depthHigh 1024
        Stack range 557360 depthHigh 1024
        [OK] tests/fixtures/PersistBlockTests/block1431916.json

    $ rm -f build/all_tests && make ENABLE_EVMC=1 test
    $ ulimit -S -s 600 # Because we can!  600k stack.
    $ ./build/all_tests 7 | tee tlog
    [Suite] new generalstate json tests
        ...
    Stack range 1392 depthHigh 2
        ...
    Stack range 248912 depthHigh 457
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json
    ...
    Stack range 264144 depthHigh 485
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json
        ...
        Stack range 557360 depthHigh 1024
    [OK] tests/fixtures/eth_tests/GeneralStateTests/stStaticCall/static_CallRecursiveBombPreCall.json

For both tests, a satisfying *544 bytes* per EVM call stack frame, and EVM
takes less than 600 kB total.  With other overheads, both tests run in 600 kB
stack total at maximum EVM depth.

We must add some headroom on this for database activity called from the EVM,
and different compile targets.  But it means the EVM itself is no longer a
stack burden.

This is much smaller than the default thread stack size on Linux (2MB), with
plenty of margin.  (Just fyi, it isn't smaller than a _small_ thread stack on
Linux from a long time ago (128kB), and some small embedded C targets.)

This size is well suited to running EVMs in threads.

Further reduction
=================

This patch solves the stack problem.  Windows and Linux 64-bit EVMC CI targets
can be re-enabled, and there is no longer a problem with stack usage.

We can reduce further to ~340 bytes per frame and 350 kB total, while still
complying with EVMC.  But as this involves changing how errors are handled to
comply fully with EVMC, and removing `dispose` calls, it's not worth doing now
while there are other EVMC changes in progress that will have the same effect.

A Nimbus-specific extension will allow us to avoid recursion with EVMC anyway,
bringing bytes per frame to zero.  We need the extension anyway, to support
Chronos `async` which parallel transaction processing is built around.

Interop with non-Nimbus over EVMC won't let us avoid recursion, but then we
can't control the stack frame size either.  To prevent stack overflow in
interop I anticipate using (this method in Aleth)
[6e96ce34e3/libethereum/ExtVM.cpp (L61)].

Smoke test other versions of GCC and Clang/LLVM
===============================================

As all builds including Windows use GCC or Apple's Clang/LLVM, this is just to
verify we're in the right ballpark on all targets.  I've only checked `x86_64`
though, not 32-bit, and not ARM.

It's interesting to see GCC 10 uses less stack.  This is because it optimises
`struct` returns better, sometimes skipping an intermediate copy.  Here it
benefits the EVMC API, but I found GCC 10 also improves the larger stack usage
of the rest of `nimbus-eth1` as well.

Apple clang 12.0.0 (clang-1200.0.26.2) on MacOS 10.15:

- 544 bytes per EVM call stack frame

GCC 10.3.0 (Ubuntu 10.3.0-1ubuntu1) on Ubuntu 21.04:

- 464 bytes per EVM call stack frame

GCC 10.2.0 (Ubuntu 10.2.0-5ubuntu1~20.04) on Ubuntu 20.04 LTS:

- 464 bytes per EVM call stack frame

GCC 11.0.1 20210417 (experimental; Ubuntu 11-20210417-1ubuntu1) on Ubuntu 21.04:

- 8 bytes per EVM call stack frame

GCC 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04) on Ubuntu 20.04 LTS:

- 544 bytes per EVM call stack frame

GCC 8.4.0 (Ubuntu 8.4.0-3ubuntu2) on Ubuntu 20.04 LTS:

- 544 bytes per EVM call stack frame

GCC 7.5.0 (Ubuntu 7.5.0-6ubuntu2) on Ubuntu 20.04 LTS:

- 544 bytes per EVM call stack frame

GCC 9.2.1 20191008 (Ubuntu 9.2.1-9ubuntu2) on Ubuntu 19.10:

- 528 bytes per EVM call stack frame

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-27 05:53:32 +01:00

403 lines
14 KiB
Nim
Raw Blame History

# Nimbus
# Copyright (c) 2018 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
# * MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)
# at your option. This file may not be copied, modified, or distributed except according to those terms.
import
tables, macros,
chronicles,
./interpreter/[opcode_values, opcodes_impl, vm_forks, gas_costs, gas_meter, utils/macros_gen_opcodes],
./code_stream, ./types, ../errors, ./precompiles, ./stack,
terminal # Those are only needed for logging
logScope:
topics = "vm opcode"
func invalidInstruction*(c: Computation) {.inline.} =
raise newException(InvalidInstruction, "Invalid instruction, received an opcode not implemented in the current fork.")
let FrontierOpDispatch {.compileTime.}: array[Op, NimNode] = block:
fill_enum_table_holes(Op, newIdentNode("invalidInstruction")):
[
Stop: newIdentNode "toBeReplacedByBreak",
Add: newIdentNode "add",
Mul: newIdentNode "mul",
Sub: newIdentNode "sub",
Div: newIdentNode "divide",
Sdiv: newIdentNode "sdiv",
Mod: newIdentNode "modulo",
Smod: newIdentNode "smod",
Addmod: newIdentNode "addmod",
Mulmod: newIdentNode "mulmod",
Exp: newIdentNode "exp",
SignExtend: newIdentNode "signExtend",
# 10s: Comparison & Bitwise Logic Operations
Lt: newIdentNode "lt",
Gt: newIdentNode "gt",
Slt: newIdentNode "slt",
Sgt: newIdentNode "sgt",
Eq: newIdentNode "eq",
IsZero: newIdentNode "isZero",
And: newIdentNode "andOp",
Or: newIdentNode "orOp",
Xor: newIdentNode "xorOp",
Not: newIdentNode "notOp",
Byte: newIdentNode "byteOp",
# 20s: SHA3
Sha3: newIdentNode "sha3",
# 30s: Environmental Information
Address: newIdentNode "address",
Balance: newIdentNode "balance",
Origin: newIdentNode "origin",
Caller: newIdentNode "caller",
CallValue: newIdentNode "callValue",
CallDataLoad: newIdentNode "callDataLoad",
CallDataSize: newIdentNode "callDataSize",
CallDataCopy: newIdentNode "callDataCopy",
CodeSize: newIdentNode "codeSize",
CodeCopy: newIdentNode "codeCopy",
GasPrice: newIdentNode "gasPrice",
ExtCodeSize: newIdentNode "extCodeSize",
ExtCodeCopy: newIdentNode "extCodeCopy",
# ReturnDataSize: introduced in Byzantium
# ReturnDataCopy: introduced in Byzantium
# 40s: Block Information
Blockhash: newIdentNode "blockhash",
Coinbase: newIdentNode "coinbase",
Timestamp: newIdentNode "timestamp",
Number: newIdentNode "blockNumber",
Difficulty: newIdentNode "difficulty",
GasLimit: newIdentNode "gasLimit",
# 50s: Stack, Memory, Storage and Flow Operations
Pop: newIdentNode "pop",
Mload: newIdentNode "mload",
Mstore: newIdentNode "mstore",
Mstore8: newIdentNode "mstore8",
Sload: newIdentNode "sload",
Sstore: newIdentNode "sstore",
Jump: newIdentNode "jump",
JumpI: newIdentNode "jumpI",
Pc: newIdentNode "pc",
Msize: newIdentNode "msize",
Gas: newIdentNode "gas",
JumpDest: newIdentNode "jumpDest",
# 60s & 70s: Push Operations.
Push1: newIdentNode "push1",
Push2: newIdentNode "push2",
Push3: newIdentNode "push3",
Push4: newIdentNode "push4",
Push5: newIdentNode "push5",
Push6: newIdentNode "push6",
Push7: newIdentNode "push7",
Push8: newIdentNode "push8",
Push9: newIdentNode "push9",
Push10: newIdentNode "push10",
Push11: newIdentNode "push11",
Push12: newIdentNode "push12",
Push13: newIdentNode "push13",
Push14: newIdentNode "push14",
Push15: newIdentNode "push15",
Push16: newIdentNode "push16",
Push17: newIdentNode "push17",
Push18: newIdentNode "push18",
Push19: newIdentNode "push19",
Push20: newIdentNode "push20",
Push21: newIdentNode "push21",
Push22: newIdentNode "push22",
Push23: newIdentNode "push23",
Push24: newIdentNode "push24",
Push25: newIdentNode "push25",
Push26: newIdentNode "push26",
Push27: newIdentNode "push27",
Push28: newIdentNode "push28",
Push29: newIdentNode "push29",
Push30: newIdentNode "push30",
Push31: newIdentNode "push31",
Push32: newIdentNode "push32",
# 80s: Duplication Operations
Dup1: newIdentNode "dup1",
Dup2: newIdentNode "dup2",
Dup3: newIdentNode "dup3",
Dup4: newIdentNode "dup4",
Dup5: newIdentNode "dup5",
Dup6: newIdentNode "dup6",
Dup7: newIdentNode "dup7",
Dup8: newIdentNode "dup8",
Dup9: newIdentNode "dup9",
Dup10: newIdentNode "dup10",
Dup11: newIdentNode "dup11",
Dup12: newIdentNode "dup12",
Dup13: newIdentNode "dup13",
Dup14: newIdentNode "dup14",
Dup15: newIdentNode "dup15",
Dup16: newIdentNode "dup16",
# 90s: Exchange Operations
Swap1: newIdentNode "swap1",
Swap2: newIdentNode "swap2",
Swap3: newIdentNode "swap3",
Swap4: newIdentNode "swap4",
Swap5: newIdentNode "swap5",
Swap6: newIdentNode "swap6",
Swap7: newIdentNode "swap7",
Swap8: newIdentNode "swap8",
Swap9: newIdentNode "swap9",
Swap10: newIdentNode "swap10",
Swap11: newIdentNode "swap11",
Swap12: newIdentNode "swap12",
Swap13: newIdentNode "swap13",
Swap14: newIdentNode "swap14",
Swap15: newIdentNode "swap15",
Swap16: newIdentNode "swap16",
# a0s: Logging Operations
Log0: newIdentNode "log0",
Log1: newIdentNode "log1",
Log2: newIdentNode "log2",
Log3: newIdentNode "log3",
Log4: newIdentNode "log4",
# f0s: System operations
Create: newIdentNode "create",
Call: newIdentNode "call",
CallCode: newIdentNode "callCode",
Return: newIdentNode "returnOp",
# StaticCall: introduced in Byzantium
# Revert: introduced in Byzantium
# Invalid: newIdentNode "invalid",
SelfDestruct: newIdentNode "selfDestruct"
]
proc genHomesteadJumpTable(ops: array[Op, NimNode]): array[Op, NimNode] {.compileTime.} =
result = ops
result[DelegateCall] = newIdentNode "delegateCall"
let HomesteadOpDispatch {.compileTime.}: array[Op, NimNode] = genHomesteadJumpTable(FrontierOpDispatch)
proc genTangerineJumpTable(ops: array[Op, NimNode]): array[Op, NimNode] {.compileTime.} =
result = ops
result[SelfDestruct] = newIdentNode "selfDestructEIP150"
let TangerineOpDispatch {.compileTime.}: array[Op, NimNode] = genTangerineJumpTable(HomesteadOpDispatch)
proc genSpuriousJumpTable(ops: array[Op, NimNode]): array[Op, NimNode] {.compileTime.} =
result = ops
result[SelfDestruct] = newIdentNode "selfDestructEIP161"
let SpuriousOpDispatch {.compileTime.}: array[Op, NimNode] = genSpuriousJumpTable(TangerineOpDispatch)
proc genByzantiumJumpTable(ops: array[Op, NimNode]): array[Op, NimNode] {.compileTime.} =
result = ops
result[Revert] = newIdentNode "revert"
result[ReturnDataSize] = newIdentNode "returnDataSize"
result[ReturnDataCopy] = newIdentNode "returnDataCopy"
result[StaticCall] = newIdentNode"staticCall"
let ByzantiumOpDispatch {.compileTime.}: array[Op, NimNode] = genByzantiumJumpTable(SpuriousOpDispatch)
proc genConstantinopleJumpTable(ops: array[Op, NimNode]): array[Op, NimNode] {.compileTime.} =
result = ops
result[Shl] = newIdentNode "shlOp"
result[Shr] = newIdentNode "shrOp"
result[Sar] = newIdentNode "sarOp"
result[ExtCodeHash] = newIdentNode "extCodeHash"
result[Create2] = newIdentNode "create2"
result[SStore] = newIdentNode "sstoreEIP1283"
let ConstantinopleOpDispatch {.compileTime.}: array[Op, NimNode] = genConstantinopleJumpTable(ByzantiumOpDispatch)
proc genPetersburgJumpTable(ops: array[Op, NimNode]): array[Op, NimNode] {.compileTime.} =
result = ops
result[SStore] = newIdentNode "sstore" # disable EIP-1283
let PetersburgOpDispatch {.compileTime.}: array[Op, NimNode] = genPetersburgJumpTable(ConstantinopleOpDispatch)
proc genIstanbulJumpTable(ops: array[Op, NimNode]): array[Op, NimNode] {.compileTime.} =
result = ops
result[ChainId] = newIdentNode "chainId"
result[SelfBalance] = newIdentNode "selfBalance"
result[SStore] = newIdentNode "sstoreEIP2200"
let IstanbulOpDispatch {.compileTime.}: array[Op, NimNode] = genIstanbulJumpTable(PetersburgOpDispatch)
proc genBerlinJumpTable(ops: array[Op, NimNode]): array[Op, NimNode] {.compileTime.} =
result = ops
result[BeginSub] = newIdentNode "beginSub"
result[ReturnSub] = newIdentNode "returnSub"
result[JumpSub] = newIdentNode "jumpSub"
result[Balance] = newIdentNode "balanceEIP2929"
result[ExtCodeHash] = newIdentNode "extCodeHashEIP2929"
result[ExtCodeSize] = newIdentNode "extCodeSizeEIP2929"
result[ExtCodeCopy] = newIdentNode "extCodeCopyEIP2929"
result[SelfDestruct] = newIdentNode "selfDestructEIP2929"
result[SLoad] = newIdentNode "sloadEIP2929"
result[SStore] = newIdentNode "sstoreEIP2929"
let BerlinOpDispatch {.compileTime.}: array[Op, NimNode] = genBerlinJumpTable(IstanbulOpDispatch)
proc opTableToCaseStmt(opTable: array[Op, NimNode], c: NimNode): NimNode =
let instr = quote do: `c`.instr
result = nnkCaseStmt.newTree(instr)
# Add a branch for each (opcode, proc) pair
# We dispatch to the next instruction at the end of each branch
for op, opImpl in opTable.pairs:
let asOp = quote do: Op(`op`) # TODO: unfortunately when passing to runtime, ops are transformed into int
let branchStmt = block:
if op == Stop:
quote do:
trace "op: Stop"
if not `c`.code.atEnd() and `c`.tracingEnabled:
# we only trace `REAL STOP` and ignore `FAKE STOP`
`c`.opIndex = `c`.traceOpCodeStarted(`asOp`)
`c`.traceOpCodeEnded(`asOp`, `c`.opIndex)
break
else:
if BaseGasCosts[op].kind == GckFixed:
quote do:
if `c`.tracingEnabled:
`c`.opIndex = `c`.traceOpCodeStarted(`asOp`)
`c`.gasMeter.consumeGas(`c`.gasCosts[`asOp`].cost, reason = $`asOp`)
`opImpl`(`c`)
if `c`.tracingEnabled:
`c`.traceOpCodeEnded(`asOp`, `c`.opIndex)
when `asOp` in {Create, Create2, Call, CallCode, DelegateCall, StaticCall}:
if not `c`.continuation.isNil:
return
else:
quote do:
if `c`.tracingEnabled:
`c`.opIndex = `c`.traceOpCodeStarted(`asOp`)
`opImpl`(`c`)
if `c`.tracingEnabled:
`c`.traceOpCodeEnded(`asOp`, `c`.opIndex)
when `asOp` in {Create, Create2, Call, CallCode, DelegateCall, StaticCall}:
if not `c`.continuation.isNil:
return
when `asOp` in {Return, Revert, SelfDestruct}:
break
result.add nnkOfBranch.newTree(
newIdentNode($op),
branchStmt
)
# Wrap the case statement in while true + computed goto
result = quote do:
if `c`.tracingEnabled:
`c`.prepareTracer()
while true:
`instr` = `c`.code.next()
#{.computedGoto.}
# computed goto causing stack overflow, it consumes a lot of space
# we could use manual jump table instead
# TODO lots of macro magic here to unravel, with chronicles...
# `c`.logger.log($`c`.stack & "\n\n", fgGreen)
`result`
macro genFrontierDispatch(c: Computation): untyped =
result = opTableToCaseStmt(FrontierOpDispatch, c)
macro genHomesteadDispatch(c: Computation): untyped =
result = opTableToCaseStmt(HomesteadOpDispatch, c)
macro genTangerineDispatch(c: Computation): untyped =
result = opTableToCaseStmt(TangerineOpDispatch, c)
macro genSpuriousDispatch(c: Computation): untyped =
result = opTableToCaseStmt(SpuriousOpDispatch, c)
macro genByzantiumDispatch(c: Computation): untyped =
result = opTableToCaseStmt(ByzantiumOpDispatch, c)
macro genConstantinopleDispatch(c: Computation): untyped =
result = opTableToCaseStmt(ConstantinopleOpDispatch, c)
macro genPetersburgDispatch(c: Computation): untyped =
result = opTableToCaseStmt(PetersburgOpDispatch, c)
macro genIstanbulDispatch(c: Computation): untyped =
result = opTableToCaseStmt(IstanbulOpDispatch, c)
macro genBerlinDispatch(c: Computation): untyped =
result = opTableToCaseStmt(BerlinOpDispatch, c)
proc frontierVM(c: Computation) =
genFrontierDispatch(c)
proc homesteadVM(c: Computation) =
genHomesteadDispatch(c)
proc tangerineVM(c: Computation) =
genTangerineDispatch(c)
proc spuriousVM(c: Computation) {.gcsafe.} =
genSpuriousDispatch(c)
proc byzantiumVM(c: Computation) {.gcsafe.} =
genByzantiumDispatch(c)
proc constantinopleVM(c: Computation) {.gcsafe.} =
genConstantinopleDispatch(c)
proc petersburgVM(c: Computation) {.gcsafe.} =
genPetersburgDispatch(c)
proc istanbulVM(c: Computation) {.gcsafe.} =
genIstanbulDispatch(c)
proc berlinVM(c: Computation) {.gcsafe.} =
genBerlinDispatch(c)
proc selectVM(c: Computation, fork: Fork) {.gcsafe.} =
# TODO: Optimise getting fork and updating opCodeExec only when necessary
case fork
of FkFrontier:
c.frontierVM()
of FkHomestead:
c.homesteadVM()
of FkTangerine:
c.tangerineVM()
of FkSpurious:
c.spuriousVM()
of FkByzantium:
c.byzantiumVM()
of FkConstantinople:
c.constantinopleVM()
of FkPetersburg:
c.petersburgVM()
of FkIstanbul:
c.istanbulVM()
else:
c.berlinVM()
proc executeOpcodes(c: Computation) =
let fork = c.fork
block:
if not c.continuation.isNil:
(c.continuation)()
c.continuation = nil
elif c.execPrecompiles(fork):
break
try:
c.selectVM(fork)
except CatchableError as e:
c.setError(&"Opcode Dispatch Error msg={e.msg}, depth={c.msg.depth}", true)
if c.isError() and c.continuation.isNil:
if c.tracingEnabled: c.traceError()
debug "executeOpcodes error", msg=c.error.info
Reference in New Issue View Git Blame Copy Permalink