nimbus-eth1/nimbus/vm/interpreter_dispatch.nim

419 lines
14 KiB
Nim
Raw Normal View History

Refactor interpreter dispatch (#65) * move forks constants, rename errors * Move vm/utils to vm/interpreter/utils * initial opcodes refactoring * Add refactored Comparison & Bitwise Logic Operations * Add sha3 and address, simplify macro, support pop 0 * balance, origin, caller, callValue * fix gas copy opcodes gas costs, add callDataLoad/Size/Copy, CodeSize/Copy and gas price opcode * Update with 30s, 40s, 50s opcodes + impl of balance + stack improvement * add push, dup, swap, log, create and call operations * finish opcode implementation * Add the new dispatching logic * Pass the opcode test * Make test_vm_json compile * halt execution without exceptions for Return, Revert, selfdestruct (fix #62) * Properly catch and recover from EVM exceptions (stack underflow ...) * Fix byte op * Fix jump regressions * Update for latest devel, don't import old dispatch code as quasiBoolean macro is broken by latest devel * Fix sha3 regression on empty memory slice and until end of range slice * Fix padding / range error on expXY_success (gas computation left) * update logging procs * Add tracing - expXY_success is not a regression, sload stub was accidentally passing the test * Reuse the same stub as OO implementation * Delete previous opcode implementation * Delete object oriented fork code * Delete exceptions that were used as control flows * delete base.nim :fire:, yet another OO remnants * Delete opcode table * Enable omputed gotos and compile-time gas fees * Revert const gasCosts -> generates SIGSEGV * inline push, swap and dup opcodes * loggers are now template again, why does this pass new tests? * Trigger CI rebuild after rocksdb fix https://github.com/status-im/nim-rocksdb/pull/5 * Address review comment on "push" + VMTests in debug mode (not release) * Address review comment: don't tag fork by default, make opcode impl grepable * Static compilation fixes after rebasing * fix the initialization of the VM database * add a missing import * Deactivate balance and sload test following #59 * Reactivate stack check (deactivated in #59, necessary to pass tests) * Merge remaining opcodes implementation from #59 * Merge callDataLoad and codeCopy fixes, todo simplify see #67
2018-07-06 07:52:31 +00:00
# Nimbus
# Copyright (c) 2018 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
# * MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)
# at your option. This file may not be copied, modified, or distributed except according to those terms.
import
tables, macros,
chronicles,
./interpreter/[opcode_values, opcodes_impl, gas_costs, gas_meter, utils/macros_gen_opcodes],
2021-08-11 19:37:00 +00:00
./code_stream, ./types, ../errors, ../forks, ./precompiles, ./stack
Refactor interpreter dispatch (#65) * move forks constants, rename errors * Move vm/utils to vm/interpreter/utils * initial opcodes refactoring * Add refactored Comparison & Bitwise Logic Operations * Add sha3 and address, simplify macro, support pop 0 * balance, origin, caller, callValue * fix gas copy opcodes gas costs, add callDataLoad/Size/Copy, CodeSize/Copy and gas price opcode * Update with 30s, 40s, 50s opcodes + impl of balance + stack improvement * add push, dup, swap, log, create and call operations * finish opcode implementation * Add the new dispatching logic * Pass the opcode test * Make test_vm_json compile * halt execution without exceptions for Return, Revert, selfdestruct (fix #62) * Properly catch and recover from EVM exceptions (stack underflow ...) * Fix byte op * Fix jump regressions * Update for latest devel, don't import old dispatch code as quasiBoolean macro is broken by latest devel * Fix sha3 regression on empty memory slice and until end of range slice * Fix padding / range error on expXY_success (gas computation left) * update logging procs * Add tracing - expXY_success is not a regression, sload stub was accidentally passing the test * Reuse the same stub as OO implementation * Delete previous opcode implementation * Delete object oriented fork code * Delete exceptions that were used as control flows * delete base.nim :fire:, yet another OO remnants * Delete opcode table * Enable omputed gotos and compile-time gas fees * Revert const gasCosts -> generates SIGSEGV * inline push, swap and dup opcodes * loggers are now template again, why does this pass new tests? * Trigger CI rebuild after rocksdb fix https://github.com/status-im/nim-rocksdb/pull/5 * Address review comment on "push" + VMTests in debug mode (not release) * Address review comment: don't tag fork by default, make opcode impl grepable * Static compilation fixes after rebasing * fix the initialization of the VM database * add a missing import * Deactivate balance and sload test following #59 * Reactivate stack check (deactivated in #59, necessary to pass tests) * Merge remaining opcodes implementation from #59 * Merge callDataLoad and codeCopy fixes, todo simplify see #67
2018-07-06 07:52:31 +00:00
logScope:
topics = "vm opcode"
func invalidInstruction*(c: Computation) {.inline.} =
2019-03-19 15:16:21 +00:00
raise newException(InvalidInstruction, "Invalid instruction, received an opcode not implemented in the current fork.")
Refactor interpreter dispatch (#65) * move forks constants, rename errors * Move vm/utils to vm/interpreter/utils * initial opcodes refactoring * Add refactored Comparison & Bitwise Logic Operations * Add sha3 and address, simplify macro, support pop 0 * balance, origin, caller, callValue * fix gas copy opcodes gas costs, add callDataLoad/Size/Copy, CodeSize/Copy and gas price opcode * Update with 30s, 40s, 50s opcodes + impl of balance + stack improvement * add push, dup, swap, log, create and call operations * finish opcode implementation * Add the new dispatching logic * Pass the opcode test * Make test_vm_json compile * halt execution without exceptions for Return, Revert, selfdestruct (fix #62) * Properly catch and recover from EVM exceptions (stack underflow ...) * Fix byte op * Fix jump regressions * Update for latest devel, don't import old dispatch code as quasiBoolean macro is broken by latest devel * Fix sha3 regression on empty memory slice and until end of range slice * Fix padding / range error on expXY_success (gas computation left) * update logging procs * Add tracing - expXY_success is not a regression, sload stub was accidentally passing the test * Reuse the same stub as OO implementation * Delete previous opcode implementation * Delete object oriented fork code * Delete exceptions that were used as control flows * delete base.nim :fire:, yet another OO remnants * Delete opcode table * Enable omputed gotos and compile-time gas fees * Revert const gasCosts -> generates SIGSEGV * inline push, swap and dup opcodes * loggers are now template again, why does this pass new tests? * Trigger CI rebuild after rocksdb fix https://github.com/status-im/nim-rocksdb/pull/5 * Address review comment on "push" + VMTests in debug mode (not release) * Address review comment: don't tag fork by default, make opcode impl grepable * Static compilation fixes after rebasing * fix the initialization of the VM database * add a missing import * Deactivate balance and sload test following #59 * Reactivate stack check (deactivated in #59, necessary to pass tests) * Merge remaining opcodes implementation from #59 * Merge callDataLoad and codeCopy fixes, todo simplify see #67
2018-07-06 07:52:31 +00:00
let FrontierOpDispatch {.compileTime.}: array[Op, NimNode] = block:
fill_enum_table_holes(Op, newIdentNode("invalidInstruction")):
Refactor interpreter dispatch (#65) * move forks constants, rename errors * Move vm/utils to vm/interpreter/utils * initial opcodes refactoring * Add refactored Comparison & Bitwise Logic Operations * Add sha3 and address, simplify macro, support pop 0 * balance, origin, caller, callValue * fix gas copy opcodes gas costs, add callDataLoad/Size/Copy, CodeSize/Copy and gas price opcode * Update with 30s, 40s, 50s opcodes + impl of balance + stack improvement * add push, dup, swap, log, create and call operations * finish opcode implementation * Add the new dispatching logic * Pass the opcode test * Make test_vm_json compile * halt execution without exceptions for Return, Revert, selfdestruct (fix #62) * Properly catch and recover from EVM exceptions (stack underflow ...) * Fix byte op * Fix jump regressions * Update for latest devel, don't import old dispatch code as quasiBoolean macro is broken by latest devel * Fix sha3 regression on empty memory slice and until end of range slice * Fix padding / range error on expXY_success (gas computation left) * update logging procs * Add tracing - expXY_success is not a regression, sload stub was accidentally passing the test * Reuse the same stub as OO implementation * Delete previous opcode implementation * Delete object oriented fork code * Delete exceptions that were used as control flows * delete base.nim :fire:, yet another OO remnants * Delete opcode table * Enable omputed gotos and compile-time gas fees * Revert const gasCosts -> generates SIGSEGV * inline push, swap and dup opcodes * loggers are now template again, why does this pass new tests? * Trigger CI rebuild after rocksdb fix https://github.com/status-im/nim-rocksdb/pull/5 * Address review comment on "push" + VMTests in debug mode (not release) * Address review comment: don't tag fork by default, make opcode impl grepable * Static compilation fixes after rebasing * fix the initialization of the VM database * add a missing import * Deactivate balance and sload test following #59 * Reactivate stack check (deactivated in #59, necessary to pass tests) * Merge remaining opcodes implementation from #59 * Merge callDataLoad and codeCopy fixes, todo simplify see #67
2018-07-06 07:52:31 +00:00
[
Stop: newIdentNode "toBeReplacedByBreak",
Add: newIdentNode "add",
Mul: newIdentNode "mul",
Sub: newIdentNode "sub",
Div: newIdentNode "divide",
Sdiv: newIdentNode "sdiv",
Mod: newIdentNode "modulo",
Smod: newIdentNode "smod",
Addmod: newIdentNode "addmod",
Mulmod: newIdentNode "mulmod",
Exp: newIdentNode "exp",
SignExtend: newIdentNode "signExtend",
# 10s: Comparison & Bitwise Logic Operations
Lt: newIdentNode "lt",
Gt: newIdentNode "gt",
Slt: newIdentNode "slt",
Sgt: newIdentNode "sgt",
Eq: newIdentNode "eq",
IsZero: newIdentNode "isZero",
And: newIdentNode "andOp",
Or: newIdentNode "orOp",
Xor: newIdentNode "xorOp",
Not: newIdentNode "notOp",
Byte: newIdentNode "byteOp",
# 20s: SHA3
Sha3: newIdentNode "sha3",
# 30s: Environmental Information
Address: newIdentNode "address",
Balance: newIdentNode "balance",
Origin: newIdentNode "origin",
Caller: newIdentNode "caller",
CallValue: newIdentNode "callValue",
CallDataLoad: newIdentNode "callDataLoad",
CallDataSize: newIdentNode "callDataSize",
CallDataCopy: newIdentNode "callDataCopy",
CodeSize: newIdentNode "codeSize",
CodeCopy: newIdentNode "codeCopy",
GasPrice: newIdentNode "gasPrice",
ExtCodeSize: newIdentNode "extCodeSize",
ExtCodeCopy: newIdentNode "extCodeCopy",
# ReturnDataSize: introduced in Byzantium
# ReturnDataCopy: introduced in Byzantium
# 40s: Block Information
Blockhash: newIdentNode "blockhash",
Coinbase: newIdentNode "coinbase",
Timestamp: newIdentNode "timestamp",
Number: newIdentNode "blockNumber",
Difficulty: newIdentNode "difficulty",
GasLimit: newIdentNode "gasLimit",
# 50s: Stack, Memory, Storage and Flow Operations
Pop: newIdentNode "pop",
Mload: newIdentNode "mload",
Mstore: newIdentNode "mstore",
Mstore8: newIdentNode "mstore8",
Sload: newIdentNode "sload",
Sstore: newIdentNode "sstore",
Jump: newIdentNode "jump",
JumpI: newIdentNode "jumpI",
Pc: newIdentNode "pc",
Msize: newIdentNode "msize",
Gas: newIdentNode "gas",
JumpDest: newIdentNode "jumpDest",
# 60s & 70s: Push Operations.
Push1: newIdentNode "push1",
Push2: newIdentNode "push2",
Push3: newIdentNode "push3",
Push4: newIdentNode "push4",
Push5: newIdentNode "push5",
Push6: newIdentNode "push6",
Push7: newIdentNode "push7",
Push8: newIdentNode "push8",
Push9: newIdentNode "push9",
Push10: newIdentNode "push10",
Push11: newIdentNode "push11",
Push12: newIdentNode "push12",
Push13: newIdentNode "push13",
Push14: newIdentNode "push14",
Push15: newIdentNode "push15",
Push16: newIdentNode "push16",
Push17: newIdentNode "push17",
Push18: newIdentNode "push18",
Push19: newIdentNode "push19",
Push20: newIdentNode "push20",
Push21: newIdentNode "push21",
Push22: newIdentNode "push22",
Push23: newIdentNode "push23",
Push24: newIdentNode "push24",
Push25: newIdentNode "push25",
Push26: newIdentNode "push26",
Push27: newIdentNode "push27",
Push28: newIdentNode "push28",
Push29: newIdentNode "push29",
Push30: newIdentNode "push30",
Push31: newIdentNode "push31",
Push32: newIdentNode "push32",
# 80s: Duplication Operations
Dup1: newIdentNode "dup1",
Dup2: newIdentNode "dup2",
Dup3: newIdentNode "dup3",
Dup4: newIdentNode "dup4",
Dup5: newIdentNode "dup5",
Dup6: newIdentNode "dup6",
Dup7: newIdentNode "dup7",
Dup8: newIdentNode "dup8",
Dup9: newIdentNode "dup9",
Dup10: newIdentNode "dup10",
Dup11: newIdentNode "dup11",
Dup12: newIdentNode "dup12",
Dup13: newIdentNode "dup13",
Dup14: newIdentNode "dup14",
Dup15: newIdentNode "dup15",
Dup16: newIdentNode "dup16",
# 90s: Exchange Operations
Swap1: newIdentNode "swap1",
Swap2: newIdentNode "swap2",
Swap3: newIdentNode "swap3",
Swap4: newIdentNode "swap4",
Swap5: newIdentNode "swap5",
Swap6: newIdentNode "swap6",
Swap7: newIdentNode "swap7",
Swap8: newIdentNode "swap8",
Swap9: newIdentNode "swap9",
Swap10: newIdentNode "swap10",
Swap11: newIdentNode "swap11",
Swap12: newIdentNode "swap12",
Swap13: newIdentNode "swap13",
Swap14: newIdentNode "swap14",
Swap15: newIdentNode "swap15",
Swap16: newIdentNode "swap16",
# a0s: Logging Operations
Log0: newIdentNode "log0",
Log1: newIdentNode "log1",
Log2: newIdentNode "log2",
Log3: newIdentNode "log3",
Log4: newIdentNode "log4",
# f0s: System operations
Create: newIdentNode "create",
Call: newIdentNode "call",
CallCode: newIdentNode "callCode",
Return: newIdentNode "returnOp",
# StaticCall: introduced in Byzantium
# Revert: introduced in Byzantium
# Invalid: newIdentNode "invalid",
SelfDestruct: newIdentNode "selfDestruct"
]
proc genHomesteadJumpTable(ops: array[Op, NimNode]): array[Op, NimNode] {.compileTime.} =
result = ops
result[DelegateCall] = newIdentNode "delegateCall"
let HomesteadOpDispatch {.compileTime.}: array[Op, NimNode] = genHomesteadJumpTable(FrontierOpDispatch)
proc genTangerineJumpTable(ops: array[Op, NimNode]): array[Op, NimNode] {.compileTime.} =
result = ops
result[SelfDestruct] = newIdentNode "selfDestructEIP150"
let TangerineOpDispatch {.compileTime.}: array[Op, NimNode] = genTangerineJumpTable(HomesteadOpDispatch)
proc genSpuriousJumpTable(ops: array[Op, NimNode]): array[Op, NimNode] {.compileTime.} =
result = ops
result[SelfDestruct] = newIdentNode "selfDestructEIP161"
let SpuriousOpDispatch {.compileTime.}: array[Op, NimNode] = genSpuriousJumpTable(TangerineOpDispatch)
2019-04-22 09:26:59 +00:00
proc genByzantiumJumpTable(ops: array[Op, NimNode]): array[Op, NimNode] {.compileTime.} =
result = ops
result[Revert] = newIdentNode "revert"
result[ReturnDataSize] = newIdentNode "returnDataSize"
result[ReturnDataCopy] = newIdentNode "returnDataCopy"
2019-04-22 09:43:07 +00:00
result[StaticCall] = newIdentNode"staticCall"
2019-04-22 09:26:59 +00:00
let ByzantiumOpDispatch {.compileTime.}: array[Op, NimNode] = genByzantiumJumpTable(SpuriousOpDispatch)
proc genConstantinopleJumpTable(ops: array[Op, NimNode]): array[Op, NimNode] {.compileTime.} =
result = ops
result[Shl] = newIdentNode "shlOp"
result[Shr] = newIdentNode "shrOp"
result[Sar] = newIdentNode "sarOp"
result[ExtCodeHash] = newIdentNode "extCodeHash"
result[Create2] = newIdentNode "create2"
2020-03-04 09:32:30 +00:00
result[SStore] = newIdentNode "sstoreEIP1283"
let ConstantinopleOpDispatch {.compileTime.}: array[Op, NimNode] = genConstantinopleJumpTable(ByzantiumOpDispatch)
2020-03-04 09:32:30 +00:00
proc genPetersburgJumpTable(ops: array[Op, NimNode]): array[Op, NimNode] {.compileTime.} =
result = ops
result[SStore] = newIdentNode "sstore" # disable EIP-1283
let PetersburgOpDispatch {.compileTime.}: array[Op, NimNode] = genPetersburgJumpTable(ConstantinopleOpDispatch)
2019-11-11 04:58:58 +00:00
proc genIstanbulJumpTable(ops: array[Op, NimNode]): array[Op, NimNode] {.compileTime.} =
result = ops
result[ChainIdOp] = newIdentNode "chainId"
2019-11-11 05:12:58 +00:00
result[SelfBalance] = newIdentNode "selfBalance"
2019-11-12 12:49:46 +00:00
result[SStore] = newIdentNode "sstoreEIP2200"
2019-11-11 04:58:58 +00:00
2020-03-04 09:32:30 +00:00
let IstanbulOpDispatch {.compileTime.}: array[Op, NimNode] = genIstanbulJumpTable(PetersburgOpDispatch)
2019-11-11 04:58:58 +00:00
2020-11-25 09:43:34 +00:00
proc genBerlinJumpTable(ops: array[Op, NimNode]): array[Op, NimNode] {.compileTime.} =
result = ops
# EIP-2315: temporary disabled
# Reason : not included in berlin hard fork
#result[BeginSub] = newIdentNode "beginSub"
#result[ReturnSub] = newIdentNode "returnSub"
#result[JumpSub] = newIdentNode "jumpSub"
2020-11-25 09:43:34 +00:00
2020-12-11 10:51:17 +00:00
result[Balance] = newIdentNode "balanceEIP2929"
result[ExtCodeHash] = newIdentNode "extCodeHashEIP2929"
result[ExtCodeSize] = newIdentNode "extCodeSizeEIP2929"
result[ExtCodeCopy] = newIdentNode "extCodeCopyEIP2929"
result[SelfDestruct] = newIdentNode "selfDestructEIP2929"
result[SLoad] = newIdentNode "sloadEIP2929"
result[SStore] = newIdentNode "sstoreEIP2929"
2020-11-25 09:43:34 +00:00
let BerlinOpDispatch {.compileTime.}: array[Op, NimNode] = genBerlinJumpTable(IstanbulOpDispatch)
proc genLondonJumpTable(ops: array[Op, NimNode]): array[Op, NimNode] {.compileTime.} =
result = ops
# EIP-3198 and EIP-3529
result[BaseFee] = newIdentNode "baseFee"
let LondonOpDispatch {.compileTime.}: array[Op, NimNode] = genLondonJumpTable(BerlinOpDispatch)
proc opTableToCaseStmt(opTable: array[Op, NimNode], c: NimNode): NimNode =
Refactor interpreter dispatch (#65) * move forks constants, rename errors * Move vm/utils to vm/interpreter/utils * initial opcodes refactoring * Add refactored Comparison & Bitwise Logic Operations * Add sha3 and address, simplify macro, support pop 0 * balance, origin, caller, callValue * fix gas copy opcodes gas costs, add callDataLoad/Size/Copy, CodeSize/Copy and gas price opcode * Update with 30s, 40s, 50s opcodes + impl of balance + stack improvement * add push, dup, swap, log, create and call operations * finish opcode implementation * Add the new dispatching logic * Pass the opcode test * Make test_vm_json compile * halt execution without exceptions for Return, Revert, selfdestruct (fix #62) * Properly catch and recover from EVM exceptions (stack underflow ...) * Fix byte op * Fix jump regressions * Update for latest devel, don't import old dispatch code as quasiBoolean macro is broken by latest devel * Fix sha3 regression on empty memory slice and until end of range slice * Fix padding / range error on expXY_success (gas computation left) * update logging procs * Add tracing - expXY_success is not a regression, sload stub was accidentally passing the test * Reuse the same stub as OO implementation * Delete previous opcode implementation * Delete object oriented fork code * Delete exceptions that were used as control flows * delete base.nim :fire:, yet another OO remnants * Delete opcode table * Enable omputed gotos and compile-time gas fees * Revert const gasCosts -> generates SIGSEGV * inline push, swap and dup opcodes * loggers are now template again, why does this pass new tests? * Trigger CI rebuild after rocksdb fix https://github.com/status-im/nim-rocksdb/pull/5 * Address review comment on "push" + VMTests in debug mode (not release) * Address review comment: don't tag fork by default, make opcode impl grepable * Static compilation fixes after rebasing * fix the initialization of the VM database * add a missing import * Deactivate balance and sload test following #59 * Reactivate stack check (deactivated in #59, necessary to pass tests) * Merge remaining opcodes implementation from #59 * Merge callDataLoad and codeCopy fixes, todo simplify see #67
2018-07-06 07:52:31 +00:00
let instr = quote do: `c`.instr
Refactor interpreter dispatch (#65) * move forks constants, rename errors * Move vm/utils to vm/interpreter/utils * initial opcodes refactoring * Add refactored Comparison & Bitwise Logic Operations * Add sha3 and address, simplify macro, support pop 0 * balance, origin, caller, callValue * fix gas copy opcodes gas costs, add callDataLoad/Size/Copy, CodeSize/Copy and gas price opcode * Update with 30s, 40s, 50s opcodes + impl of balance + stack improvement * add push, dup, swap, log, create and call operations * finish opcode implementation * Add the new dispatching logic * Pass the opcode test * Make test_vm_json compile * halt execution without exceptions for Return, Revert, selfdestruct (fix #62) * Properly catch and recover from EVM exceptions (stack underflow ...) * Fix byte op * Fix jump regressions * Update for latest devel, don't import old dispatch code as quasiBoolean macro is broken by latest devel * Fix sha3 regression on empty memory slice and until end of range slice * Fix padding / range error on expXY_success (gas computation left) * update logging procs * Add tracing - expXY_success is not a regression, sload stub was accidentally passing the test * Reuse the same stub as OO implementation * Delete previous opcode implementation * Delete object oriented fork code * Delete exceptions that were used as control flows * delete base.nim :fire:, yet another OO remnants * Delete opcode table * Enable omputed gotos and compile-time gas fees * Revert const gasCosts -> generates SIGSEGV * inline push, swap and dup opcodes * loggers are now template again, why does this pass new tests? * Trigger CI rebuild after rocksdb fix https://github.com/status-im/nim-rocksdb/pull/5 * Address review comment on "push" + VMTests in debug mode (not release) * Address review comment: don't tag fork by default, make opcode impl grepable * Static compilation fixes after rebasing * fix the initialization of the VM database * add a missing import * Deactivate balance and sload test following #59 * Reactivate stack check (deactivated in #59, necessary to pass tests) * Merge remaining opcodes implementation from #59 * Merge callDataLoad and codeCopy fixes, todo simplify see #67
2018-07-06 07:52:31 +00:00
result = nnkCaseStmt.newTree(instr)
# Add a branch for each (opcode, proc) pair
# We dispatch to the next instruction at the end of each branch
for op, opImpl in opTable.pairs:
2019-02-22 09:05:21 +00:00
let asOp = quote do: Op(`op`) # TODO: unfortunately when passing to runtime, ops are transformed into int
Refactor interpreter dispatch (#65) * move forks constants, rename errors * Move vm/utils to vm/interpreter/utils * initial opcodes refactoring * Add refactored Comparison & Bitwise Logic Operations * Add sha3 and address, simplify macro, support pop 0 * balance, origin, caller, callValue * fix gas copy opcodes gas costs, add callDataLoad/Size/Copy, CodeSize/Copy and gas price opcode * Update with 30s, 40s, 50s opcodes + impl of balance + stack improvement * add push, dup, swap, log, create and call operations * finish opcode implementation * Add the new dispatching logic * Pass the opcode test * Make test_vm_json compile * halt execution without exceptions for Return, Revert, selfdestruct (fix #62) * Properly catch and recover from EVM exceptions (stack underflow ...) * Fix byte op * Fix jump regressions * Update for latest devel, don't import old dispatch code as quasiBoolean macro is broken by latest devel * Fix sha3 regression on empty memory slice and until end of range slice * Fix padding / range error on expXY_success (gas computation left) * update logging procs * Add tracing - expXY_success is not a regression, sload stub was accidentally passing the test * Reuse the same stub as OO implementation * Delete previous opcode implementation * Delete object oriented fork code * Delete exceptions that were used as control flows * delete base.nim :fire:, yet another OO remnants * Delete opcode table * Enable omputed gotos and compile-time gas fees * Revert const gasCosts -> generates SIGSEGV * inline push, swap and dup opcodes * loggers are now template again, why does this pass new tests? * Trigger CI rebuild after rocksdb fix https://github.com/status-im/nim-rocksdb/pull/5 * Address review comment on "push" + VMTests in debug mode (not release) * Address review comment: don't tag fork by default, make opcode impl grepable * Static compilation fixes after rebasing * fix the initialization of the VM database * add a missing import * Deactivate balance and sload test following #59 * Reactivate stack check (deactivated in #59, necessary to pass tests) * Merge remaining opcodes implementation from #59 * Merge callDataLoad and codeCopy fixes, todo simplify see #67
2018-07-06 07:52:31 +00:00
let branchStmt = block:
if op == Stop:
quote do:
Tracing: Remove some trace messages that occur a lot during sync Disable some trace messages which appeared a lot in the output and probably aren't so useful any more, when block processing is functioning well at high speed. Turning on the trace level globally is useful to get a feel for what's happening, but only if each category is kept to a reasonable amount. As well as overwhelming the output so that it's hard to see general activity, some of these messages happen so much they severely slow down processing. Ones called every time an EVM opcode uses some gas are particularly extreme. These messages have all been chosen as things which are probably not useful any more (the relevant functionality has been debugged and is tested plenty). These have been commented out rather than removed. It may be that turning trace topics on/off, or other selection, is a better longer term solution, but that will require better command line options and good defaults for sure. (I think higher levels `tracev` and `tracevv` levels (extra verbose) would be more useful for this sort of deep tracing on request.) For now, enabling `--log-level:TRACE` on the command line is quite useful as long as we keep each category reasonable, and this patch tries to keep that balance. - Don't show "has transactions" on virtually every block imported. - Don't show "Sender" and "txHash" lines on every transaction processed. - Don't show "GAS CONSUMPTION" on every opcode executed", this is way too much. - Don't show "GAS RETURNED" and "GAS REFUND" on each contract call. - Don't show "op: Stop" on every Stop opcode, which means every transaction. - Don't show "Insufficient funds" whenever a contract can't call another. - Don't show "ECRecover", "SHA256 precompile", "RIPEMD160", "Identity" or even "Call precompile" every time a precompile is called. These are very well tested now. - Don't show "executeOpcodes error" whenever a contract returns an error. (This is changed to `trace` too, it's a normal event that is well tested.) Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-22 13:35:41 +00:00
#trace "op: Stop"
if not `c`.code.atEnd() and `c`.tracingEnabled:
2019-02-22 09:05:21 +00:00
# we only trace `REAL STOP` and ignore `FAKE STOP`
`c`.opIndex = `c`.traceOpCodeStarted(`asOp`)
`c`.traceOpCodeEnded(`asOp`, `c`.opIndex)
break
Refactor interpreter dispatch (#65) * move forks constants, rename errors * Move vm/utils to vm/interpreter/utils * initial opcodes refactoring * Add refactored Comparison & Bitwise Logic Operations * Add sha3 and address, simplify macro, support pop 0 * balance, origin, caller, callValue * fix gas copy opcodes gas costs, add callDataLoad/Size/Copy, CodeSize/Copy and gas price opcode * Update with 30s, 40s, 50s opcodes + impl of balance + stack improvement * add push, dup, swap, log, create and call operations * finish opcode implementation * Add the new dispatching logic * Pass the opcode test * Make test_vm_json compile * halt execution without exceptions for Return, Revert, selfdestruct (fix #62) * Properly catch and recover from EVM exceptions (stack underflow ...) * Fix byte op * Fix jump regressions * Update for latest devel, don't import old dispatch code as quasiBoolean macro is broken by latest devel * Fix sha3 regression on empty memory slice and until end of range slice * Fix padding / range error on expXY_success (gas computation left) * update logging procs * Add tracing - expXY_success is not a regression, sload stub was accidentally passing the test * Reuse the same stub as OO implementation * Delete previous opcode implementation * Delete object oriented fork code * Delete exceptions that were used as control flows * delete base.nim :fire:, yet another OO remnants * Delete opcode table * Enable omputed gotos and compile-time gas fees * Revert const gasCosts -> generates SIGSEGV * inline push, swap and dup opcodes * loggers are now template again, why does this pass new tests? * Trigger CI rebuild after rocksdb fix https://github.com/status-im/nim-rocksdb/pull/5 * Address review comment on "push" + VMTests in debug mode (not release) * Address review comment: don't tag fork by default, make opcode impl grepable * Static compilation fixes after rebasing * fix the initialization of the VM database * add a missing import * Deactivate balance and sload test following #59 * Reactivate stack check (deactivated in #59, necessary to pass tests) * Merge remaining opcodes implementation from #59 * Merge callDataLoad and codeCopy fixes, todo simplify see #67
2018-07-06 07:52:31 +00:00
else:
if BaseGasCosts[op].kind == GckFixed:
quote do:
if `c`.tracingEnabled:
`c`.opIndex = `c`.traceOpCodeStarted(`asOp`)
`c`.gasMeter.consumeGas(`c`.gasCosts[`asOp`].cost, reason = $`asOp`)
`opImpl`(`c`)
if `c`.tracingEnabled:
`c`.traceOpCodeEnded(`asOp`, `c`.opIndex)
EVM: Small patch that reduces EVM stack usage to almost nothing There's been a lot of talk about the Nimbus EVM "stack problem". I think we assumed changing it would require big changes to the interpreter code, touching a lot of functions. It turned out to be a low hanging fruit. This patch solves the stack problem, but hardly touches anything. The change in EVM stack memory is from 13 MB worst case to just 48 kB, a 250x reduction. I've been doing work on the database/storage/trie code. While looking at the API between the EVM and the database/storage/trie, this stack patch stood out and made itself obvious. As it's tiny, rather than more talk, here it is. Note: This patch is intentionally small, non-invasive, and hopefully easy to understand, so that it doesn't conflict with other work done on the EVM, and can easily be grafted into any other EVM structure. Motivation ========== - We run out of space and crash on some targets, unless the stack limit is raised above its default. Surprise segmentation faults are unhelpful. - Some CI targets have been disabled for months due to this. - Because usage borders on the system limits, when working on database/storage/trie/sync code (called from the EVM), segmentation faults occur and are misleading. They cause lost time due to thinking there's a crash bug in the code being worked on, when there's nothing wrong with it. - Sometimes unrelated, trivial code changes elsewhere trigger CI test failures. It looks like abrupt termination. A simple, recent patch was crashing in `make test` even though it was a trivial refactor. Turns out it pushed the stack over the edge. - A large stack has to be scanned by the Nim garbage collector sometimes. Larger stack means slower GC and memory allocation. - The structure of this small patch suggests how to weave async into the EVM with almost no changes to the EVM, and no async transformation overhead. - The patch seemed obvious when working on the API between EVM and storage. Measurements before =================== All these tests were run on Ubuntu 20.04 server, x86-64. This is one of the targets that has been disabled for a while in CI in EVMC mode due to crashing, and excessive stack usage is the cause. Testing commit 0c34a8e3 `2021-04-08 17:46:00 +0200 CI: use MSYS2 on Windows`. $ rm -f build/all_tests && make ENABLE_EVMC=1 test $ ulimit -S -s 16384 # Requires larger stack than default to avoid crash. $ ./build/all_tests 9 | tee tlog [Suite] persist block json tests ... Stack range 38496 depthHigh 3 ... Stack range 13140272 depthHigh 1024 [OK] tests/fixtures/PersistBlockTests/block1431916.json These tests use 13.14 MB of stack to run, and so crash with the default stack limit on Ubuntu Server 20.04 (8MB). Exactly 12832 bytes per EVM call stack frame. It's interesting to see some stack frames take a bit more. $ rm -f build/all_tests && make ENABLE_EVMC=1 test $ ulimit -S -s 16384 # Requires larger stack than default. $ ./build/all_tests 7 | tee tlog [Suite] new generalstate json tests ... Stack range 15488 depthHigh 2 ... Stack range 3539312 depthHigh 457 [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json ... Stack range 3756144 depthHigh 485 [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json ... Stack range 7929968 depthHigh 1024 [OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json These tests use 7.92MB of stack to run. About 7264 bytes per EVM call stack frame. It _only just_ avoids crashing with the default Ubuntu Server stack limit of 8 MB. However, it still crashes on Windows x86-64, which is why the CI target is currently disabled. On Linux where this passes, this is so borderline that it affects work and testing of storage and sync code, because that's called from the EVM. Which was a motivation for dealing with the stack instead of letting this linger. Also, this stack greatly exceeds the default thread stack size. $ rm -f build/all_tests && make ENABLE_EVMC=0 test $ ulimit -S -s 16384 # Requires larger stack than default to avoid crash. $ ./build/all_tests 9 | tee tlog [Suite] persist block json tests ... Stack range 33216 depthHigh 3 ... Stack range 11338032 depthHigh 1024 [OK] tests/fixtures/PersistBlockTests/block1431916.json These tests use 11.33 MB stack to run, and so crash with a default stack limit of 8MB. Exactly 11072 bytes per EVM call stack frame. It's interesting to see some stack frames take a bit more. $ rm -f build/all_tests && make ENABLE_EVMC=0 test $ ulimit -S -s 16384 # Requires larger stack than default. $ ./build/all_tests 7 | tee tlog [Suite] new generalstate json tests ... Stack range 10224 depthHigh 2 ... Stack range 2471760 depthHigh 457 [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json ... Stack range 2623184 depthHigh 485 [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json ... Stack range 5537824 depthHigh 1024 [OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json These tests use 5.54 MB of stack to run, and avoid crashing on with a default stack limit of 8 MB. About 5408 bytes per EVM call stack frame. However, this is uncomfortably close to the limit, as the stack frame size is sensitive to changes in the code. Also, this stack greatly exceeds the default thread stack size. Measurements after ================== (This patch doesn't address EVMC mode, which is not our default. EVMC stack usage remains about the same. EVMC mode is addressed in another tiny patch.) $ rm -f build/all_tests && make ENABLE_EVMC=0 test $ ulimit -S -s 80 # Because we can! 80k stack. $ ./build/all_tests 9 | tee tlog [Suite] persist block json tests ... Stack range 496 depthHigh 3 ... Stack range 49504 depthHigh 1024 [OK] tests/fixtures/PersistBlockTests/block1431916.json $ rm -f build/all_tests && make ENABLE_EVMC=0 test $ ulimit -S -s 72 # Because we can! 72k stack. $ ./build/all_tests 7 | tee tlog [Suite] new generalstate json tests ... Stack range 448 depthHigh 2 ... Stack range 22288 depthHigh 457 [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json ... Stack range 23632 depthHigh 485 [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json ... Stack range 49504 depthHigh 1024 [OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json For both tests, a satisfying *48 bytes* per EVM call stack frame, and EVM takes not much more than 48 kB. With other overheads, both tests run in 80 kB stack total at maximum EVM depth. We must add some headroom on this for database activity called from the EVM, and different compile targets. But it means the EVM itself is no longer a stack burden. This is much smaller than the default thread stack size on Linux (2MB), with plenty of margin. It's even smaller than Linux from a long time ago (128kB), and some small embedded C targets. (Just fyi, though, some JVM environments allocated just 32 kB to thread stacks.) This size is also well suited to running EVMs in threads, if that's useful. Subtle exception handling and `dispose` ======================================= It is important that each `snapshot` has a corresponding `dispose` in the event of an exception being raised. This code does do that, but in a subtle way. The pair of functions `execCallOrCreate` and `execCallOrCreateAux` are equivalent to the following code, where you can see `dispose` more clearly: proc execCallOrCreate*(c: Computation) = defer: c.dispose() if c.beforeExec(): return c.executeOpcodes() while not c.continuation.isNil: c.child.execCallOrCreate() c.child = nil (c.continuation)() c.executeOpcodes() c.afterExec() That works fine, but only reduces the stack used to 300-700 kB instead of 48 kB. To get lower we split the above into separate `execCallOrCreate` and `execCallOrCreateAux`. Only the outermost has `defer`, and instead of handling one level, it walks the entire `c.parent` chain calling `dispose` if needed. The inner one avoids `defer`, which greatly reduces the size of its stackframe. `c` is a `var` parameter, at each level of recursion. So the outermost proc sees the temporary changes made by all inner calls. This is why `c` is updated and the `c.parent` chain is maintained at each step. Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-12 17:06:31 +00:00
when `asOp` in {Create, Create2, Call, CallCode, DelegateCall, StaticCall}:
if not `c`.continuation.isNil:
return
Refactor interpreter dispatch (#65) * move forks constants, rename errors * Move vm/utils to vm/interpreter/utils * initial opcodes refactoring * Add refactored Comparison & Bitwise Logic Operations * Add sha3 and address, simplify macro, support pop 0 * balance, origin, caller, callValue * fix gas copy opcodes gas costs, add callDataLoad/Size/Copy, CodeSize/Copy and gas price opcode * Update with 30s, 40s, 50s opcodes + impl of balance + stack improvement * add push, dup, swap, log, create and call operations * finish opcode implementation * Add the new dispatching logic * Pass the opcode test * Make test_vm_json compile * halt execution without exceptions for Return, Revert, selfdestruct (fix #62) * Properly catch and recover from EVM exceptions (stack underflow ...) * Fix byte op * Fix jump regressions * Update for latest devel, don't import old dispatch code as quasiBoolean macro is broken by latest devel * Fix sha3 regression on empty memory slice and until end of range slice * Fix padding / range error on expXY_success (gas computation left) * update logging procs * Add tracing - expXY_success is not a regression, sload stub was accidentally passing the test * Reuse the same stub as OO implementation * Delete previous opcode implementation * Delete object oriented fork code * Delete exceptions that were used as control flows * delete base.nim :fire:, yet another OO remnants * Delete opcode table * Enable omputed gotos and compile-time gas fees * Revert const gasCosts -> generates SIGSEGV * inline push, swap and dup opcodes * loggers are now template again, why does this pass new tests? * Trigger CI rebuild after rocksdb fix https://github.com/status-im/nim-rocksdb/pull/5 * Address review comment on "push" + VMTests in debug mode (not release) * Address review comment: don't tag fork by default, make opcode impl grepable * Static compilation fixes after rebasing * fix the initialization of the VM database * add a missing import * Deactivate balance and sload test following #59 * Reactivate stack check (deactivated in #59, necessary to pass tests) * Merge remaining opcodes implementation from #59 * Merge callDataLoad and codeCopy fixes, todo simplify see #67
2018-07-06 07:52:31 +00:00
else:
quote do:
if `c`.tracingEnabled:
`c`.opIndex = `c`.traceOpCodeStarted(`asOp`)
`opImpl`(`c`)
if `c`.tracingEnabled:
`c`.traceOpCodeEnded(`asOp`, `c`.opIndex)
EVM: Small patch that reduces EVM stack usage to almost nothing There's been a lot of talk about the Nimbus EVM "stack problem". I think we assumed changing it would require big changes to the interpreter code, touching a lot of functions. It turned out to be a low hanging fruit. This patch solves the stack problem, but hardly touches anything. The change in EVM stack memory is from 13 MB worst case to just 48 kB, a 250x reduction. I've been doing work on the database/storage/trie code. While looking at the API between the EVM and the database/storage/trie, this stack patch stood out and made itself obvious. As it's tiny, rather than more talk, here it is. Note: This patch is intentionally small, non-invasive, and hopefully easy to understand, so that it doesn't conflict with other work done on the EVM, and can easily be grafted into any other EVM structure. Motivation ========== - We run out of space and crash on some targets, unless the stack limit is raised above its default. Surprise segmentation faults are unhelpful. - Some CI targets have been disabled for months due to this. - Because usage borders on the system limits, when working on database/storage/trie/sync code (called from the EVM), segmentation faults occur and are misleading. They cause lost time due to thinking there's a crash bug in the code being worked on, when there's nothing wrong with it. - Sometimes unrelated, trivial code changes elsewhere trigger CI test failures. It looks like abrupt termination. A simple, recent patch was crashing in `make test` even though it was a trivial refactor. Turns out it pushed the stack over the edge. - A large stack has to be scanned by the Nim garbage collector sometimes. Larger stack means slower GC and memory allocation. - The structure of this small patch suggests how to weave async into the EVM with almost no changes to the EVM, and no async transformation overhead. - The patch seemed obvious when working on the API between EVM and storage. Measurements before =================== All these tests were run on Ubuntu 20.04 server, x86-64. This is one of the targets that has been disabled for a while in CI in EVMC mode due to crashing, and excessive stack usage is the cause. Testing commit 0c34a8e3 `2021-04-08 17:46:00 +0200 CI: use MSYS2 on Windows`. $ rm -f build/all_tests && make ENABLE_EVMC=1 test $ ulimit -S -s 16384 # Requires larger stack than default to avoid crash. $ ./build/all_tests 9 | tee tlog [Suite] persist block json tests ... Stack range 38496 depthHigh 3 ... Stack range 13140272 depthHigh 1024 [OK] tests/fixtures/PersistBlockTests/block1431916.json These tests use 13.14 MB of stack to run, and so crash with the default stack limit on Ubuntu Server 20.04 (8MB). Exactly 12832 bytes per EVM call stack frame. It's interesting to see some stack frames take a bit more. $ rm -f build/all_tests && make ENABLE_EVMC=1 test $ ulimit -S -s 16384 # Requires larger stack than default. $ ./build/all_tests 7 | tee tlog [Suite] new generalstate json tests ... Stack range 15488 depthHigh 2 ... Stack range 3539312 depthHigh 457 [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json ... Stack range 3756144 depthHigh 485 [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json ... Stack range 7929968 depthHigh 1024 [OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json These tests use 7.92MB of stack to run. About 7264 bytes per EVM call stack frame. It _only just_ avoids crashing with the default Ubuntu Server stack limit of 8 MB. However, it still crashes on Windows x86-64, which is why the CI target is currently disabled. On Linux where this passes, this is so borderline that it affects work and testing of storage and sync code, because that's called from the EVM. Which was a motivation for dealing with the stack instead of letting this linger. Also, this stack greatly exceeds the default thread stack size. $ rm -f build/all_tests && make ENABLE_EVMC=0 test $ ulimit -S -s 16384 # Requires larger stack than default to avoid crash. $ ./build/all_tests 9 | tee tlog [Suite] persist block json tests ... Stack range 33216 depthHigh 3 ... Stack range 11338032 depthHigh 1024 [OK] tests/fixtures/PersistBlockTests/block1431916.json These tests use 11.33 MB stack to run, and so crash with a default stack limit of 8MB. Exactly 11072 bytes per EVM call stack frame. It's interesting to see some stack frames take a bit more. $ rm -f build/all_tests && make ENABLE_EVMC=0 test $ ulimit -S -s 16384 # Requires larger stack than default. $ ./build/all_tests 7 | tee tlog [Suite] new generalstate json tests ... Stack range 10224 depthHigh 2 ... Stack range 2471760 depthHigh 457 [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json ... Stack range 2623184 depthHigh 485 [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json ... Stack range 5537824 depthHigh 1024 [OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json These tests use 5.54 MB of stack to run, and avoid crashing on with a default stack limit of 8 MB. About 5408 bytes per EVM call stack frame. However, this is uncomfortably close to the limit, as the stack frame size is sensitive to changes in the code. Also, this stack greatly exceeds the default thread stack size. Measurements after ================== (This patch doesn't address EVMC mode, which is not our default. EVMC stack usage remains about the same. EVMC mode is addressed in another tiny patch.) $ rm -f build/all_tests && make ENABLE_EVMC=0 test $ ulimit -S -s 80 # Because we can! 80k stack. $ ./build/all_tests 9 | tee tlog [Suite] persist block json tests ... Stack range 496 depthHigh 3 ... Stack range 49504 depthHigh 1024 [OK] tests/fixtures/PersistBlockTests/block1431916.json $ rm -f build/all_tests && make ENABLE_EVMC=0 test $ ulimit -S -s 72 # Because we can! 72k stack. $ ./build/all_tests 7 | tee tlog [Suite] new generalstate json tests ... Stack range 448 depthHigh 2 ... Stack range 22288 depthHigh 457 [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json ... Stack range 23632 depthHigh 485 [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json ... Stack range 49504 depthHigh 1024 [OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json For both tests, a satisfying *48 bytes* per EVM call stack frame, and EVM takes not much more than 48 kB. With other overheads, both tests run in 80 kB stack total at maximum EVM depth. We must add some headroom on this for database activity called from the EVM, and different compile targets. But it means the EVM itself is no longer a stack burden. This is much smaller than the default thread stack size on Linux (2MB), with plenty of margin. It's even smaller than Linux from a long time ago (128kB), and some small embedded C targets. (Just fyi, though, some JVM environments allocated just 32 kB to thread stacks.) This size is also well suited to running EVMs in threads, if that's useful. Subtle exception handling and `dispose` ======================================= It is important that each `snapshot` has a corresponding `dispose` in the event of an exception being raised. This code does do that, but in a subtle way. The pair of functions `execCallOrCreate` and `execCallOrCreateAux` are equivalent to the following code, where you can see `dispose` more clearly: proc execCallOrCreate*(c: Computation) = defer: c.dispose() if c.beforeExec(): return c.executeOpcodes() while not c.continuation.isNil: c.child.execCallOrCreate() c.child = nil (c.continuation)() c.executeOpcodes() c.afterExec() That works fine, but only reduces the stack used to 300-700 kB instead of 48 kB. To get lower we split the above into separate `execCallOrCreate` and `execCallOrCreateAux`. Only the outermost has `defer`, and instead of handling one level, it walks the entire `c.parent` chain calling `dispose` if needed. The inner one avoids `defer`, which greatly reduces the size of its stackframe. `c` is a `var` parameter, at each level of recursion. So the outermost proc sees the temporary changes made by all inner calls. This is why `c` is updated and the `c.parent` chain is maintained at each step. Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-12 17:06:31 +00:00
when `asOp` in {Create, Create2, Call, CallCode, DelegateCall, StaticCall}:
if not `c`.continuation.isNil:
return
Refactor interpreter dispatch (#65) * move forks constants, rename errors * Move vm/utils to vm/interpreter/utils * initial opcodes refactoring * Add refactored Comparison & Bitwise Logic Operations * Add sha3 and address, simplify macro, support pop 0 * balance, origin, caller, callValue * fix gas copy opcodes gas costs, add callDataLoad/Size/Copy, CodeSize/Copy and gas price opcode * Update with 30s, 40s, 50s opcodes + impl of balance + stack improvement * add push, dup, swap, log, create and call operations * finish opcode implementation * Add the new dispatching logic * Pass the opcode test * Make test_vm_json compile * halt execution without exceptions for Return, Revert, selfdestruct (fix #62) * Properly catch and recover from EVM exceptions (stack underflow ...) * Fix byte op * Fix jump regressions * Update for latest devel, don't import old dispatch code as quasiBoolean macro is broken by latest devel * Fix sha3 regression on empty memory slice and until end of range slice * Fix padding / range error on expXY_success (gas computation left) * update logging procs * Add tracing - expXY_success is not a regression, sload stub was accidentally passing the test * Reuse the same stub as OO implementation * Delete previous opcode implementation * Delete object oriented fork code * Delete exceptions that were used as control flows * delete base.nim :fire:, yet another OO remnants * Delete opcode table * Enable omputed gotos and compile-time gas fees * Revert const gasCosts -> generates SIGSEGV * inline push, swap and dup opcodes * loggers are now template again, why does this pass new tests? * Trigger CI rebuild after rocksdb fix https://github.com/status-im/nim-rocksdb/pull/5 * Address review comment on "push" + VMTests in debug mode (not release) * Address review comment: don't tag fork by default, make opcode impl grepable * Static compilation fixes after rebasing * fix the initialization of the VM database * add a missing import * Deactivate balance and sload test following #59 * Reactivate stack check (deactivated in #59, necessary to pass tests) * Merge remaining opcodes implementation from #59 * Merge callDataLoad and codeCopy fixes, todo simplify see #67
2018-07-06 07:52:31 +00:00
when `asOp` in {Return, Revert, SelfDestruct}:
break
result.add nnkOfBranch.newTree(
newIdentNode($op),
branchStmt
)
# Wrap the case statement in while true + computed goto
result = quote do:
if `c`.tracingEnabled:
`c`.prepareTracer()
while true:
`instr` = `c`.code.next()
2019-04-04 10:23:28 +00:00
#{.computedGoto.}
# computed goto causing stack overflow, it consumes a lot of space
# we could use manual jump table instead
# TODO lots of macro magic here to unravel, with chronicles...
# `c`.logger.log($`c`.stack & "\n\n", fgGreen)
`result`
Refactor interpreter dispatch (#65) * move forks constants, rename errors * Move vm/utils to vm/interpreter/utils * initial opcodes refactoring * Add refactored Comparison & Bitwise Logic Operations * Add sha3 and address, simplify macro, support pop 0 * balance, origin, caller, callValue * fix gas copy opcodes gas costs, add callDataLoad/Size/Copy, CodeSize/Copy and gas price opcode * Update with 30s, 40s, 50s opcodes + impl of balance + stack improvement * add push, dup, swap, log, create and call operations * finish opcode implementation * Add the new dispatching logic * Pass the opcode test * Make test_vm_json compile * halt execution without exceptions for Return, Revert, selfdestruct (fix #62) * Properly catch and recover from EVM exceptions (stack underflow ...) * Fix byte op * Fix jump regressions * Update for latest devel, don't import old dispatch code as quasiBoolean macro is broken by latest devel * Fix sha3 regression on empty memory slice and until end of range slice * Fix padding / range error on expXY_success (gas computation left) * update logging procs * Add tracing - expXY_success is not a regression, sload stub was accidentally passing the test * Reuse the same stub as OO implementation * Delete previous opcode implementation * Delete object oriented fork code * Delete exceptions that were used as control flows * delete base.nim :fire:, yet another OO remnants * Delete opcode table * Enable omputed gotos and compile-time gas fees * Revert const gasCosts -> generates SIGSEGV * inline push, swap and dup opcodes * loggers are now template again, why does this pass new tests? * Trigger CI rebuild after rocksdb fix https://github.com/status-im/nim-rocksdb/pull/5 * Address review comment on "push" + VMTests in debug mode (not release) * Address review comment: don't tag fork by default, make opcode impl grepable * Static compilation fixes after rebasing * fix the initialization of the VM database * add a missing import * Deactivate balance and sload test following #59 * Reactivate stack check (deactivated in #59, necessary to pass tests) * Merge remaining opcodes implementation from #59 * Merge callDataLoad and codeCopy fixes, todo simplify see #67
2018-07-06 07:52:31 +00:00
macro genFrontierDispatch(c: Computation): untyped =
result = opTableToCaseStmt(FrontierOpDispatch, c)
Refactor interpreter dispatch (#65) * move forks constants, rename errors * Move vm/utils to vm/interpreter/utils * initial opcodes refactoring * Add refactored Comparison & Bitwise Logic Operations * Add sha3 and address, simplify macro, support pop 0 * balance, origin, caller, callValue * fix gas copy opcodes gas costs, add callDataLoad/Size/Copy, CodeSize/Copy and gas price opcode * Update with 30s, 40s, 50s opcodes + impl of balance + stack improvement * add push, dup, swap, log, create and call operations * finish opcode implementation * Add the new dispatching logic * Pass the opcode test * Make test_vm_json compile * halt execution without exceptions for Return, Revert, selfdestruct (fix #62) * Properly catch and recover from EVM exceptions (stack underflow ...) * Fix byte op * Fix jump regressions * Update for latest devel, don't import old dispatch code as quasiBoolean macro is broken by latest devel * Fix sha3 regression on empty memory slice and until end of range slice * Fix padding / range error on expXY_success (gas computation left) * update logging procs * Add tracing - expXY_success is not a regression, sload stub was accidentally passing the test * Reuse the same stub as OO implementation * Delete previous opcode implementation * Delete object oriented fork code * Delete exceptions that were used as control flows * delete base.nim :fire:, yet another OO remnants * Delete opcode table * Enable omputed gotos and compile-time gas fees * Revert const gasCosts -> generates SIGSEGV * inline push, swap and dup opcodes * loggers are now template again, why does this pass new tests? * Trigger CI rebuild after rocksdb fix https://github.com/status-im/nim-rocksdb/pull/5 * Address review comment on "push" + VMTests in debug mode (not release) * Address review comment: don't tag fork by default, make opcode impl grepable * Static compilation fixes after rebasing * fix the initialization of the VM database * add a missing import * Deactivate balance and sload test following #59 * Reactivate stack check (deactivated in #59, necessary to pass tests) * Merge remaining opcodes implementation from #59 * Merge callDataLoad and codeCopy fixes, todo simplify see #67
2018-07-06 07:52:31 +00:00
macro genHomesteadDispatch(c: Computation): untyped =
result = opTableToCaseStmt(HomesteadOpDispatch, c)
macro genTangerineDispatch(c: Computation): untyped =
result = opTableToCaseStmt(TangerineOpDispatch, c)
macro genSpuriousDispatch(c: Computation): untyped =
result = opTableToCaseStmt(SpuriousOpDispatch, c)
macro genByzantiumDispatch(c: Computation): untyped =
result = opTableToCaseStmt(ByzantiumOpDispatch, c)
2019-04-22 09:26:59 +00:00
macro genConstantinopleDispatch(c: Computation): untyped =
result = opTableToCaseStmt(ConstantinopleOpDispatch, c)
2020-03-04 09:32:30 +00:00
macro genPetersburgDispatch(c: Computation): untyped =
result = opTableToCaseStmt(PetersburgOpDispatch, c)
macro genIstanbulDispatch(c: Computation): untyped =
result = opTableToCaseStmt(IstanbulOpDispatch, c)
2019-11-11 04:58:58 +00:00
2020-11-25 09:43:34 +00:00
macro genBerlinDispatch(c: Computation): untyped =
result = opTableToCaseStmt(BerlinOpDispatch, c)
macro genLondonDispatch(c: Computation): untyped =
result = opTableToCaseStmt(LondonOpDispatch, c)
proc frontierVM(c: Computation) =
genFrontierDispatch(c)
Refactor interpreter dispatch (#65) * move forks constants, rename errors * Move vm/utils to vm/interpreter/utils * initial opcodes refactoring * Add refactored Comparison & Bitwise Logic Operations * Add sha3 and address, simplify macro, support pop 0 * balance, origin, caller, callValue * fix gas copy opcodes gas costs, add callDataLoad/Size/Copy, CodeSize/Copy and gas price opcode * Update with 30s, 40s, 50s opcodes + impl of balance + stack improvement * add push, dup, swap, log, create and call operations * finish opcode implementation * Add the new dispatching logic * Pass the opcode test * Make test_vm_json compile * halt execution without exceptions for Return, Revert, selfdestruct (fix #62) * Properly catch and recover from EVM exceptions (stack underflow ...) * Fix byte op * Fix jump regressions * Update for latest devel, don't import old dispatch code as quasiBoolean macro is broken by latest devel * Fix sha3 regression on empty memory slice and until end of range slice * Fix padding / range error on expXY_success (gas computation left) * update logging procs * Add tracing - expXY_success is not a regression, sload stub was accidentally passing the test * Reuse the same stub as OO implementation * Delete previous opcode implementation * Delete object oriented fork code * Delete exceptions that were used as control flows * delete base.nim :fire:, yet another OO remnants * Delete opcode table * Enable omputed gotos and compile-time gas fees * Revert const gasCosts -> generates SIGSEGV * inline push, swap and dup opcodes * loggers are now template again, why does this pass new tests? * Trigger CI rebuild after rocksdb fix https://github.com/status-im/nim-rocksdb/pull/5 * Address review comment on "push" + VMTests in debug mode (not release) * Address review comment: don't tag fork by default, make opcode impl grepable * Static compilation fixes after rebasing * fix the initialization of the VM database * add a missing import * Deactivate balance and sload test following #59 * Reactivate stack check (deactivated in #59, necessary to pass tests) * Merge remaining opcodes implementation from #59 * Merge callDataLoad and codeCopy fixes, todo simplify see #67
2018-07-06 07:52:31 +00:00
proc homesteadVM(c: Computation) =
genHomesteadDispatch(c)
proc tangerineVM(c: Computation) =
genTangerineDispatch(c)
proc spuriousVM(c: Computation) {.gcsafe.} =
genSpuriousDispatch(c)
proc byzantiumVM(c: Computation) {.gcsafe.} =
genByzantiumDispatch(c)
2019-04-22 09:26:59 +00:00
proc constantinopleVM(c: Computation) {.gcsafe.} =
genConstantinopleDispatch(c)
2020-03-04 09:32:30 +00:00
proc petersburgVM(c: Computation) {.gcsafe.} =
genPetersburgDispatch(c)
proc istanbulVM(c: Computation) {.gcsafe.} =
genIstanbulDispatch(c)
2019-11-11 04:58:58 +00:00
2020-11-25 09:43:34 +00:00
proc berlinVM(c: Computation) {.gcsafe.} =
genBerlinDispatch(c)
proc londonVM(c: Computation) {.gcsafe.} =
genLondonDispatch(c)
proc selectVM(c: Computation, fork: Fork) {.gcsafe.} =
# TODO: Optimise getting fork and updating opCodeExec only when necessary
2019-03-19 16:30:35 +00:00
case fork
2020-04-12 10:33:17 +00:00
of FkFrontier:
c.frontierVM()
2020-04-12 10:33:17 +00:00
of FkHomestead:
c.homesteadVM()
of FkTangerine:
c.tangerineVM()
of FkSpurious:
c.spuriousVM()
2019-11-11 04:58:58 +00:00
of FkByzantium:
c.byzantiumVM()
2019-11-11 04:58:58 +00:00
of FkConstantinople:
c.constantinopleVM()
2020-03-04 09:32:30 +00:00
of FkPetersburg:
c.petersburgVM()
2020-11-25 09:43:34 +00:00
of FkIstanbul:
c.istanbulVM()
of FkBerlin:
2020-11-25 09:43:34 +00:00
c.berlinVM()
else:
c.londonVM()
proc executeOpcodes(c: Computation) =
let fork = c.fork
2019-05-08 16:18:06 +00:00
block:
if c.continuation.isNil and c.execPrecompiles(fork):
2019-05-08 16:18:06 +00:00
break
2019-04-15 04:34:41 +00:00
2019-05-08 16:18:06 +00:00
try:
if not c.continuation.isNil:
(c.continuation)()
c.selectVM(fork)
2019-12-04 12:36:16 +00:00
except CatchableError as e:
c.setError(&"Opcode Dispatch Error msg={e.msg}, depth={c.msg.depth}", true)
2019-05-08 16:18:06 +00:00
EVM: Small patch that reduces EVM stack usage to almost nothing There's been a lot of talk about the Nimbus EVM "stack problem". I think we assumed changing it would require big changes to the interpreter code, touching a lot of functions. It turned out to be a low hanging fruit. This patch solves the stack problem, but hardly touches anything. The change in EVM stack memory is from 13 MB worst case to just 48 kB, a 250x reduction. I've been doing work on the database/storage/trie code. While looking at the API between the EVM and the database/storage/trie, this stack patch stood out and made itself obvious. As it's tiny, rather than more talk, here it is. Note: This patch is intentionally small, non-invasive, and hopefully easy to understand, so that it doesn't conflict with other work done on the EVM, and can easily be grafted into any other EVM structure. Motivation ========== - We run out of space and crash on some targets, unless the stack limit is raised above its default. Surprise segmentation faults are unhelpful. - Some CI targets have been disabled for months due to this. - Because usage borders on the system limits, when working on database/storage/trie/sync code (called from the EVM), segmentation faults occur and are misleading. They cause lost time due to thinking there's a crash bug in the code being worked on, when there's nothing wrong with it. - Sometimes unrelated, trivial code changes elsewhere trigger CI test failures. It looks like abrupt termination. A simple, recent patch was crashing in `make test` even though it was a trivial refactor. Turns out it pushed the stack over the edge. - A large stack has to be scanned by the Nim garbage collector sometimes. Larger stack means slower GC and memory allocation. - The structure of this small patch suggests how to weave async into the EVM with almost no changes to the EVM, and no async transformation overhead. - The patch seemed obvious when working on the API between EVM and storage. Measurements before =================== All these tests were run on Ubuntu 20.04 server, x86-64. This is one of the targets that has been disabled for a while in CI in EVMC mode due to crashing, and excessive stack usage is the cause. Testing commit 0c34a8e3 `2021-04-08 17:46:00 +0200 CI: use MSYS2 on Windows`. $ rm -f build/all_tests && make ENABLE_EVMC=1 test $ ulimit -S -s 16384 # Requires larger stack than default to avoid crash. $ ./build/all_tests 9 | tee tlog [Suite] persist block json tests ... Stack range 38496 depthHigh 3 ... Stack range 13140272 depthHigh 1024 [OK] tests/fixtures/PersistBlockTests/block1431916.json These tests use 13.14 MB of stack to run, and so crash with the default stack limit on Ubuntu Server 20.04 (8MB). Exactly 12832 bytes per EVM call stack frame. It's interesting to see some stack frames take a bit more. $ rm -f build/all_tests && make ENABLE_EVMC=1 test $ ulimit -S -s 16384 # Requires larger stack than default. $ ./build/all_tests 7 | tee tlog [Suite] new generalstate json tests ... Stack range 15488 depthHigh 2 ... Stack range 3539312 depthHigh 457 [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json ... Stack range 3756144 depthHigh 485 [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json ... Stack range 7929968 depthHigh 1024 [OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json These tests use 7.92MB of stack to run. About 7264 bytes per EVM call stack frame. It _only just_ avoids crashing with the default Ubuntu Server stack limit of 8 MB. However, it still crashes on Windows x86-64, which is why the CI target is currently disabled. On Linux where this passes, this is so borderline that it affects work and testing of storage and sync code, because that's called from the EVM. Which was a motivation for dealing with the stack instead of letting this linger. Also, this stack greatly exceeds the default thread stack size. $ rm -f build/all_tests && make ENABLE_EVMC=0 test $ ulimit -S -s 16384 # Requires larger stack than default to avoid crash. $ ./build/all_tests 9 | tee tlog [Suite] persist block json tests ... Stack range 33216 depthHigh 3 ... Stack range 11338032 depthHigh 1024 [OK] tests/fixtures/PersistBlockTests/block1431916.json These tests use 11.33 MB stack to run, and so crash with a default stack limit of 8MB. Exactly 11072 bytes per EVM call stack frame. It's interesting to see some stack frames take a bit more. $ rm -f build/all_tests && make ENABLE_EVMC=0 test $ ulimit -S -s 16384 # Requires larger stack than default. $ ./build/all_tests 7 | tee tlog [Suite] new generalstate json tests ... Stack range 10224 depthHigh 2 ... Stack range 2471760 depthHigh 457 [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json ... Stack range 2623184 depthHigh 485 [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json ... Stack range 5537824 depthHigh 1024 [OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json These tests use 5.54 MB of stack to run, and avoid crashing on with a default stack limit of 8 MB. About 5408 bytes per EVM call stack frame. However, this is uncomfortably close to the limit, as the stack frame size is sensitive to changes in the code. Also, this stack greatly exceeds the default thread stack size. Measurements after ================== (This patch doesn't address EVMC mode, which is not our default. EVMC stack usage remains about the same. EVMC mode is addressed in another tiny patch.) $ rm -f build/all_tests && make ENABLE_EVMC=0 test $ ulimit -S -s 80 # Because we can! 80k stack. $ ./build/all_tests 9 | tee tlog [Suite] persist block json tests ... Stack range 496 depthHigh 3 ... Stack range 49504 depthHigh 1024 [OK] tests/fixtures/PersistBlockTests/block1431916.json $ rm -f build/all_tests && make ENABLE_EVMC=0 test $ ulimit -S -s 72 # Because we can! 72k stack. $ ./build/all_tests 7 | tee tlog [Suite] new generalstate json tests ... Stack range 448 depthHigh 2 ... Stack range 22288 depthHigh 457 [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest639.json ... Stack range 23632 depthHigh 485 [OK] tests/fixtures/eth_tests/GeneralStateTests/stRandom2/randomStatetest458.json ... Stack range 49504 depthHigh 1024 [OK] tests/fixtures/eth_tests/GeneralStateTests/stCreate2/Create2OnDepth1024.json For both tests, a satisfying *48 bytes* per EVM call stack frame, and EVM takes not much more than 48 kB. With other overheads, both tests run in 80 kB stack total at maximum EVM depth. We must add some headroom on this for database activity called from the EVM, and different compile targets. But it means the EVM itself is no longer a stack burden. This is much smaller than the default thread stack size on Linux (2MB), with plenty of margin. It's even smaller than Linux from a long time ago (128kB), and some small embedded C targets. (Just fyi, though, some JVM environments allocated just 32 kB to thread stacks.) This size is also well suited to running EVMs in threads, if that's useful. Subtle exception handling and `dispose` ======================================= It is important that each `snapshot` has a corresponding `dispose` in the event of an exception being raised. This code does do that, but in a subtle way. The pair of functions `execCallOrCreate` and `execCallOrCreateAux` are equivalent to the following code, where you can see `dispose` more clearly: proc execCallOrCreate*(c: Computation) = defer: c.dispose() if c.beforeExec(): return c.executeOpcodes() while not c.continuation.isNil: c.child.execCallOrCreate() c.child = nil (c.continuation)() c.executeOpcodes() c.afterExec() That works fine, but only reduces the stack used to 300-700 kB instead of 48 kB. To get lower we split the above into separate `execCallOrCreate` and `execCallOrCreateAux`. Only the outermost has `defer`, and instead of handling one level, it walks the entire `c.parent` chain calling `dispose` if needed. The inner one avoids `defer`, which greatly reduces the size of its stackframe. `c` is a `var` parameter, at each level of recursion. So the outermost proc sees the temporary changes made by all inner calls. This is why `c` is updated and the `c.parent` chain is maintained at each step. Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-12 17:06:31 +00:00
if c.isError() and c.continuation.isNil:
if c.tracingEnabled: c.traceError()
Tracing: Remove some trace messages that occur a lot during sync Disable some trace messages which appeared a lot in the output and probably aren't so useful any more, when block processing is functioning well at high speed. Turning on the trace level globally is useful to get a feel for what's happening, but only if each category is kept to a reasonable amount. As well as overwhelming the output so that it's hard to see general activity, some of these messages happen so much they severely slow down processing. Ones called every time an EVM opcode uses some gas are particularly extreme. These messages have all been chosen as things which are probably not useful any more (the relevant functionality has been debugged and is tested plenty). These have been commented out rather than removed. It may be that turning trace topics on/off, or other selection, is a better longer term solution, but that will require better command line options and good defaults for sure. (I think higher levels `tracev` and `tracevv` levels (extra verbose) would be more useful for this sort of deep tracing on request.) For now, enabling `--log-level:TRACE` on the command line is quite useful as long as we keep each category reasonable, and this patch tries to keep that balance. - Don't show "has transactions" on virtually every block imported. - Don't show "Sender" and "txHash" lines on every transaction processed. - Don't show "GAS CONSUMPTION" on every opcode executed", this is way too much. - Don't show "GAS RETURNED" and "GAS REFUND" on each contract call. - Don't show "op: Stop" on every Stop opcode, which means every transaction. - Don't show "Insufficient funds" whenever a contract can't call another. - Don't show "ECRecover", "SHA256 precompile", "RIPEMD160", "Identity" or even "Call precompile" every time a precompile is called. These are very well tested now. - Don't show "executeOpcodes error" whenever a contract returns an error. (This is changed to `trace` too, it's a normal event that is well tested.) Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-07-22 13:35:41 +00:00
#trace "executeOpcodes error", msg=c.error.info