Commit Graph

39 Commits

Author SHA1 Message Date
andri lim 4fa3756860
Convert GasInt to uint64, bump nim-eth and nimbus-eth2 (#2461)
* Convert GasInt to uint64, bump nim-eth and nimbus-eth2

* Bump nimbus-eth2

* int64.high.GasInt instead of 0x7fffffffffffffff.GasInt
2024-07-07 06:52:11 +00:00
andri lim 23c00ce88c
Separate evmc gasCosts and nim-evm gasCosts (#2454)
This is part of preparations before converting GasInt to uint64
2024-07-05 07:00:03 +07:00
Jacek Sieka 768307d91d
Cache code and invalid jump destination tables (fixes #2268) (#2404)
It is common for many accounts to share the same code - at the database
level, code is stored by hash meaning only one copy exists per unique
program but when loaded in memory, a copy is made for each account.

Further, every time we execute the code, it must be scanned for invalid
jump destinations which slows down EVM exeuction.

Finally, the extcodesize call causes code to be loaded even if only the
size is needed.

This PR improves on all these points by introducing a shared
CodeBytesRef type whose code section is immutable and that can be shared
between accounts. Further, a dedicated `len` API call is added so that
the EXTCODESIZE opcode can operate without polluting the GC and code
cache, for cases where only the size is requested - rocksdb will in this
case cache the code itself in the row cache meaning that lookup of the
code itself remains fast when length is asked for first.

With 16k code entries, there's a 90% hit rate which goes up to 99%
during the 2.3M attack - the cache significantly lowers memory
consumption and execution time not only during this event but across the
board.
2024-06-21 09:44:10 +02:00
andri lim 61a809cf4d
Remove EVM indirect imports and unused EVM errors (#2370)
Those indirect imports are used when there was two EVMs.
2024-06-17 09:56:39 +02:00
andri lim 5a18537450
Bump nim-eth, nim-web3, nimbus-eth2 (#2344)
* Bump nim-eth, nim-web3, nimbus-eth2

- Replace std.Option with results.Opt
- Fields name changes

* More fixes

* Fix Portal stream async raises and portal testnet Opt usage

* Bump eth + nimbus-eth2 + more fixes related to eth_types changes

* Fix in utp test app and nimbus-eth2 bump

* Fix test_blockchain_json rebase conflict

* Fix EVMC block_timestamp conversion plus commentary

---------

Co-authored-by: kdeme <kim.demey@gmail.com>
2024-06-14 14:31:08 +07:00
andri lim b09fba0b49
Refactor EVM raises to reduce compiler warnings (#2040)
* Fix style check when EVMC enabled

* Refactor EVM raises to reduce compiler warnings

* Fix EVM handler raises when EVMC enabled

* Workaround stupid style checker false complaints

* Fix Windows/clang linking error due to function pointer incompatibility
2024-02-20 14:16:12 +07:00
andri lim bea558740f
Reduce compiler warnings (#2030)
* Reduce compiler warnings

* Reduce compiler warnings in test code
2024-02-16 16:08:07 +07:00
Jordan Hrycaj b623909c44
Ledger activate unified accounts cache wrapper (#1939)
* Activate `LedgerRef` wrapper for `AccountsCache`

details:
  `accounts_cache.nim` methods are indirectly processed by the wrapper
  methods from `ledger.nim`.

  This works for all sources except `test_state_db.nim` where the source
  `accounts_cache.nim` is included (rather than imported) in order to
  access objects privy to the very source.

* Provide facility to switch to a preselected `LedgerRef` type

details:
  Can be set as suggestion when initialising `CommonRef`

* Update `CoreDb` test suite for better time tracking

details:
+ Allow time logging by pre-defined block intervals
+ Print `CoreDb`/`Ledger`profiling results (if enabled)
2023-12-12 19:12:56 +00:00
andri lim 04c7ed8ec9
Change EthTime from std.Time to distinct uint64 (#1820)
* Change EthTime from std.Time to distinct uint64

* Bump nimbus-eth2
2023-10-18 09:16:11 +07:00
andri lim b7365085ae
Implement EIP-7516: BLOBBASEFEE opcode (#1791)
* Implement EIP-7516: BLOBBASEFEE opcode
2023-10-01 14:24:15 +07:00
andri lim 6848b2010b
EIP-6780: Fix self destruct operation (#1777) 2023-09-26 09:03:21 +07:00
andri lim dc1dcfb206
EIP-4844: Fix excessBlobGas handling (#1771)
* EIP-4844: Fix excessBlobGas handling

* Change vmState.difficulty to vmState.difficultyOrPrevRandao
2023-09-24 22:25:41 +07:00
jangko 26620eb672
EVM embrace more EVMC types
Also embed evmc_status_code to computation.error, and make
the tracer produce cleaner output. No more "Revert opcode executed"
error message. We can distinguish error code between REVERT
and FAILURE in a more cleaner way.
2023-08-28 21:36:23 +07:00
jangko 467e6fffa6
Implement EIP-6780: SELFDESTRUCT only in same transaction 2023-08-18 10:22:34 +07:00
jangko dd9e181acc
EIP-1153: completing transient storage on EVMC side 2023-08-18 10:19:34 +07:00
andri lim bdaeedb09f
rename data gas to blob gas (#1659)
* rename data gas to blob gas

* bump more submodules
* extend evmc tx_context with EIP-4844 blob_hashes
2023-08-04 19:43:30 +07:00
Kim De Mey 408394a2bd
Bump nim-eth and remove unneeded Defect raises (#1575) 2023-05-10 18:04:35 +02:00
jangko 2a3c67b4be
refactor touched accounts, selfdestruct, and log
simplify EVM and delegete those things to accounts cache.
also no more manual state clearing, accounts cache will be
responsible for both collecting touched account and perform
state clearing.
2023-03-21 08:14:06 +07:00
jangko 633f135c72
align accounts cache with EIP158/161
Some nomenclature used in accounts cache are not what described
in EIP158/161, therefore causing confusion and introduce bugs.
Now it should be fixed.
2023-03-18 18:13:06 +07:00
jangko 94a94c5b65 implement better hardfork management 2022-12-02 13:51:42 +07:00
jangko 4b142ac52d
upgrade evmc to v10.0.0
fixes #1172, fixes #950
2022-10-03 16:38:32 +07:00
jangko 28cdfcaf6b
fix EIP-4399 'random' opcode
- fix previous implementation of EIP-4399
- now `random` opcode can be used with evmc_enabled
2022-02-08 20:23:40 +07:00
jangko 71aa7e4b5c EIP-4399 implementation of nim-vm
what's new:
- `RANDOM` OPCODE
- `random` field of BlockHeader(previously `mixDigest`)
- `PostMerge` temporary name
2022-02-01 18:11:14 +02:00
Jordan Hrycaj 261c0b51a7
Redesign of BaseVMState descriptor (#923)
* Redesign of BaseVMState descriptor

why:
  BaseVMState provides an environment for executing transactions. The
  current descriptor also provides data that cannot generally be known
  within the execution environment, e.g. the total gasUsed which is
  available not before after all transactions have finished.

  Also, the BaseVMState constructor has been replaced by a constructor
  that does not need pre-initialised input of the account database.

also:
  Previous constructor and some fields are provided with a deprecated
  annotation (producing a lot of noise.)

* Replace legacy directives in production sources

* Replace legacy directives in unit test sources

* fix CI (missing premix update)

* Remove legacy directives

* chase CI problem

* rebased

* Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation

why:
  Constructing a new 'AccountsCache' descriptor can be avoided sometimes
  when the current state root is properly positioned already. Such a
  feature existed already as the update function 'initStateDB()' for the
  'BaseChanDB' where the accounts cache was linked into this desctiptor.

  The function 'initStateDB()' was removed and re-implemented into the
  'BaseVmState' constructor without optimisation. The old version was of
  restricted use as a wrong accounts cache state would unconditionally
  throw an exception rather than conceptually ask for a remedy.

  The optimised 'BaseVmState' re-initialisation has been implemented for
  the 'persistBlocks()' function.

also:
  moved some test helpers to 'test/replay' folder

* Remove unused & undocumented fields from Chain descriptor

why:
  Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
Jamie Lokier 6ef9bfd21b
EVMC: Byte-endian conversions for 256-bit numeric values
Perform byte-endian conversion for 256-bit numeric values, but not 256-bit
hashes.  These conversions are necessary for EVMC binary compatibility.

In new EVMC, all host-side conversions are explicit, calling `flip256`.

These conversions are performed in the EVMC "glue" code, which deals with the
binary interface, so the host services aren't aware of conversions.

We intend to skip these conversions when Nimbus host calls Nimbus EVM, even
when it's a shared library, using a negotiated EVMC extension.  But for now
we're focused on correctness and cross-validation with third party EVMs.

The overhead of endian conversion is not too high because most EVMC host calls
access the database anyway.  `getTxContext` does not, so the conversions from
that are cached here.  Also, well-optimised EVMs don't call it often.

It is arguable whether endian conversion should occur for storage slots (`key`).

In favour of no conversion: Slot keys are 32-byte blobs, and this is clear in
the EVMC definition where slot keys are `evmc_bytes32` (not `evmc_uint256be`),
meaning treating as a number is _not_ expected by EVMC.  Although they are
often small numbers, sometimes they are a hash from the contract code plus a
number.  Slot keys are hashed on the host side with Keccak256 before any
database calls, so the host side does not look at them numerically.

In favour of conversion: They are often small numbers and it is helpful to log
them as such, rather than a long string of zero digits with 1-2 non-zero.  The
representation in JSON has leading zeros removed, like a number rather than a
32-byte blob.  There is also an interesting space optimisation when the keys
are used unhashed in storage.

Nimbus currently treats slot keys on the host side as numbers, and the tests
pass when endian conversion is done.  So to remain consistent with other parts
of Nimbus we convert slot keys.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-12-10 16:23:27 +00:00
bmoo b09ad5cacb
code cleanup removed unused imports 2021-08-18 10:35:36 +07:00
Jamie Lokier 3047c839dc
EVMC: Improve host call tracing and fix nested call C stack usage
This combines two things, a C stack usage change with EVM nested calls
via EVMC, and changes to host call tracing.

Feature-wise, the tracing is improved:

- Storage keys and values are make more sense.
- The message/result/context objects are shown with all relevant fields.
- `call` trace is split into entry/exit, so these can be shown around the
  called contract's operations, instead of only showing the `call` parameters
  after the nested call is finished.
- Nested calls are indented, which helps to highlight the flow.
- C stack usage considerably reduced in nested calls when more functionality
  is enabled (either tracing here, or other things to come).

This will seem like a minor patch, but C stack usage was the real motivation,
after plenty of time in the debugger.

Nobody cares about stack when `showTxCalls` (you can just use a big stack when
debugging).  But these subtle changes around the `call` path were found to be
necessary for passing all tests when the EVMC nested call code is completed,
and that's a prerequisite for many things: async EVM, dynamic EVM, Beam Sync,
and to fix https://github.com/status-im/nimbus-eth1/issues/345.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-08-12 07:48:56 +07:00
Jamie Lokier 74f53c7761
EVMC: Add missing EIP-2929 (Berlin) functions to EVMC host
The update for London (EIP-1559) in 1cdb30df ("bump nim-emvc with evmc revision
8.0.0 to 9.0.0") really bumped EVMC ABI version from 7.5 up to 9.

In other words, it skipped Berlin, going direct from Istanbul to London.

That was accompanied by EVMC changes in 05e9b891 ("EIP-3198: add baseFee op
code in nim-evm"), which added the API changes needed for London.

But the missing Berlin functions weren't added in the move to London.

As a result, our EVMC host became incompatible with Berlin, London, and really
all revisions of the ABI, and if a third party EVM was loaded, it crashed.

This commit adds the missing Berlin host support, and makes our ABI
binary-compatible with real EVMC again.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-08-11 19:47:34 +07:00
jangko 697b38b844
EIP-3529: Replace SSTORE_CLEARS_SCHEDULE in evmc host_service 2021-06-30 20:35:10 +07:00
jangko 05e9b891f0
EIP-3198: add baseFee op code in nim-evm 2021-06-29 07:35:16 +07:00
Jamie Lokier 6d4205b0b0
Transaction: Just enough support to work with nested calls
Proper nested call functionality is being skipped in this iteration of new EVMC
host code to keep it simpler, to allow testing and architecture to be built
around the less complicated non-nested cases first.

Instead, nested calls use the old `Computation` path, and bypass any
third-party EVM that may be loaded.  The results are the same, and mixing
different EVMs in this way is actually permitted in the EVMC specification.

This approach also means third-party EVMs we test don't need to support
precompiles and we don't need to specially handle those cases.
(E.g. "evmone" doesn't support precompiles, just EVM1 opcodes).

(These before/after scope actions are approximately copy-pasted from
`nimbus/vm/evmc_host.nim`, making their detailed behaviour "obviously correct".
Of course they are subject to tests as well.  The small stack property of
a3c8a5c3 "EVMC: Small stacks when using EVMC, closes #575 (segfaults)" is
carefully retained.)

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 18:29:41 +01:00
Jamie Lokier 97b4aa5619
Transaction: Calculate EIP-1283/2200/2929 gas refund in `setStorage`
Make the host service `setStorage` calculate the gas refund itself, instead of
depending on EVM internals.

In EVMC, host `setStorage` is responsible for adding the gas refund using the
rules of EIP-1283 (Constantinople), EIP-2200 (Istanbul) or EIP-2929 (Berlin).

It is not obvious that the host must do it from EVMC documentation, but it's
how it has to be.  Note, this is very different from the gas _cost_, which the
host does not calculate.

Gas refund doesn't affect computation.  It is applied after the whole
transaction is complete, so it can be tracked on the host or EVM side.  But
`setStorage` doesn't return enough information for the EVM to calculate the
refund, so the host must do it when `setStorage` is used.

For now, this continues using Nimbus `Computation` just to hold the gas refund,
to fit around existing structures and get new EVMC working.  But the host can't
keep using `Computation`, so gas refund will be moved out of it in due course.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 18:29:40 +01:00
Jamie Lokier eb52fd3906
Transaction: Make `emitLog` use `Computation` to pass tests
When processing log operations on the EVMC host side, it causes incorrect
`rootHash` results in some tests.  This patch fixes the results.

The cause of these results is known: `Computation` is still doing parts of
contract scope entry/exit which need to be moved to the host.  For now, as a
temporary workaround, update logs in `Computation` as it did before.

This makes test pass when using Nimbus EVM.  (It breaks third-party EVMs when
`LOG*` ops are used, although most other tests pass.)

We can't keep this as it prevents complete host/EVM separation, but it's useful
in the current code, and it's fine to develop other functionality on top.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 18:29:40 +01:00
Jamie Lokier ce0c13c4ca
Transaction: Make `selfDestruct` use `Computation` to pass tests
When processing self destructs on the EVMC host side, it causes incorrect
`rootHash` results in some tests.  This patch fixes the results.

The cause of these results is known: `Computation` is still doing parts of
contract scope entry/exit which need to be moved to the host.  For now, as a
temporary workaround, update self destructs in `Computation` as it did before.

This makes test pass when using Nimbus EVM.  (It breaks third-party EVMs when
`SELFDESTRUCT` ops are used, although most other tests pass.)

We can't keep this as it prevents complete host/EVM separation, but it's useful
in the current code, and it's fine to develop other functionality on top.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 18:29:40 +01:00
Jamie Lokier b734240291
Transaction: Fix bounds error getting data address in empty seq
In the unusual case where log data is zero-length, `data[0].addr` is invalid
and Nim thoughtfully raises `IndexOutOfBounds`, a `Defect` so it's not even
in `CatchableError`.

This is done in the EVMC host services to handle `LOG*` ops, and it made one
of the EVM tests silently fail with no error message.  The fix is obvious.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 18:29:40 +01:00
Jamie Lokier ffd34a69fe
EVMC: Using `{.show.}` trace all calls from EVM to host services
When `show_tx_calls` is manually set to true, show all the calls from the EVM
to the host, including name, arguments and results.

For example this shows each call to `setStorage`, the key, value and storage
result.  This output allows the externally-visible activity of an EVM to be
seen, and it's been useful for guessing what went wrong when a test fails.

In theory, if two EVMs show the same activity in this log, they should have the
same effect on account states, gas, etc. and the same final `roothash`
(which is the only value some tests check).

ps. Ideally we'd use `{.push show.}`...`{.pop.}`, just like with `inline`.
But we can't: https://github.com/nim-lang/Nim/issues/12867

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 18:29:40 +01:00
Jamie Lokier 43b66a3a05
EVMC: `{.show.}` pragma to show EVMC host call arguments and results
New pragma `{.show.}` on a proc definition turns on tracing for that proc.
Every call to it shows the name, arguments and results, if `show_tx_calls` is
manually set to true.  This is to trace calls from EVM to host.

This started as a template which took a block expression, but the closure it
used led to illegal capture errors.  It was easier to write a macro.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 18:29:40 +01:00
Jamie Lokier 0b19f42158
EVMC: Binary compatibility glue on the host side
1. This provides the necessary type adjustments for host services to be
   (optionally) callable via the EVMC binary-compatible interface.  This layer
   is stashed away in a glue module so the host services continue to use
   appropriate Nim types, and are still callable directly.

   Inlining is used to ensure there should be no real overhead, including stack
   frame size for the `call` function.  Note, `import` must be used for
   `{.inline.}` to work effectively.

2. This also provides a key call in the other direction, the version of host to
   EVM `execute` that is called on the host side.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 18:29:39 +01:00
Jamie Lokier d49fa0bb86
Transaction: Add "host services", accessors to host state from EVM
This provides "host services", functions provided by the application to an EVM.
They are a key part of EVMC compatibility, but we will switch to using these
with "native" EVM as well.

These are functions like `getStorage`, `setStorage` and `emitLog` for accessing
the account state, because the EVM is not allowed direct access to the database.

This code is adapted from `nimbus/vm/evmc_host.nim` and other places, but there
is more emphasis on being host-side only, no dependency on the EVM or
`Computation` type.  It uses `TransactionHost` and types in `host_types`.

These host services have two goals: To be compatible with EVMC, and to be a
good way for the Nimbus EVM to access the data it needs.  In our new Nimbus
internal architecture, the EVM will only access the databases and other
application state via these host service functions.

The reason for containing the EVM like this, even "native" EVM, is that having
one good interface to the data makes it a lot easier to change how the database
works, which is on the roadmap.

These functions almost have EVMC signatures, but they are not binary compatible
with EVMC.  (Binary compatibility is provided by another module).  It would be
fine for Nimbus EVM to call these functions directly when linked directly.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 18:29:39 +01:00