Commit Graph

63 Commits

Author SHA1 Message Date
Jordan Hrycaj 221e6c9e2f
Unified database frontend integration (#1670)
* Nimbus folder environment update

details:
* Integrated `CoreDbRef` for the sources in the `nimbus` sub-folder.
* The `nimbus` program does not compile yet as it needs the updates
  in the parallel `stateless` sub-folder.

* Stateless environment update

details:
* Integrated `CoreDbRef` for the sources in the `stateless` sub-folder.
* The `nimbus` program compiles now.

* Premix environment update

details:
* Integrated `CoreDbRef` for the sources in the `premix` sub-folder.

* Fluffy environment update

details:
* Integrated `CoreDbRef` for the sources in the `fluffy` sub-folder.

* Tools environment update

details:
* Integrated `CoreDbRef` for the sources in the `tools` sub-folder.

* Nodocker environment update

details:
* Integrated `CoreDbRef` for the sources in the
  `hive_integration/nodocker` sub-folder.

* Tests environment update

details:
* Integrated `CoreDbRef` for the sources in the `tests` sub-folder.
* The unit tests compile and run cleanly now.

* Generalise `CoreDbRef` to any `select_backend` supported database

why:
  Generalisation was just missed due to overcoming some compiler oddity
  which was tied to rocksdb for testing.

* Suppress compiler warning for `newChainDB()`

why:
  Warning was added to this function which must be wrapped so that
  any `CatchableError` is re-raised as `Defect`.

* Split off persistent `CoreDbRef` constructor into separate file

why:
  This allows to compile a memory only database version without linking
  the backend library.

* Use memory `CoreDbRef` database by default

detail:
 Persistent DB constructor needs to import `db/core_db/persistent

why:
 Most tests use memory DB anyway. This avoids linking `-lrocksdb` or
 any other backend by default.

* fix `toLegacyBackend()` availability check

why:
  got garbled after memory/persistent split.

* Clarify raw access to MPT for snap sync handler

why:
  Logically, `kvt` is not the raw access for the hexary trie (although
  this holds for the legacy database)
2023-08-04 12:10:09 +01:00
andri lim 26a8759c34
implementation of EIP-4844: Shard Blob Transactions (#1440)
* EIP-4844: add pointEvaluation precompiled contract

* EIP-4844: validate transaction and block header

* EIP-4844: implement DataHash Op Code

* EIP-4844: txPool support excessDataGas calculation

* EIP-4844: make sure tx produce correct txHash

* EIP-4844: node should not automatically broadcast blob tx to it's peers

* EIP-4844: add test cases

* EIP-4844: add EIP-4844 support to t8n tool

* EIP-4844: update nim-eth to branch eip-4844

* fix t8n transaction decoding

* add t8n test data

* EIP-4844: fix blobHash opcode

* disable blobHash test when evmc_enable
2023-06-24 20:56:44 +07:00
jangko 34adb86167
fix macro assembler and reenable disabled tests
now macro assembler support merge fork, shanghai, etc without using ugly hack.
also each assembler test have their own `setup` section that can access
`vmState` and perform various custom setup.
2023-03-22 18:18:37 +07:00
jangko 2a3c67b4be
refactor touched accounts, selfdestruct, and log
simplify EVM and delegete those things to accounts cache.
also no more manual state clearing, accounts cache will be
responsible for both collecting touched account and perform
state clearing.
2023-03-21 08:14:06 +07:00
Adam Spitz 11771f8444
Recreating some of the old stateless code that had bitrotted. (#1493)
Note that we're not really using it yet; I'm just refactoring and
adding new code, working in small steps.
2023-03-10 17:16:42 -05:00
Adam Spitz c921135e2e
Refactored to introduce distinct types AccountsTrie and StorageTrie. (#1490)
This is in preparation for working on stateless mode; it's useful to
have a clearer and more type-safe interface for accessing accounts
versus slots.
2023-03-10 13:42:37 -05:00
Kim De Mey 8523d8b40e
Bump submodules and fix related issues (#1448)
* Updated to the latest nim-eth, nim-rocksdb, nim-web3

* Bump nimbus-eth2 module and fix related issues

Temporarily disabling Portal beacon light client network as it is
a lot of copy pasted code that did not yet take into account
forks. This will require a bigger rework and was not yet tested
in an actual network anyhow.

* More nimbus fixes after module bumps

---------

Co-authored-by: Adam Spitz <adamspitz@status.im>
Co-authored-by: jangko <jangko128@gmail.com>
2023-01-27 15:57:48 +01:00
Kim De Mey 9b1193c402
Revert "Updated to the latest nim-eth, nim-rocksdb, nim-web3. (#1441)" (#1443)
This reverts commit 1c1a3352b3.
2023-01-23 10:50:17 +01:00
Adam Spitz 1c1a3352b3
Updated to the latest nim-eth, nim-rocksdb, nim-web3. (#1441) 2023-01-21 15:04:22 +01:00
jangko 94a94c5b65 implement better hardfork management 2022-12-02 13:51:42 +07:00
Adam Spitz e040e2671a
Added basic async capabilities for vm2. (#1260)
* Added basic async capabilities for vm2.

This is a whole new Git branch, not the same one as last time
(https://github.com/status-im/nimbus-eth1/pull/1250) - there wasn't
much worth salvaging. Main differences:

I didn't do the "each opcode has to specify an async handler" junk
that I put in last time. Instead, in oph_memory.nim you can see
sloadOp calling asyncChainTo and passing in an async operation.
That async operation is then run by the execCallOrCreate (or
asyncExecCallOrCreate) code in interpreter_dispatch.nim.

In the test code, the (previously existing) macro called "assembler"
now allows you to add a section called "initialStorage", specifying
fake data to be used by the EVM computation run by that test. (In
the long run we'll obviously want to write tests that for-real use
the JSON-RPC API to asynchronously fetch data; for now, this was
just an expedient way to write a basic unit test that exercises the
async-EVM code pathway.)

There's also a new macro called "concurrentAssemblers" that allows
you to write a test that runs multiple assemblers concurrently (and
then waits for them all to finish). There's one example test using
this, in test_op_memory_lazy.nim, though you can't actually see it
doing so unless you uncomment some echo statements in
async_operations.nim (in which case you can see the two concurrently
running EVM computations each printing out what they're doing, and
you'll see that they interleave).

A question: is it possible to make EVMC work asynchronously? (For
now, this code compiles and "make test" passes even if ENABLE_EVMC
is turned on, but it doesn't actually work asynchronously, it just
falls back on doing the usual synchronous EVMC thing. See
FIXME-asyncAndEvmc.)

* Moved the AsyncOperationFactory to the BaseVMState object.

* Made the AsyncOperationFactory into a table of fn pointers.

Also ditched the plain-data Vm2AsyncOperation type; it wasn't
really serving much purpose. Instead, the pendingAsyncOperation
field directly contains the Future.

* Removed the hasStorage idea.

It's not the right solution to the "how do we know whether we
still need to fetch the storage value or not?" problem. I
haven't implemented the right solution yet, but at least
we're better off not putting in a wrong one.

* Added/modified/removed some comments.

(Based on feedback on the PR.)

* Removed the waitFor from execCallOrCreate.

There was some back-and-forth in the PR regarding whether nested
waitFor calls are acceptable:

https://github.com/status-im/nimbus-eth1/pull/1260#discussion_r998587449

The eventual decision was to just change the waitFor to a doAssert
(since we probably won't want this extra functionality when running
synchronously anyway) to make sure that the Future is already
finished.
2022-11-01 11:35:46 -04:00
jangko 667cb6d750
EIP 4399 changes: random -> prevRandao 2022-02-27 14:37:35 +07:00
jangko 5e5afaeacc
EIP-4399 'RANDOM' opcode tests 2022-02-05 16:17:41 +07:00
Jordan Hrycaj 261c0b51a7
Redesign of BaseVMState descriptor (#923)
* Redesign of BaseVMState descriptor

why:
  BaseVMState provides an environment for executing transactions. The
  current descriptor also provides data that cannot generally be known
  within the execution environment, e.g. the total gasUsed which is
  available not before after all transactions have finished.

  Also, the BaseVMState constructor has been replaced by a constructor
  that does not need pre-initialised input of the account database.

also:
  Previous constructor and some fields are provided with a deprecated
  annotation (producing a lot of noise.)

* Replace legacy directives in production sources

* Replace legacy directives in unit test sources

* fix CI (missing premix update)

* Remove legacy directives

* chase CI problem

* rebased

* Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation

why:
  Constructing a new 'AccountsCache' descriptor can be avoided sometimes
  when the current state root is properly positioned already. Such a
  feature existed already as the update function 'initStateDB()' for the
  'BaseChanDB' where the accounts cache was linked into this desctiptor.

  The function 'initStateDB()' was removed and re-implemented into the
  'BaseVmState' constructor without optimisation. The old version was of
  restricted use as a wrong accounts cache state would unconditionally
  throw an exception rather than conceptually ask for a remedy.

  The optimised 'BaseVmState' re-initialisation has been implemented for
  the 'persistBlocks()' function.

also:
  moved some test helpers to 'test/replay' folder

* Remove unused & undocumented fields from Chain descriptor

why:
  Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
jangko baf508f6ae
move stateDB from VMState to chainDB
previously, every time the VMState was created, it will also create
new stateDB, and this action will nullify the advantages of cached accounts.

the new changes will conserve the accounts cache if the executed blocks
are contiguous. if not the stateDB need to be reinited.

this changes also allow rpcCallEvm and rpcEstimateGas executed properly
using current stateDB instead of creating new one each time they are called.
2021-10-28 18:57:08 +07:00
jangko d93a8bc4a1
make macro_assembler to use the same testEvmCall
it also allow the macro_assembler to write more test
of CALL and CREATE family which previously not possible
2021-10-14 15:10:12 +07:00
Jamie Lokier 7c90d8de70
EVM: Remove `vm_forks` everywhere, use common forks list instead
The common forks list was already used, redirected via `vm_forks` for
historical compatibility.  Remove the old `vm_forks` now and divert all imports
to the common forks list outside the EVM.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 15:36:31 +01:00
Jamie Lokier aee0fe39d2
EVM: Remove `vm_types2` everywhere, use common forks list instead
File `vm_types2` is obsolete.  Remove this file and divert all imports to the
common forks list outside the EVM, or in some cases they don't need it anyway.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 15:36:31 +01:00
Jamie Lokier 84269ddccf
Forks: Use capitalized names again for presentation (logging etc.)
Fork names were originally capitalized, and were made lower case by @narimiran
in commit 36a7519 to satisfy `parseEnum` in some tests.  Restore the
capitalization and make the tests work with it.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-08 15:36:31 +01:00
Jamie Lokier 5e718bcbe2
EVM: Remove most unused imports of `vm_*` files
Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-06-01 16:54:38 +01:00
Jamie Lokier d2586c3a73
Assembler: Make macro_assembler tests use new function asmCallEvm
Move the EVM setup and call in `macro_assembler` (`runVM`) entirely to new
function `asmCallEvm` in `call_evm`.  Extra return values needed for
testing are returned specially from `asmCallEvm`.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-05-04 01:36:00 +01:00
Jamie Lokier 834449d943
Assembler: Second asmSetupComputation, calls the first
The second `asmSetupComputation looks up state by block number and preceding
block number, modifies the first transaction with code for testing, and uses
some parts of that transaction to setup an an EVM test.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-05-04 01:03:55 +01:00
Jamie Lokier 5728491d60
Assembler: First asmSetupComputation to support asmCallEvm
In the `macro_assembler` test code, `initComputation` is another variant of
`rpcSetupComputation` and `txSetupComputation` with slightly different
paremeters.  The similarity is obvious.

It is a special setup for testing, though, as it requires a contract-creation
transaction for parameters, but sets up a `CALL` execution not `CREATE`.

Gather this into `call_evm`: `initComputation` -> `asmSetupComputation`.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-05-04 01:03:55 +01:00
Jordan Hrycaj c8582583ef
merge vm_memory, vm_interpreter, and vm_utils_numeric => vm_internals
why:
  currently used for tests only
2021-03-31 18:15:27 +01:00
Jordan Hrycaj 00ba7a2718
merge vm_forks and vm_opcode_values => vm_type2
why:
  all types, but they cannot be merged int vm_types because of a circular
  dependency.
2021-03-31 17:53:15 +01:00
Jordan Hrycaj 99568c9b46
provide vm_opcode_values as import/export wrapper
details:
  moved original vm/interpreter/opcode_values.nim => vm/interpreter/nvm_opcode_values.nim
2021-03-31 16:49:03 +01:00
Jordan Hrycaj cf63b9b03f
provide vm_memory as import/export wrapper
details:
  moved original vm/memory.nim => vm/nvm_memory.nim
2021-03-31 16:48:44 +01:00
Jordan Hrycaj eee24de450
provide vm_message as import/export wrapper
details:
  moved original vm/message.nim => vm/nvm_message.nim
2021-03-31 16:47:02 +01:00
Jordan Hrycaj cf3a356d76
provide vm_computation as import/export wrapper
details:
  moved original vm/computation.nim => vm/nvm_computation.nim
2021-03-31 16:38:10 +01:00
Jordan Hrycaj 3a3e4d5707
provide vm_forks as import/export wrapper
details:
  moved original vm/interpreter/vm_forks.nim => vm/interpreter/nvm_forks.nim
2021-03-31 16:03:34 +01:00
jangko a263e6b1a6
implement EIP2315 tests 2020-11-25 18:23:02 +07:00
jangko d089a61539
add compiletime evm bytecode generator to macro_assembler 2020-07-24 19:54:27 +07:00
jangko 165f9fea2e
reduce warnings 2020-07-21 13:15:06 +07:00
jangko 8ee0ab4e89
fix macro_assembler -> add call to acounts_cache.persist 2020-06-01 13:58:14 +07:00
jangko 844071033a
fix at various places related to missing accounts_cache.persist call 2020-06-01 13:45:32 +07:00
Jacek Sieka 4ade5797ee
rlp: don't use ranges / experimental features (#495) 2020-04-20 20:12:44 +02:00
andri lim af02a3b1b2
reduce unused import warnings 2020-04-15 19:05:57 +07:00
andri lim 87bae2bb78
switch to new toFork 2020-04-12 18:02:59 +07:00
andri lim d656f7f701 rename 'txContext' to 'setupTxContext' 2020-01-23 18:07:44 +02:00
andri lim 933b2dad78 remove code from EVM Message and load code in 'Computation' 2020-01-23 18:07:44 +02:00
andri lim c1ef8632b2 move fork and gasCosts from Computation to vmState 2020-01-20 18:36:58 +02:00
andri lim bab359cabf move origin and gasPrice from Message to vmState 2020-01-20 18:36:58 +02:00
andri lim 0b99b76cd1 change 'BaseComputation' to 'Computation' 2020-01-20 18:36:58 +02:00
andri lim 79df931234 simplifies computation.getFork 2020-01-20 18:36:58 +02:00
andri lim 8f7597e23b [skip ci] add evmc CallKind to EVM Message 2020-01-19 19:17:37 +02:00
andri lim dfa2f0099f [skip ci] fixes macro_assembler 2020-01-19 19:17:37 +02:00
andri lim 836e087984 [skip ci] change storageAddress to contractAddres 2020-01-19 19:17:37 +02:00
andri lim c4dcf5da30
reduce compiler warnings 2019-11-13 21:49:39 +07:00
Ștefan Talpalaru d84e4810d5
unittest2 (parallelism disabled)
This is https://github.com/status-im/nimbus/pull/332 with parallelism
disabled, while some threadpool deadlock is being debugged.
2019-10-31 06:17:01 +01:00
Jacek Sieka 2763bd0dd5
std_shims -> stew 2019-07-07 12:12:01 +02:00