Commit Graph

1210 Commits

Author SHA1 Message Date
jangko c704626ea8
mark the new chain as canonical when calling db.setHead
the new chain marked as canonical is from the new head down to
forking point. most of the time, the forking point is the parent
itself.
2022-03-26 13:23:08 +07:00
jangko e628c8dc88
temporary fix db.setHead before actual fix 2022-03-25 14:01:32 +07:00
jangko 54724b87dc
fix safeBlockHash handling in engine_forkchoiceUpdatedV1 2022-03-25 14:01:26 +07:00
jangko 1008e87ec2
disable discovery if maxPeers set to zero 2022-03-25 14:01:18 +07:00
jangko a90bf1c99a
fix ttd handling in sealing engine 2022-03-25 14:00:44 +07:00
jangko effc874d47
fix header.extraData printer in debug.nim 2022-03-25 14:00:44 +07:00
Jordan Hrycaj 0d99e8003c
remove cruft (#1007)
why:
  Accidentally left over after trying several solutions
2022-03-24 10:28:28 +00:00
Jordan Hrycaj 7b9fe94d1a
Handle EIP-3675 (TheMerge) switch block (#1005)
why:
  If configured/activated, this block is advertised as FORK_NEXT in the
  status(0x01) message of the RPLx wire protocol.
2022-03-24 08:54:11 +00:00
jangko 727a3ee7b1
fix BlockHeader object of json-rpc
- add missing baseFeePerGas field
- add missing mixHash field
2022-03-22 08:35:07 +07:00
jangko 89fd986a00
fix ttd logic in sealing engine
also fix isBlockAfterTtd in chain_desc.nim
2022-03-22 08:35:06 +07:00
jangko cf0cc531c3
add block header debug/pretty print 2022-03-22 08:35:06 +07:00
Jordan Hrycaj 046c97f18b
Activate wire protocol eth/66 (#993)
* Activate wire protocol eth/66

and:
 Disentangle protocol_eth66.nim from import sections

why:
  Importing the protocol_eth66 module is not necessary. There is
  no need to know too many details of the underlying wire protocol. All
  that is needed will be exported by blockchain_sync.nim.

* fixes, and rebase

* Update nimbus/p2p/blockchain_sync.nim

Co-authored-by: Kim De Mey <kim.demey@gmail.com>

* Fixes and rebase

Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2022-03-21 17:12:07 +00:00
Kim De Mey b00ac490a9
Bump nim-eth and adjust API to support binding to specific address (#994)
- Fixes also a bug which would not start an EthereumNode properly
when no bootstrap nodes are provided
2022-03-16 09:01:35 +01:00
Jordan Hrycaj 534fb528a4
Update terminal total difficulty handling (#992)
why:
  Testing against a replay unit test for Devnet4 made it necessary to
  adjust the TTD handling. Without updated, importing fails at block #5646
  which is the parent of the terminal PoW block. Similar considerations
  apply for Devnet5 and Kiln.
2022-03-15 17:21:41 +00:00
jangko 400722f1fa
inserting header without set head from engine api should also validate the header 2022-03-11 15:13:59 +07:00
jangko 7dbc92f54c
make sure sealing engine produced block and ExecutionPayloadV1 have the same blockHash
- prevRandao/mixDigest = payloadAttrs.prevRandao
- timestamp = payloadAttrs.timestamp
- coinbase = payloadAttrs.suggestedFeeRecipient
- extraData = it can be anything 0-32 bytes, but currently set to empty bytes.
2022-03-11 13:30:46 +07:00
jangko e461248e86
remove cruft 2022-03-09 21:43:51 +07:00
jangko d41a3dedf0
write block ttd in `persistHeaderToDbWithoutSetHead` 2022-03-08 21:47:51 +07:00
jangko d74e9cbf49
add warning if engine-api enabled without enabling sealing engine 2022-03-08 14:51:27 +07:00
jangko 9230e6a024
fixes engine api help message in config.nim
somehow a const string still need `$` to display the
content instead of the identifier itself.
2022-03-08 13:29:34 +07:00
jangko d1dd5d5cad
ttd from command-line takes precedence over ttd from config-file 2022-03-08 11:27:14 +07:00
jangko f782327fcf
reimplement engine API rpc kiln spec v2 2022-03-03 11:28:18 +07:00
jangko 2970fc4b02
refactor chain_config and add decodeNetworkParams
allow custom network not only loaded from file but also
decoded from json string
2022-03-03 08:58:55 +07:00
jangko 73e28694b5
move ttd from vm/state to chain_db 2022-03-03 08:58:55 +07:00
jangko 9a1c8fc779
refactor genesis.nim 2022-03-03 08:28:00 +07:00
jangko 667cb6d750
EIP 4399 changes: random -> prevRandao 2022-02-27 14:37:35 +07:00
jangko 6374c9f66d
fixes regression caused by #962 2022-02-14 21:22:39 +07:00
Jordan Hrycaj 215e9856d3
Jordan/custom network (#962)
* Rearrange/rename test_kintsugu => test_custom_network

why:
  Debug, fix and test more general problems related to running
  nimbus on a custom network.

* Update UInt265/Json parser for --custom-network command line option

why:
  As found out with the Kintsugi configuration, block number and balance
  have the same Nim type which led to misunderstandings. This patch makes
  sure that UInt265 encoded string values "0x11" decodes to 17, and "b"
  and "11" to 11.

* Refactored genesis.toBlock() => genesis.toBlockHeader()

why:
  The function toBlock(g,db) may return different results depending on
  whether the db descriptor argument is nil, or initialised. This is due
  to the db.config data sub-descriptor which may give various outcomes
  for the baseFee field of the genesis header.

  Also, the version where db is non-nil initialised is used internally
  only. So the public rewrite toBlockHeader() that replaces the toBlock()
  function expects a full set of NetworkParams.

* update comments

* Rename toBlockHeader() => toGenesisHeader()

why:
  Polymorphic prototype used for BaseChainDB or NetworkParams argument.
  With a BaseChainDB descriptor argument, the name shall imply that the
  header is generated from the config fields rather than fetched from
  the database.

* Added command line option --static-peers-file

why:
  Handy feature to keep peer nodes in a file, similar to the
  --bootstrap-file option.
2022-02-11 16:28:39 +00:00
jangko b4283aeb1f
fix nonce overflow related to CREATE/CREATE2
if the caller nonce == u64.high, the contract creation cannot
go on.
2022-02-10 15:48:38 +07:00
jangko a6f1ed30ee
EIP-3607: Reject transactions from senders with deployed code
Any transaction where tx.sender has a CODEHASH != EMPTYCODEHASH MUST
be rejected as invalid, where EMPTYCODEHASH =
0xc5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470.
The invalid transaction MUST be rejected by the client and not be included
in a block. A block containing such a transaction MUST be considered invalid.
2022-02-10 15:48:18 +07:00
jangko 4846405c0e
EIP-3675: fix mixDigest validation
because EIP-4399 override mixDigest validation rule,
there is no need to check mixDigest == ZERO_HASH
`mixDigest` will carry POS block randomness value
2022-02-08 20:23:41 +07:00
jangko ec59c691aa
EIP-3675: disable reward for coinbase and uncles miner 2022-02-08 20:23:41 +07:00
jangko 28cdfcaf6b
fix EIP-4399 'random' opcode
- fix previous implementation of EIP-4399
- now `random` opcode can be used with evmc_enabled
2022-02-08 20:23:40 +07:00
jangko d3fbe1d94d
fixes related to EIP-4399/EIP-3675
- fix `RANDOM` opcode export
- fix `difficulty` return value in VMState
2022-02-05 16:15:50 +07:00
jangko d7f1d698ce EIP-4399 implementation of nim-vm2
new addition:
  - `RANDOM` opcode
  - `random` field of BlockHeader(previously `mixDigest`)
  - `PostMerge` temporary name of this new EVM version
2022-02-01 18:11:14 +02:00
jangko 71aa7e4b5c EIP-4399 implementation of nim-vm
what's new:
- `RANDOM` OPCODE
- `random` field of BlockHeader(previously `mixDigest`)
- `PostMerge` temporary name
2022-02-01 18:11:14 +02:00
Jordan Hrycaj 77c9b8c2f0
Kludge needed for setting up custom network (#944)
* Kludge needed for setting up custom network

why:
  Some non-features in the persistent hexary trie DB produce an assert
  error when initiating the Kinsugi network.

details:
  This fix should be temporary, only.

* Fix OS detection

why:
  directive detectOs() bails out on Windows if checking for Ubuntu
2022-02-01 12:04:20 +00:00
Zahary Karadjov 3cbb920406
Migrate to Engine API spec version v1.0.0-alpha.5; More progress towards working M1 2022-01-25 12:52:50 +02:00
jangko 01980041d5
setup nimbus tx pool for app wide usage
currently the txpool is used at:
- sealer engine
- graphql ethapi
- json rpc ethapi
- ws rpc ethapi
2022-01-25 11:38:49 +07:00
Zahary Karadjov 137eb97766 Initial implementation of the merge spec
Includes a simple test harness for the merge interop M1 milestone

This aims to enable connecting nimbus-eth2 to nimbus-eth1 within
the testing protocol described here:

https://github.com/status-im/nimbus-eth2/blob/amphora-merge-interop/docs/interop_merge.md

To execute the work-in-progress test, please run:

In terminal 1:
tests/amphora/launch-nimbus.sh

In terminal 2:
tests/amphora/check-merge-test-vectors.sh
2022-01-24 09:44:39 +02:00
Jordan Hrycaj 103656dbb5 TxPool implementation
details:
  For documentation, see comments in the file tx_pool.nim.

  For prettified manual pages run 'make docs' in the nimbus directory and
  point your web browser to the newly created 'docs' directory.
2022-01-22 08:26:57 +02:00
Jordan Hrycaj 3f0139c5b6 Kludge for 'make docs' to produce pretty nim manuals
why:
  Some helper file will not be generated by the nim document gereator,
  so they have been stashed from a later nim version to be provided when
  missing.

  This problem was known with an earlier nim version (see here
  https://github.com/nim-lang/Nim/issues/8952) but was reported solved.
  Maybe we need a second look into that.
2022-01-22 08:26:57 +02:00
Jordan Hrycaj 1eb79c34c6 Fixing Win64/CI unit test segfault
why:
  Previously, the function 'snapshot_desc.loadSnapshot()' contained the
  equivalent of 'eth.decode(@[],SnapshotData)' for some type 'SnapshotData'
  which should result in an exception of type 'RlpTypeMismatch'.

  Before mid October, this worked for all systems on the Github CI. Since
  then, a segfault message in the Github CI can be reproduced on all 64bit
  Windows wuns when running 'build/all_tests <id-of-test_txpool>' after the
  failed 'make test' directive (the latter one needs to be extended by
  '|| true'.)  This error cannot be reproduced on my local Win7/64 system
  with the same MSYS2 and gcc 11.2.0 compiler.

  The fix is, rather than catching an exception, to explicitly check the
  first argument of 'eth.decode(@[],SnapshotData)' and act if it is empty.

also:
  removed some obsolete {.inline.} annotations.
2022-01-22 08:26:57 +02:00
Jordan Hrycaj 9545767c53 update ec_recover
why:
  Previous version was based on lru_cache which is ugly. This module is
  based on the stew/keyed_queue library module.

other:
  There are still some other modules rely on lru_cache which should be
  removed.
2022-01-22 08:26:57 +02:00
Jordan Hrycaj 261c0b51a7
Redesign of BaseVMState descriptor (#923)
* Redesign of BaseVMState descriptor

why:
  BaseVMState provides an environment for executing transactions. The
  current descriptor also provides data that cannot generally be known
  within the execution environment, e.g. the total gasUsed which is
  available not before after all transactions have finished.

  Also, the BaseVMState constructor has been replaced by a constructor
  that does not need pre-initialised input of the account database.

also:
  Previous constructor and some fields are provided with a deprecated
  annotation (producing a lot of noise.)

* Replace legacy directives in production sources

* Replace legacy directives in unit test sources

* fix CI (missing premix update)

* Remove legacy directives

* chase CI problem

* rebased

* Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation

why:
  Constructing a new 'AccountsCache' descriptor can be avoided sometimes
  when the current state root is properly positioned already. Such a
  feature existed already as the update function 'initStateDB()' for the
  'BaseChanDB' where the accounts cache was linked into this desctiptor.

  The function 'initStateDB()' was removed and re-implemented into the
  'BaseVmState' constructor without optimisation. The old version was of
  restricted use as a wrong accounts cache state would unconditionally
  throw an exception rather than conceptually ask for a remedy.

  The optimised 'BaseVmState' re-initialisation has been implemented for
  the 'persistBlocks()' function.

also:
  moved some test helpers to 'test/replay' folder

* Remove unused & undocumented fields from Chain descriptor

why:
  Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
Jordan Hrycaj cea2a83b0a
Update/clarify tx validation (#917)
details:
  1. The check for cumulativeGasUsed + tx.gasLimit <= header.gasLimit
     makes neither sense nor is it part of the Eip1559 specs. Nevertheless
     a check tx.gasLimit <= header.gasLimit is added to satisfy some
     unit test (see comments in validateTransaction() body.)
  2. As a replacement check for the one removed in 1, a check for
     cumulativeGasUsed + gasBurned <= header.gasLimit has been added
     (see comments in processTransactionImpl() body.)
  3. Prototypes for processTransaction() variants have been cleaned up and
     commented.

why:
  Detail 1. in particular produces an error for tightly packed blocks when
  the last tx in the list has a generous gasLimit.
2022-01-10 09:04:06 +00:00
Jamie Lokier 6a7803a9e4
EVMC: Improve `--evm`, remove it in non-EVMC builds, change imports
- Remove the `--evm` option on non-EVMC builds.
  `when` around an option doesn't work with confutils; it fails to compile.
  Workaround that by setting the `ignore` pragma on EVMC-specific options.
  (Thanks @jangko for that new pragma).  I prefer this to a solution which
  moves the whole option's pragma elsewhere, especially if we add more options.

- Improve the help text, so that it shows the standard library extension on
  each target platform (or none if on another platform).

- Undo b3f21bf4 "add missing evmc_enabled conditional compilation in
  evmc_dynamic_loader".  Move the conditional to `nimbus.nim`, and take more
  care there to only use the loader function in EVMC builds.

  It's ok to just not include this EVMC-only module (like some other EVMC
  modules), rather than making the module itself a bit broken: Without this
  change, it references a function that's not imported or linked to, and it
  only links because there is no call sequence reaching that function.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-12-14 15:51:09 +00:00
jangko d5082df5d8
fix config using new features from nim-confutils
- `network` and `discovery` got additional longDesc,
  the help text now become more descriptive.
- `networkId` and `networkParams` now is ignored by confutils
  and become ordinary fields of NimbusConf.
2021-12-12 16:45:58 +07:00
Jamie Lokier 4b89ca3215
EVM: `writeContract` fixes, never return contract code as `RETURNDATA`
This fixes #867 "EIP-170 related consensus error at Goerli block 5080941", and
equivalent on other networks.

This combines a change on the EVM-caller side with an EVM-side change from
@jangko 6548ff98 "fixes CREATE/CREATE2's `returndata` bug", making the caller
EVM ignore any data except from `REVERT`.

Either change works by itself.  The reason for both is to ensure we definitely
comply with ambiguous EVMC expectations from either side of that boundary, and
it makes the internal API clearer.

As well as fixing a specific consensus issue, there are some other EVM logic
changes too: Refactored `writeContract`, how `RETURNDATA` is handled inside the
EVM, and changed behaviour with quirks before EIP-2 (Homestead).

The fix allows sync to pass block 5080941 on Goerli, and probably equivalent on
other networks.  Here's a trace at batch 5080897..5081088:

```
TRC 2021-10-01 21:18:12.883+01:00 Persisting blocks                  file=persist_blocks.nim:43 fromBlock=5080897 toBlock=5081088
...
DBG 2021-10-01 21:18:13.270+01:00 Contract code size exceeds EIP170  topics="vm computation" file=computation.nim:236 limit=24577 actual=31411
DBG 2021-10-01 21:18:13.271+01:00 gasUsed neq cumulativeGasUsed      file=process_block.nim:68 block=5080941/0A3537BC5BDFC637349E1C77D9648F2F65E2BF973ABF7956618F854B769DF626 gasUsed=3129669 cumulativeGasUsed=3132615
TRC 2021-10-01 21:18:13.271+01:00 peer disconnected                  file=blockchain_sync.nim:407 peer=<IP:PORT>
```

Although it says "Contract code size" and "gasUsed", this bug is more general
than either contract size or gas.  It's due to incorrect behaviour of EVM
instructions `RETURNDATA` and `RETURNDATASIZE`.

Sometimes when `writeContract` decides to reject writing the contract for any
of several reasons (for example just insufficient gas), the unwritten contract
code was being used as the "return data", and given to the caller.  If the
caller used `RETURNDATA` or `RETURNDATASIZE` ops, those incorrectly reported
the contract code that didn't get written.

EIP-211 (https://eips.ethereum.org/EIPS/eip-211) describes `RETURNDATA`:
> "`CREATE` and `CREATE2` are considered to return the empty buffer in the
> success case and the failure data in the failure case".

The language is ambiguous.  In fact "failure case" means when the contract uses
`REVERT` to finish.  It doesn't mean other failures like out of gas, EIP-170
limit, EIP-3541, etc.

To be thorough, and to ensure we always do the right thing with real EVMC when
that's finalised, this patch fixes the `RETURNDATA` issue in two places, either
of which make Goerli block 5080941 pass.

`writeContract` has been refactored to be caller, and so has where it's called.
It sets an error in the usual way if contract writing is rejected -- that's
anticipating EVMC, where we'll use different error codes later.

Overall four behaviour changes:

1. On the callee side, it doesn't set `c.outputData` except for `REVERT`.
2. On the caller side, it doesn't read `child.outputData` except for `REVERT`.
3. There was a bug in processing before Homestead fork (EIP-2).  We did not
   match the spec or other implementations; now we do.  When there's
   insufficient gas, before Homestead it's treated as success but with an empty
   contract.

   d117c8f3fd/ethereum/processblock.py (L304)
   https://github.com/ethereum/go-ethereum/blob/401354976bb4/core/vm/instructions.go#L586

4. The Byzantium check has been removed, as it's unnecessary.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-12-12 16:34:13 +07:00
jangko b3f21bf40c
add missing evmc_enabled conditional compilation in evmc_dynamic_loader 2021-12-12 16:32:13 +07:00
Jamie Lokier cf49b8d4fc
EVMC: Option `--evm`, load third-party EVM as a shared library
This patch adds:

- Load and use a third-party EVM in a shared library, instead of Nimbus EVM.
- New option `--evm` to specify which library to load.
- The library and this loader conforms to the [EVMC]
  (https://evmc.ethereum.org/) 9.x specification.

Any third-party EVM which is compatible with EVMC version 9.x and supports EVM1
contract code will be accepted.  The operating system's shared library format
applies.  These are `.so*` files on Linux, `.dll` files on Windows and `.dylib`
files on Mac.

The alternative EVM can be selected in two ways:

- Nimbus command line option `--evm:<path>`.
- Environment variable `NIMBUS_EVM=<path>`.

The reason for an environment variable is this allows all the test programs to
run with a third-party EVM as well.  Some don't parse command line options.

There are some limitations to be aware of:

- The third-party EVM must use EVMC version 9.x, no other major version.
  EVMC 9.x supports EIP-1559 / London fork and older transactions.

- Nested `*CALL` and `CREATE*` operations don't use the third-party EVM yet.
  These call the built-in Nimbus EVM.  This mixing of different EVMs between
  levels is explicitly allowed in specs, so there is no problem doing it.

- The third-party EVM doesn't need to support precompiles, because those are
  nested calls, which use the built-in Nimbus EVM.

- Third-party EVMs execute contracts correctly, but fail the final `rootHash`
  match.  The reason is that some account state changes, which are correct, are
  currently inside the Nimbus EVM and need to be moved to EVMC host logic.
  *This is a known work in progress*.  The EVM execution itself is fine.

Test results using "evmone" third-party EVM:

- [evmone](https://github.com/ethereum/evmone) has been tested.  Only on
  Linux but it "should" work on Windows and Mac equally well.

- [Version 0.8.1](https://github.com/ethereum/evmone/releases/tag/v0.8.1) was
  used because it is compatible with EVMC 9.x, which is required for the
  EIP-1559 / London fork, which Nimbus supports.  Version 0.8.0 could be used
  but it looks like an important bug was fixed in 0.8.1.

- evmone runs fine and the trace output looks good.  The calls and arguments
  are the same as the built-in Nimbus EVM for tests that have been checked
  manually, except evmone skips some calls that can be safely skipped.

- The final `rootHash` is incorrect, due to the *work in progress* mentioned
  above which is not part of the evmone execution.  Due to this, it's possible
  to try evmone and verify expected behaviours, which also validates our own
  EVMC implementation, but it can't be used as a full substitute yet.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-12-10 16:32:18 +00:00
Jamie Lokier 072934272b
Transaction: Map `evmc_result` back to `Computation` result
This missing part of EVMC processing allows third-party EVMs to work.

It fixes EVMC result processing (at the top-level of calls, not nested calls)
to use the EVMC result object, instead of reading so much internal state of the
Nimbus `Computation` object.

It has been tested by calling [`evmone`](https://github.com/ethereum/evmone)
and getting useful results with tracing enabled (`showTxCalls = true`).  It's
even able to run parts of the fixtures test suite.

There are other issues with account balances, etc that need to be worked on to
get the correct _final_ results, but the EVM execution is correct with this.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-12-10 16:27:53 +00:00
Jamie Lokier 6ef9bfd21b
EVMC: Byte-endian conversions for 256-bit numeric values
Perform byte-endian conversion for 256-bit numeric values, but not 256-bit
hashes.  These conversions are necessary for EVMC binary compatibility.

In new EVMC, all host-side conversions are explicit, calling `flip256`.

These conversions are performed in the EVMC "glue" code, which deals with the
binary interface, so the host services aren't aware of conversions.

We intend to skip these conversions when Nimbus host calls Nimbus EVM, even
when it's a shared library, using a negotiated EVMC extension.  But for now
we're focused on correctness and cross-validation with third party EVMs.

The overhead of endian conversion is not too high because most EVMC host calls
access the database anyway.  `getTxContext` does not, so the conversions from
that are cached here.  Also, well-optimised EVMs don't call it often.

It is arguable whether endian conversion should occur for storage slots (`key`).

In favour of no conversion: Slot keys are 32-byte blobs, and this is clear in
the EVMC definition where slot keys are `evmc_bytes32` (not `evmc_uint256be`),
meaning treating as a number is _not_ expected by EVMC.  Although they are
often small numbers, sometimes they are a hash from the contract code plus a
number.  Slot keys are hashed on the host side with Keccak256 before any
database calls, so the host side does not look at them numerically.

In favour of conversion: They are often small numbers and it is helpful to log
them as such, rather than a long string of zero digits with 1-2 non-zero.  The
representation in JSON has leading zeros removed, like a number rather than a
32-byte blob.  There is also an interesting space optimisation when the keys
are used unhashed in storage.

Nimbus currently treats slot keys on the host side as numbers, and the tests
pass when endian conversion is done.  So to remain consistent with other parts
of Nimbus we convert slot keys.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-12-10 16:23:27 +00:00
Jamie Lokier 74253c88e3
Arrow Glacier fork
Add the new [Arrow Glacier fork](https://eips.ethereum.org/EIPS/eip-4345).

Only the difficulty calculation is changed, but as a new fork it still affects
a number of places in the code.

To the best of my knowledge the change is only scheduled on Mainnet.

In addition:

- The fork date comments in `chain_config.nim` have been checked against the
  real networks, set consistently in UTC instead of random timezones, and made
  neater.  Maybe we'll keep these when transferring config to a file someday.

- It's added to forkid hash tests (EIP-2124/EIP-2364), of course.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-12-10 13:40:51 +00:00
Jordan Hrycaj 55f7a4425f
Jordan/pow cache management (#888)
* PoW wrapper for verification & mining

why:
  It eases data management of per-Epoch lookup tables. Also some unit
  tests show limits of usefulness on non-specialised machines for
  mining besides developing tests.

details:
  For PoW verification, this patch provides a pretty wrapper hiding the
  details of the ethash/Hashimoto lookup cache management.

  For mining on my development system without special hardware, the
  underlying ethash functions are prohibitively slow. It takes
   * ~20 minutes to prepare the full ethash/Hashimoto lookup dataset
   * a second to run ~25k nonce tests (in the mining loop)

  The mining part might be of some use for generating test data for
  the tx-pool, though.

* Using PowRef as replacement for EpochHashCache + hashimotoLight()

* Fix typo (CI failed)

why:
  was below log level when testing locally

* fix canonical naming
2021-12-10 08:49:57 +00:00
jangko f051c2530e
fixes related to nim-json-rpc bump 2021-11-30 14:13:20 +07:00
Kim De Mey 02afda1b45
Run everything fluffy with chronosStrictException (#889) 2021-11-18 17:52:44 +01:00
jangko 7b67914453
fixes regression caused by recent changes in eth state handling
detected when running hive consensus simulator.
when processing an invalid block header and then
a new valid block header with the same block number,
the state root of the stateDB object should be updated
or reverted to parent stateRoot.

using intermediate stateRoot will trigger the hexary trie assertion.
2021-11-09 17:55:06 +07:00
jangko e710aa9423
fixes missing emptyRlp in genesis.nim 2021-11-02 17:25:22 +07:00
jangko 960539df81
unify evm call for both json-rpc and graphql
also fixes rpcEstimateGas for both of json-rpc and graphql
2021-10-31 12:24:40 +07:00
jangko baf508f6ae
move stateDB from VMState to chainDB
previously, every time the VMState was created, it will also create
new stateDB, and this action will nullify the advantages of cached accounts.

the new changes will conserve the accounts cache if the executed blocks
are contiguous. if not the stateDB need to be reinited.

this changes also allow rpcCallEvm and rpcEstimateGas executed properly
using current stateDB instead of creating new one each time they are called.
2021-10-28 18:57:08 +07:00
jangko cec628e620
cleanup: remove unused accessLogs code from vm_state
they are not used anywhere at present, nor in the future
2021-10-28 11:30:18 +07:00
jangko f0a0c1b878
cleanup: replace unnecessary initHexaryTrie with emptyRlpHash
it's simple non-sense
2021-10-28 10:48:28 +07:00
jangko e1abf81cac
cleanup: remove nimbus/rpc/key_storage.nim
this is an unused file
2021-10-28 10:32:41 +07:00
jangko df85cd779b
cleanup: remove setupComputation from vm2
setupComputation already replaced by runComputation
in transaction/call_common.nim
2021-10-28 10:30:56 +07:00
Jamie Lokier 5a5edb392a Bugfix: Incorrect processing of self-destructed, new contract
Fixes #868 "Gas usage consensus error at Mainnet block 6001128", and equivalent
on other networks.  Mainnet sync is able to continue past 6001128 after this.

Here's a trace:

```
TRC 2021-09-29 15:13:21.532+01:00 Persisting blocks                  file=persist_blocks.nim:43 fromBlock=6000961 toBlock=6001152
...
DBG 2021-09-29 15:14:35.925+01:00 gasUsed neq cumulativeGasUsed      file=process_block.nim:68 gasUsed=7999726 cumulativeGasUsed=7989726
TRC 2021-09-29 15:14:35.925+01:00 peer disconnected                  file=blockchain_sync.nim:407 peer=<PEER:IP>
```

Similar output is seen at many blocks in the range 6001128..6001204.

The bug is when handling a combination of `CREATE` or `CREATE2`, along with
`SELFDESTRUCT` applied to the new contract address.

Init code for a contract can't return non-empty code and do `SELFDESTRUCT` at
the same time, because `SELFDESTRUCT` returns empty data.

But it is possible to return non-empty code in a newly created, self-destructed
account if the init code calls `DELEGATECALL` or `CALLCODE` to other code which
uses `SELFDESTRUCT`.

In this case we must still charge gas and write the code.  This shows on
Mainnet blocks 6001128..6001204, where the gas difference matters.  The code
must be written because the new code can be called later in the transaction
too, before self-destruction wipes the account at the end.

There are actually three semantic changes here for a self-destructed, new
contract:

- Gas is charged.
- The code is written to the account.
- It can fail due to insufficient gas.

This patch almost exactly reverts a15805e4 "fix applyCreateMessage" from
2019-02-28.  I wonder what that fixed.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-10-19 14:24:46 +01:00
Jamie Lokier 242dfdd5ac
Bugfix: Off by 1 in EIP-170 code size checks in `stateless`
Fixes an off by 1 error where `EIP170_CODE_SIZE_LIMIT` was being treated as the
lowest invalid value by EVM code, but the highest valid value by witness code.

To remove confusion, this is renamed to `EIP170_MAX_CODE_SIZE` with value
0x6000, which matches the name (`MAX_CODE_SIZE`) and value used for this limit
in [EIP-170](https://eips.ethereum.org/EIPS/eip-170).

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-10-19 10:30:53 +01:00
Jamie Lokier 40fbed49cf
Sync fix: `GetBlockBodies` logic preventing sync, dropping peers
Fixes #864 "Sync progress stops at Goerli block 4494913", and equivalent on
other networks.

The block body fetcher in `blockchain_sync.nim` had an incorrect assumption
about how peers respond to `GetBlockBodies`.  It was issuing requests for N
block bodies and incorrectly handling replies which contained fewer than N
bodies.

Having received up to 192 headers in a batch, it split the range into smaller
`GetBlockBodies` requests, fetched each reply, then combined replies.  The
effect was Nimbus requested batches of 128+64 block bodies, received gaps in
the reply sequence, then aborted.

That meant it repeatedly fetched data, then discarded it, and fetched it again,
dropping good peers in the process.

Aborted and restarted batches occurred with earlier blocks too, but this became
more pronounced until there were no suitable peers at batch 4494913..4495104.

Here's a trace:

```
TRC 2021-09-29 02:40:24.977+01:00 Requesting block headers                   file=blockchain_sync.nim:224 start=4494913 count=192 peer=<ENODE>
TRC 2021-09-29 02:40:24.977+01:00 >> Sending eth.GetBlockHeaders (0x03)      file=protocol_eth65.nim:51 peer=<PEER> startBlock=4494913 max=192
TRC 2021-09-29 02:40:25.005+01:00 << Got reply eth.BlockHeaders (0x04)       file=protocol_eth65.nim:51 peer=<PEER> count=192
TRC 2021-09-29 02:40:25.007+01:00 >> Sending eth.GetBlockBodies (0x05)       file=protocol_eth65.nim:51 peer=<PEER> count=128
TRC 2021-09-29 02:40:25.209+01:00 << Got reply eth.BlockBodies (0x06)        file=protocol_eth65.nim:51 peer=<PEER> count=13
TRC 2021-09-29 02:40:25.210+01:00 >> Sending eth.GetBlockBodies (0x05)       file=protocol_eth65.nim:51 peer=<PEER> count=64
TRC 2021-09-29 02:40:25.290+01:00 << Got reply eth.BlockBodies (0x06)        file=protocol_eth65.nim:51 peer=<PEER> count=64
WRN 2021-09-29 02:40:25.306+01:00 Bodies len != headers.len                  file=blockchain_sync.nim:276 bodies=77 headers=192
TRC 2021-09-29 02:40:25.306+01:00 peer disconnected                          file=blockchain_sync.nim:403 peer=<PEER>
TRC 2021-09-29 02:40:25.306+01:00 Finished obtaining blocks                  file=blockchain_sync.nim:303 peer=<PEER>
```

In practice, for modern peers, Nimbus received shorter replies than it assumed
depending on the block sizes on the chain.  Geth/Erigon has 2MiB `BlockBodies`
response size soft limit.  OpenEthereum has 4MiB.

Up to Berlin (EIP-2929), Nimbus's fetcher failed often, but there were still
some peers serving what Nimbus needed.

Just after the start of Berlin, at batch 4494913..4495104 on Goerli, zero peers
responded with full size replies for the whole batch, so Nimbus couldn't
progress past that point.  But there was already a problem happening before
that for large blocks, dropping good peers and repeatedly fetching the same
block data.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-10-19 10:20:26 +01:00
jangko d93a8bc4a1
make macro_assembler to use the same testEvmCall
it also allow the macro_assembler to write more test
of CALL and CREATE family which previously not possible
2021-10-14 15:10:12 +07:00
jangko 71273f2f4c
remove error field from evm CallResult 2021-10-14 15:10:12 +07:00
jangko 08f8652790
remove noTransfer field from evm CallParams 2021-10-14 15:10:11 +07:00
jangko eb2251ec37
simplify evm call of test_precompiles
first step towards evm call variation reduction
2021-10-14 15:10:11 +07:00
jangko 5c4c1784a0
fix missing EIP-799 extra range check in header validation 2021-10-12 11:06:39 +07:00
jangko 460432f154
fixes EIP1559 tx gasCost validation
pre EIP1559 max(gasCost) is tx.gasLimit * tx.gasPrice
the new EIP1559 max(gasCost) before the transaction can be executed is
tx.gasLimit * tx.maxFeePerGas
2021-09-29 10:55:32 +07:00
jangko 44394d9ffd
fixes nimbus evm tracer, add missing networkParams when constructing chainDB 2021-09-26 10:45:52 +07:00
jangko 908dc21478
evm: fixes EIP2929 opcodes
op balanceEIP2929, extCodeHashEIP2929, extCodeSizeEIP2929, and
extCodeCopyEIP2929 are fixed due to their wrong gasConsume
position
2021-09-22 11:58:06 +07:00
jangko f3d194c05e
grpahql: add EIP-2718 and EIP-1559 features to graphql API
EIP-2718:
- chainID: Long! of Query
- chainID: Long of Transaction

EIP-1559:
- baseFeePerGas: BigInt of Block
- effectiveGasPrice: BigInt of Transaction
- maxFeePerGas: BigInt of Transaction
- maxPriorityFeePerGas: BigInt of Transaction
2021-09-21 13:35:49 +07:00
jangko a3badea928
config: fix new config based on input from jamie and zahary 2021-09-18 17:34:51 +07:00
jangko 69f2a0f95a
config: replace stdlib parseOpt with nim-confutils
fixes #581
2021-09-18 17:34:46 +07:00
jangko 48d497580a
config: remove last instance of getConfiguration usage from nimbus code
this is a preparation for migration to confutils based config
although there is still some getConfiguration usage in tests code
it will be removed after new config arrived
2021-09-08 21:25:14 +07:00
jangko c9cfebfa97
config: rearrange getConfiguration usage
avoid using getConfiguration inside object construction and
replace it with passing suitable param
2021-09-08 08:07:10 +07:00
jangko 9108301eef
config: remove global rng from NimbusConfiguration
move the rng to EthContext
2021-09-07 22:02:29 +07:00
jangko 34972c6cea
config: remove accounts management from NimbusConfiguration
a new AccountsManager and EthContext is created for managing
keystore and accounts

this is a preparation for new config using ConfUtils
2021-09-07 22:02:29 +07:00
jangko 14d2edcb26
chain config preset: add london block number
MainNet     12_965_000
RopstenNet  10_499_401
RinkebyNet   8_897_988
GoerliNet    5_062_605
2021-09-02 12:24:04 +07:00
jangko 4be35712fc
sealing engine: remove redundant clique epoch and period check
both clique epoch and clique period already checked in
newClique and will use default configuration they are not set.

this redundant check in sealing engine also failed with
some configuration where only one of them is set and the
other one not set.
2021-08-30 20:24:55 +07:00
jangko 521f29c0a0
remove unused calcGasLimit code
we have new calcGasLimit tested in sealing engine.
so we can safely remove the old unused calcGasLimit
2021-08-24 18:30:52 +07:00
jangko a0ee842367
fixes comments in clique.seal func 2021-08-24 16:14:17 +07:00
jangko 7dbc44f88c
implement simple PoA sealing engine
the goal of this module is to pass hive/smoke/clique test
and also support for hive/ethereum/rpc test

fixes #801
2021-08-24 14:49:13 +07:00
jangko c99153df22
fixes clique signerFn return type
and also add test related to this signerFn
2021-08-19 19:00:30 +07:00
jangko 18b26a0089
implement calcEIP1559GasLimit
CalcGasLimit1559 calculates the next block gas limit under 1559 rules.
this function is needed in upcoming sealing engine implementation
2021-08-18 20:23:38 +07:00
bmoo b09ad5cacb
code cleanup removed unused imports 2021-08-18 10:35:36 +07:00
Jamie Lokier 8fcd8354b1
EVMC: Use the same host interface for nested calls as top-level
Prior to this patch, top-level EVM executions and nested EVM executions did
their `getStorage` and other requests using a completely different set of host
functions.  It was just unfinished, to get top-level "new" EVMC working.

This finishes the job - it stops using the old methods.  Effect:

- Functionality added at the EVMC host level will be used by all EVM calls.
  (The target here is Beam Sync).

- The old set of functions are no longer used, so they can be removed.

- When EVMC host call tracing is enabled (`showTxCalls = true`), it traces
  the calls from nested EVM executions as well as top-level.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-08-17 17:18:26 +01:00
Jamie Lokier b783756ff3
EVMC: Make `hostInterface` a statically initialised global
We've been filling a "vtable"-like at run time, but it's not necessary.

The new object is a global `let x = evmc_host_interface(...)`, we assume it's
initialised before the first use, and we take its address with `.unsafeAddr`.

(If we use `ref evmc_host_interface`, Nim decides (correctly) that the
functions which use it aren't GC-safe because it's a global.)

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-08-17 17:18:26 +01:00
Jamie Lokier 3047c839dc
EVMC: Improve host call tracing and fix nested call C stack usage
This combines two things, a C stack usage change with EVM nested calls
via EVMC, and changes to host call tracing.

Feature-wise, the tracing is improved:

- Storage keys and values are make more sense.
- The message/result/context objects are shown with all relevant fields.
- `call` trace is split into entry/exit, so these can be shown around the
  called contract's operations, instead of only showing the `call` parameters
  after the nested call is finished.
- Nested calls are indented, which helps to highlight the flow.
- C stack usage considerably reduced in nested calls when more functionality
  is enabled (either tracing here, or other things to come).

This will seem like a minor patch, but C stack usage was the real motivation,
after plenty of time in the debugger.

Nobody cares about stack when `showTxCalls` (you can just use a big stack when
debugging).  But these subtle changes around the `call` path were found to be
necessary for passing all tests when the EVMC nested call code is completed,
and that's a prerequisite for many things: async EVM, dynamic EVM, Beam Sync,
and to fix https://github.com/status-im/nimbus-eth1/issues/345.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-08-12 07:48:56 +07:00
Jamie Lokier a7b40b0762
EVM: Use the EVMC calls for EIP-2929 access-list and refactor in EVM
Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-08-11 19:47:38 +07:00
Jamie Lokier 74f53c7761
EVMC: Add missing EIP-2929 (Berlin) functions to EVMC host
The update for London (EIP-1559) in 1cdb30df ("bump nim-emvc with evmc revision
8.0.0 to 9.0.0") really bumped EVMC ABI version from 7.5 up to 9.

In other words, it skipped Berlin, going direct from Istanbul to London.

That was accompanied by EVMC changes in 05e9b891 ("EIP-3198: add baseFee op
code in nim-evm"), which added the API changes needed for London.

But the missing Berlin functions weren't added in the move to London.

As a result, our EVMC host became incompatible with Berlin, London, and really
all revisions of the ABI, and if a third party EVM was loaded, it crashed.

This commit adds the missing Berlin host support, and makes our ABI
binary-compatible with real EVMC again.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-08-11 19:47:34 +07:00
jangko 7972e7a55c
clique: connect period and epoch from chain_config to engine
transfer cliquePeriod and cliqueEpoch from chain_config to
PoA engine.
2021-08-11 17:42:41 +07:00
jangko 77092641b5
add websocket rpc server 2021-08-06 07:32:19 +07:00
jangko 5e87624315
config: copy chainId to networkid if networkid not set in cli
although they are technically different, but in reality,
many networks are using the same id for ChainId dan NetworkId.
in this commit, we set networkid from config file's chainId.
2021-08-06 07:31:02 +07:00
jangko 1da4346295
config: fixes bug networkid parser
previously it mistakenly parse into the `result`
now it correctly parse networkId into res.
2021-08-06 07:31:01 +07:00