Commit Graph

116 Commits

Author SHA1 Message Date
andri lim bea558740f
Reduce compiler warnings (#2030)
* Reduce compiler warnings

* Reduce compiler warnings in test code
2024-02-16 16:08:07 +07:00
Kim De Mey 7b2dc8f8e9
Fix ReceiptType in premix + Apply validate flag for receipts (#2012) 2024-02-08 11:50:35 +01:00
web3-developer a5ac5d3078
Premix persist tool improvements (#2000)
* Add premix persist tool to Makefile to build as separate binary.

* Fix rootBytes.len > 0 assertion defect in HexaryTrie by adding call to com.db.compensateLegacySetup().

* Run hardForkTransition before executing transactions in getBlockWitness.

* Improve logging and add help message.

* Retry requests for block data.
2024-02-01 22:11:41 +08:00
jangko 3e21281d12
Bump nim-web3 and others
Bump nim-json-rpc and nimbus-eth2 too.
Reason: both nim-json-rpc and nim-web3 migrate from
stdlib/json to nim-json-serialization
2024-01-14 10:41:23 +07:00
jangko c0d52ba179
Get rid of HexQuantityStr and HexDataStr usage
Both types are not safe and require validation/conversion
from rpc implementer.
This PR change it to safer types and delegate the conversion
and validation to the rpc library.
2023-12-27 07:48:53 +07:00
jangko 5fd54961e3
Bump nim-web3: remove rpc types duplicates 2023-12-13 07:58:12 +07:00
Jordan Hrycaj 3e88589eb1
Optional accounts cache module for creating genesis (#1897)
* Split off `ReadOnlyStateDB` from `AccountStateDB` from `state_db.nim`

why:
  Apart from testing, applications use `ReadOnlyStateDB` as an easy
  way to access the accounts ledger. This is well supported by the
  `Aristo` db, but writable mode is only parially supported.

  The writable AccountStateDB` object for modifying accounts is not
  used by production code.

  So, for lecgacy and testing apps, the full support of the previous
  `AccountStateDB` is now enabled by `import db/state_db/read_write`
  and the `import db/state_db` provides read-only mode.

* Encapsulate `AccountStateDB` as `GenesisLedgerRef` or genesis creation

why:
  `AccountStateDB` has poor support for `Aristo` and is not widely used
   in favour of `AccountsLedger` (which will be abstracted as `ledger`.)

   Currently, using other than the `AccountStateDB` ledgers within the
   `GenesisLedgerRef` wrapper is experimental and test only. Eventually,
    the wrapper should disappear so that the `Ledger` object (which
    encapsulates `AccountsCache` and `AccountsLedger`) will prevail.

* For the `Ledger`, provide access to raw accounts `MPT`

why:
  This gives to the `CoreDbMptRef` descriptor from the `CoreDb` (which is
  the legacy version of 	CoreDxMptRef`.) For the new `ledger` API, the
  accounts are based on the `CoreDxMAccRef` descriptor which uses a
  particular sub-system for accounts while legacy applications use the
  `CoreDbPhkRef` equivalent of the `SecureHexaryTrie`.

  The only place where this feature will currently be used is the
 `genesis.nim` source file.

* Fix `Aristo` bugs, missing boundary checks, typos, etc.

* Verify root vertex in `MPT` and account constructors

why:
  Was missing so far, in particular the accounts constructor must
  verify `VertexID(1)

* Fix include file
2023-11-20 11:51:43 +00:00
jangko b0000eed8b
Add check copyright year linter to CI 2023-11-01 10:41:20 +07:00
jangko 26ede94a73
Add Holesky testnet configuration 2023-10-25 14:06:36 +07:00
jangko 63ff17efbf
Remove unused testnet configuration 2023-10-24 16:00:44 +07:00
andri lim 04c7ed8ec9
Change EthTime from std.Time to distinct uint64 (#1820)
* Change EthTime from std.Time to distinct uint64

* Bump nimbus-eth2
2023-10-18 09:16:11 +07:00
andri lim 34b1e29ac9
Fix test_blockchain_json and pyspec simulator for Cancun (#1805)
* Fix test_blockchain_json and pyspec simulator for Cancun

* Preserve applyDeletes comments

* Fix redefinition error

* Move test_macro to tests folder
2023-10-05 10:04:12 +07:00
andri lim 990718aa07
Prepare state test and blockchain test for Cancun (#1772) 2023-09-25 06:53:20 +07:00
andri lim 56215ed83f
Bump stint to v2.0: new array backend (#1747)
* Bump stint to v2.0: new array backend
2023-09-13 09:32:38 +07:00
jangko 92713ef326
Fix rpc.sendRawTransaction and txPool: reject invalid transaction earlier 2023-08-21 09:11:10 +07:00
Jordan Hrycaj 221e6c9e2f
Unified database frontend integration (#1670)
* Nimbus folder environment update

details:
* Integrated `CoreDbRef` for the sources in the `nimbus` sub-folder.
* The `nimbus` program does not compile yet as it needs the updates
  in the parallel `stateless` sub-folder.

* Stateless environment update

details:
* Integrated `CoreDbRef` for the sources in the `stateless` sub-folder.
* The `nimbus` program compiles now.

* Premix environment update

details:
* Integrated `CoreDbRef` for the sources in the `premix` sub-folder.

* Fluffy environment update

details:
* Integrated `CoreDbRef` for the sources in the `fluffy` sub-folder.

* Tools environment update

details:
* Integrated `CoreDbRef` for the sources in the `tools` sub-folder.

* Nodocker environment update

details:
* Integrated `CoreDbRef` for the sources in the
  `hive_integration/nodocker` sub-folder.

* Tests environment update

details:
* Integrated `CoreDbRef` for the sources in the `tests` sub-folder.
* The unit tests compile and run cleanly now.

* Generalise `CoreDbRef` to any `select_backend` supported database

why:
  Generalisation was just missed due to overcoming some compiler oddity
  which was tied to rocksdb for testing.

* Suppress compiler warning for `newChainDB()`

why:
  Warning was added to this function which must be wrapped so that
  any `CatchableError` is re-raised as `Defect`.

* Split off persistent `CoreDbRef` constructor into separate file

why:
  This allows to compile a memory only database version without linking
  the backend library.

* Use memory `CoreDbRef` database by default

detail:
 Persistent DB constructor needs to import `db/core_db/persistent

why:
 Most tests use memory DB anyway. This avoids linking `-lrocksdb` or
 any other backend by default.

* fix `toLegacyBackend()` availability check

why:
  got garbled after memory/persistent split.

* Clarify raw access to MPT for snap sync handler

why:
  Logically, `kvt` is not the raw access for the hexary trie (although
  this holds for the legacy database)
2023-08-04 12:10:09 +01:00
Adam Spitz d8a1adacaa
More work on withdrawals (#1482)
* Part of EIP-4895: add withdrawals processing to block processing.

* Refactoring: extracted the engine API handler bodies into procs.

Intending to implement the V2 versions next. (I need the bodies to be
in separate procs so that multiple versions can use them.)

* Working on Engine API changes for Shanghai.

* Updated nim-web3, resolved ambiguity in Hash256 type.

* Updated nim-eth3 to point to master, now that I've merged that.

* I'm confused about what's going on with engine_client.

But let's try resolving this Hash256 ambiguity.

* Still trying to fix this conflict with the Hash256 types.

* Does this work now that nimbus-eth2 has been updated?

* Corrected blockValue in getPayload responses back to UInt256.

c834f67a37

* Working on getting the withdrawals-related tests to pass.

* Fixing more of those Hash256 ambiguities.

(I'm not sure why the nim-web3 library introduced a conflicting type
named Hash256, but right now I just want to get this code to compile again.)

* Bumped a couple of libraries to fix some error messages.

* Needed to get "make fluffy-tools" to pass, too.

* Getting "make nimbus_verified_proxy" to build.
2023-03-09 18:40:55 -05:00
Kim De Mey 73e93f1f11
Reduce Nim 1.6 compiler warnings & hints
* Reduce Nim 1.6 compiler warnings/hints for Fluffy and Nimbus proxy

Mostly raises Defect removals, TaintedString removal and some
unnecessary imports.

Also updating the copyright years alongside.

* Further reduce Nim 1.6 compiler warnings/hints for Nimbus
2023-01-31 13:38:08 +01:00
jangko 94a94c5b65 implement better hardfork management 2022-12-02 13:51:42 +07:00
Kim De Mey 74a83c1229
Bump nim-eth and remove all now unneeded p2p related imports (#1273) 2022-10-20 10:34:59 +02:00
Jacek Sieka c2ed731fa5
eth: adapt to smaller eth_types (#1210) 2022-09-03 20:15:35 +02:00
jangko 709d8ef255
fix markCanonicalChain bug 2022-07-04 19:32:14 +07:00
KonradStaniec 196428dcf5
Use one ws connection to transfer data from source (#1136) 2022-06-22 08:50:58 +02:00
jangko f2f204293e
first step into styleCheck fixes 2022-04-14 08:39:50 +07:00
jangko 4d126f2461
hive: add ethereum/engine simulator
some test with multiple client still need polishing.
merge test using specially crafted blocks need to be added.
2022-04-13 08:05:58 +07:00
Jordan Hrycaj 261c0b51a7
Redesign of BaseVMState descriptor (#923)
* Redesign of BaseVMState descriptor

why:
  BaseVMState provides an environment for executing transactions. The
  current descriptor also provides data that cannot generally be known
  within the execution environment, e.g. the total gasUsed which is
  available not before after all transactions have finished.

  Also, the BaseVMState constructor has been replaced by a constructor
  that does not need pre-initialised input of the account database.

also:
  Previous constructor and some fields are provided with a deprecated
  annotation (producing a lot of noise.)

* Replace legacy directives in production sources

* Replace legacy directives in unit test sources

* fix CI (missing premix update)

* Remove legacy directives

* chase CI problem

* rebased

* Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation

why:
  Constructing a new 'AccountsCache' descriptor can be avoided sometimes
  when the current state root is properly positioned already. Such a
  feature existed already as the update function 'initStateDB()' for the
  'BaseChanDB' where the accounts cache was linked into this desctiptor.

  The function 'initStateDB()' was removed and re-implemented into the
  'BaseVmState' constructor without optimisation. The old version was of
  restricted use as a wrong accounts cache state would unconditionally
  throw an exception rather than conceptually ask for a remedy.

  The optimised 'BaseVmState' re-initialisation has been implemented for
  the 'persistBlocks()' function.

also:
  moved some test helpers to 'test/replay' folder

* Remove unused & undocumented fields from Chain descriptor

why:
  Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
jangko f051c2530e
fixes related to nim-json-rpc bump 2021-11-30 14:13:20 +07:00
jangko baf508f6ae
move stateDB from VMState to chainDB
previously, every time the VMState was created, it will also create
new stateDB, and this action will nullify the advantages of cached accounts.

the new changes will conserve the accounts cache if the executed blocks
are contiguous. if not the stateDB need to be reinited.

this changes also allow rpcCallEvm and rpcEstimateGas executed properly
using current stateDB instead of creating new one each time they are called.
2021-10-28 18:57:08 +07:00
jangko 8552dda7bb
premix: fixes premix parser dan graphql_downloader
premix parser:
- fix incorrect tx type comparison
- fix chainId parser to support both graphql and json returned data

graphql_downloadse:
- allow downloader to skip parsing transactions
2021-09-26 10:38:41 +07:00
jangko e23300e2cc
fix premix tools due to recent changes
- persist: fix related to new config
- parser: add EIP 2930 and EIP 1559 features in parseBlockHeader
  and parseTransaction
- add graphql_downloader as a supplement json-rpc downloader
2021-09-21 13:35:52 +07:00
jangko 69f2a0f95a
config: replace stdlib parseOpt with nim-confutils
fixes #581
2021-09-18 17:34:46 +07:00
jangko 48d497580a
config: remove last instance of getConfiguration usage from nimbus code
this is a preparation for migration to confutils based config
although there is still some getConfiguration usage in tests code
it will be removed after new config arrived
2021-09-08 21:25:14 +07:00
jangko c9cfebfa97
config: rearrange getConfiguration usage
avoid using getConfiguration inside object construction and
replace it with passing suitable param
2021-09-08 08:07:10 +07:00
Jordan Hrycaj ca07c40a48
Fearture/poa clique tuning (#765)
* Provide API

details:
  API is bundled via clique.nim.

* Set extraValidation as default for PoA chains

why:
  This triggers consensus verification and an update of the list
  of authorised signers. These signers are integral part of the
  PoA block chain.

todo:
  Option argument to control validation for the nimbus binary.

* Fix snapshot state block number

why:
  Using sub-sequence here, so the len() function was wrong.

* Optional start where block verification begins

why:
  Can speed up time building loading initial parts of block chain. For
  PoA, this allows to prove & test that authorised signers can be
  (correctly) calculated starting at any point on the block chain.

todo:
  On Goerli around blocks #193537..#197568, processing time increases
  disproportionally -- needs to be understand

* For Clique test, get old grouping back (7 transactions per log entry)

why:
  Forgot to change back after troubleshooting

* Fix field/function/module-name misunderstanding

why:
  Make compilation work

* Use eth_types.blockHash() rather than utils.hash() in Clique modules

why:
  Prefer lib module

* Dissolve snapshot_misc.nim

details:
  .. into clique_verify.nim (the other source file clique_unused.nim
  is inactive)

* Hide unused AsyncLock in Clique descriptor

details:
  Unused here but was part of the Go reference implementation

* Remove fakeDiff flag from Clique descriptor

details:
  This flag was a kludge in the Go reference implementation used for the
  canonical tests. The tests have been adapted so there is no need for
  the fakeDiff flag and its implementation.

* Not observing minimum distance from epoch sync point

why:
  For compiling PoA state, the go implementation will walk back to the
  epoch header with at least 90000 blocks apart from the current header
  in the absence of other synchronisation points.

  Here just the nearest epoch header is used. The assumption is that all
  the checkpoints before have been vetted already regardless of the
  current branch.

details:
  The behaviour of using the nearest vs the minimum distance epoch is
  controlled by a flag and can be changed at run time.

* Analysing processing time (patch adds some debugging/visualisation support)

why:
  At the first half million blocks of the Goerli replay, blocks on the
  interval #194854..#196224 take exceptionally long to process, but not
  due to PoA processing.

details:
  It turns out that much time is spent in p2p/excecutor.processBlock()
  where the elapsed transaction execution time is significantly greater
  for many of these blocks.

  Between the 1371 blocks #194854..#196224 there are 223 blocks with more
  than 1/2 seconds execution time whereas there are only 4 such blocks
  before and 13 such after this range up to #504192.

* fix debugging symbol in clique_desc (causes CI failing)

* Fixing canonical reference tests

why:
  Two errors were introduced earlier but ovelooked:
   1. "Remove fakeDiff flag .." patch was incomplete
   2. "Not observing minimum distance .." introduced problem w/tests 23/24

details:
  Fixing 2. needed to revert the behaviour by setting the
  applySnapsMinBacklog flag for the Clique descriptor. Also a new
  test was added to lock the new behaviour.

* Remove cruft

why:
  Clique/PoA processing was intended to take place somewhere in
  executor/process_block.processBlock() but was decided later to run
  from chain/persist_block.persistBlock() instead.

* Update API comment

* ditto
2021-07-30 15:06:51 +01:00
Jordan Hrycaj fbff3aea68
Feature/goerli replay clique poa (#743)
* extract unused clique/mining support into separate file

why:
  mining is currently unsupported by nimbus

* Replay first 51840 transactions from Goerli block chain

why:
  Currently Goerli is loaded but the block headers are not verified.
  Replaying allows real data PoA development.

details:
  Simple stupid gzipped dump/undump layer for debugging based on
  the zlib module (no nim-faststream support.)

  This is a replay running against p2p/chain.persistBlocks() where
  the data were captured from.

* prepare stubs for PoA engine

* split executor source into sup-modules

why:
  make room for updates, clique integration should go into
  executor/update_poastate.nim

* Simplify p2p/executor.processBlock() function prototype

why:
  vmState argument always wraps basicChainDB

* split processBlock() into sub-functions

why:
  isolate the part where it will support clique/poa

* provided additional processTransaction() function prototype without _fork_ argument

why:
  with the exception of some tests, the _fork_ argument is always derived
  from the other prototype argument _vmState_

details:
  similar situation with makeReceipt()

* provide new processBlock() version explicitly supporting PoA

details:
  The new processBlock() version supporting PoA is the general one also
  supporting non-PoA networks, it needs an additional _Clique_ descriptor
  function argument for PoA state (if any.)
  The old processBlock() function without the _Clique_ descriptor argument
  retorns an error on PoA networgs (e.g. Goerli.)

* re-implemented Clique descriptor as _ref object_

why:
  gives more flexibility when moving around the descriptor object

details:
  also cleaned up a bit the clique sources

* comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 14:14:45 +01:00
jangko d47be9047f
London: fix test_blockchain_json and test_generalstate_json
new London HF requires update to test code
2021-06-30 20:41:29 +07:00
jangko 4a188788bd
preparation for EIP-1559 implementation
- unify signTx in test_helper and signTransaction in rpc_utils
  and put it into transaction.nim
- clean up mess by previous EIP-2930
2021-06-29 07:33:48 +07:00
jangko a0d10f5728
drop PublicNetwork enum usage and replace it with NetworkId
we cannot limit the `--networkid` switch to values available in
`PublicNetwork` enum. it should able to accept very wide range of
custom NetworkId.
2021-05-20 14:04:16 +07:00
jangko 79044f1e92
eip2718: test_blockchain_json pass test 2021-05-15 18:09:35 +07:00
jangko 39ce2390ae
fixes `getRecipient`: using `sender` param instead of calculating sender itself
usually, there is always a sender around `getRecipient` call.
no need to recalculate sender. and more important, in some of
JSON-RPC/GraphQL call, the sender is come from `rpcCallData`,
not from `tx.getSender`. or in ohter situation when the tx is
an unsigned tx, without `r,s,v` fields to calculate sender.
2021-05-04 15:31:47 +07:00
Jamie Lokier 33481064fd
Premix: Remove `MordenNet` to match main program
Morden network has not had a working configuration in Nimbus for a few years,
so it's been removed form the main program.  Remove it from premix as well.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-27 11:11:24 +01:00
jangko 14cf2c8cac fixes premix/downloader.nim to compile with new json_rpc 2021-03-19 12:30:00 +07:00
jangko f6774b636a
fix premix/parser 2020-07-30 15:24:22 +07:00
jangko 9c38266ba7
implement eth_estimateGas 2020-07-29 12:42:32 +07:00
jangko f82dff64fa
implement more eth rpc and keystore management 2020-07-23 14:54:32 +07:00
jangko 165f9fea2e
reduce warnings 2020-07-21 13:15:06 +07:00
jangko 2e583e9aa0
fix persist tool to support testnet 2020-06-19 17:52:19 +07:00
andri lim 02db881eea
upgrade jquery to 3.5.0, uikit to 3.4.0 and silence github security alert 2020-05-01 11:24:49 +07:00
Jacek Sieka 4ade5797ee
rlp: don't use ranges / experimental features (#495) 2020-04-20 20:12:44 +02:00
andri lim f66f49168a
fix bc/gst tester for istanbul [skip ci] 2019-11-14 21:20:34 +07:00