* Experimental MP-trie
why:
Deleting records is a infeasible with the current structure
* Added vertex ID recycling management
Todo:
Provide some unit tests
* DB layout update
why:
Main news is the separation of `Merkel` hashes into an extra table.
details:
The code fragments cover conversion between compact MPT records and
Aristo DB records as well as some rudimentary cache handling for
the `Merkel` hashes (i.e. the extra table entries.)
todo:
Add some simple unit test for the descriptor record (currently used
for vertex ID management, only.)
* Updated vertex ID recycling management
details:
added simple unit tests (mainly testing ABI)
* docu update
* Recreating some of the speculative-execution code.
Not really using it yet. Also there's some new inefficiency in
memory.nim, but it's fixable - just haven't gotten around to it yet.
The big thing introduced here is the idea of "cells" for stack,
memory, and storage values. A cell is basically just a Future (though
there's also the option of making it an Identity - just a simple
distinct wrapper around a value - if you want to turn off the
asynchrony).
* Bumped nim-eth.
* Cleaned up a few comments.
* Bumped nim-secp256k1.
* Oops.
* Fixing a few compiler errors that show up with EVMC enabled.
* Update sync scheduler pool mode
why:
The pool mode allows to loop over active peers one after another. This
is ideal for soft re-starting peers. As this is a two tier experience
(start/stop, setup/release) the loop must be run twice. This is
controlled by a more rigid re-definition of how to use the `poolMode`
flag.
* Mitigate RLP serialiser deficiency
why:
Currently, serialising the `BlockBody` in not conevrtible and need
to be checked in the `eth` module. Currently a local fix for the
wire protocol applies. Unit tests will stay (after this local solution
will have been removed.)
* Code cosmetics and massage
details:
Main part is `types.toStr()` as a unified function for logging block
numbers.
* Allow to use a logical genesis replacement (start of history)
why:
Snap sync will set up an arbitrary pivot at a block number different
from zero. In fact, the higher the block number the better.
details:
A non-genesis start of history will currently only affect the score
values which were derived from the difficulty.
* Provide function to store the snap pivot block header in chain db
why:
Together with the start of history facility, this allows to proceed
with full syncing once snap has finished.
details:
Snap db storage was switched from a sub-tables to the flat chain db.
* Provide database completeness and sanity checker
details:
For debugging on smaller databases, only
* Implement snap -> full sync switch
t8n: a silly bug contract address generator, should use original
tx nonce instead of read the nonce from sender address in state db.
Although in EVM contract address generated by reading nonce from state db
is correct, outside EVM that nonce value might have been modified,
thus generating incorrect contract address.
accounts cache: when clearing account storage, the originalValue
cache is not cleared, only the storageRoot set to empty storage root,
this will cause getStorage and getCommitedStorage return wrong value
if the originalValue cache contains old value.
simplify EVM and delegete those things to accounts cache.
also no more manual state clearing, accounts cache will be
responsible for both collecting touched account and perform
state clearing.
* Gwei conversion should use u256 because u64 can overflow.
* Make withdrawals follow the EIP-158 state-clearing rules.
(i.e. Empty accounts should be deleted.)
* Allow the zero address in normalizeNumber.
(Necessary for one of the new withdrawals-related tests.)
* Silence some compiler gossip -- part 5, common
details:
Mostly removing redundant imports and `Defect` tracer after switch
to nim 1.6
* Silence some compiler gossip -- part 6, db, rpc, utils
details:
Mostly removing redundant imports and `Defect` tracer after switch
to nim 1.6
* Silence some compiler gossip -- part 7, randomly collected source files
details:
Mostly removing redundant imports and `Defect` tracer after switch
to nim 1.6
* Silence some compiler gossip -- part 8, assorted tests
details:
Mostly removing redundant imports and `Defect` tracer after switch
to nim 1.6
* Clique update
why:
More impossible exceptions (undoes temporary fix from previous PR)
* Updated to the latest nim-eth, nim-rocksdb, nim-web3
* Bump nimbus-eth2 module and fix related issues
Temporarily disabling Portal beacon light client network as it is
a lot of copy pasted code that did not yet take into account
forks. This will require a bigger rework and was not yet tested
in an actual network anyhow.
* More nimbus fixes after module bumps
---------
Co-authored-by: Adam Spitz <adamspitz@status.im>
Co-authored-by: jangko <jangko128@gmail.com>
Two unresolved items currently:
- Three tests that are temporarily disabled as they fail in the
macro_assembler code, which seems to be due to an ambigious
identifier Stop (Ops and chronos ServerCommand enum).
- i386 CI disabled as it fails at Nim compilation already. Failed
tests where already ignored for this target.
* Piecemeal trie inspection
details:
Trie inspection will stop after maximum number of nodes visited.
The inspection can be resumed using the returned state from the
last session.
why:
This feature allows for task switch between `piecemeal` sessions.
* Extract pivot helper code from `worker.nim` => `pivot_helper.nim`
* Accounts import will now return dangling paths from `proof` nodes
why:
With proper bookkeeping, this can be used to start healing without
analysing the the probably full trie.
* Update `unprocessed` account range handling
why:
More generally, the API of a pairs of unprocessed intervals favours
the first set and not before that is exhausted the second set comes
into play.
This was unfortunately implemented which caused the ranges to be
unnecessarily fractioned. Now the number of range interval typically
remains in the lower single digit numbers.
* Save sync state after end of downloading some accounts
details:
restore/resume to be implemented later
* Update log ticker, using time interval rather than ticker count
why:
Counting and logging ticker occurrences is inherently imprecise. So
time intervals are used.
* Use separate storage tables for snap sync data
* Left boundary proof update
why:
Was not properly implemented, yet.
* Capture pivot in peer worker (aka buddy) tasks
why:
The pivot environment is linked to the `buddy` descriptor. While
there is a task switch, the pivot may change. So it is passed on as
function argument `env` rather than retrieved from the buddy at
the start of a sub-function.
* Split queues `fetchStorage` into `fetchStorageFull` and `fetchStoragePart`
* Remove obsolete account range returned from `GetAccountRange` message
why:
Handler returned the wrong right value of the range. This range was
for convenience, only.
* Prioritise storage slots if the queue becomes large
why:
Currently, accounts processing is prioritised up until all accounts
are downloaded. The new prioritisation has two thresholds for
+ start processing storage slots with a new worker
+ stop account processing and switch to storage processing
also:
Provide api for `SnapTodoRanges` pair of range sets in `worker_desc.nim`
* Generalise left boundary proof for accounts or storage slots.
why:
Detailed explanation how this works is documented with
`snapdb_accounts.importAccounts()`.
Instead of enforcing a left boundary proof (which is still the default),
the importer functions return a list of `holes` (aka node paths) found in
the argument ranges of leaf nodes. This in turn is used by the book
keeping software for data download.
* Forgot to pass on variable in function wrapper
also:
+ Start healing not before 99% accounts covered (previously 95%)
+ Logging updated/prettified
* Provided common scheduler API, applied to `full` sync
* Use hexary trie as storage for proofs_db records
also:
+ Store metadata with account for keeping track of account state
+ add iterator over accounts
* Common scheduler API applied to `snap` sync
* Prepare for accounts bulk import
details:
+ Added some ad-hoc checks for proving accounts data received from the
snap/1 (will be replaced by proper database version when ready)
+ Added code that dumps some of the received snap/1 data into a file
(turned of by default, see `worker_desc.nim`)
* Relocated `IntervalSets` to nim-stew repo
* Accumulate accounts on temporary kv-DB
why:
Explore the data as returned from snap/1. Will be converted to a
`eth/db` next.
details:
Verify and accumulate per/state-root accounts downloaded via snap.
also:
Some unit tests
* Replace `Table` by `TrieDatabaseRef` for accounts accumulator
* update ticker statistics
details:
mean/variance based counter update
* allow persistent db for proved accounts
* rebase, and globally activate unit test
* fix statistics
Includes a simple test harness for the merge interop M1 milestone
This aims to enable connecting nimbus-eth2 to nimbus-eth1 within
the testing protocol described here:
https://github.com/status-im/nimbus-eth2/blob/amphora-merge-interop/docs/interop_merge.md
To execute the work-in-progress test, please run:
In terminal 1:
tests/amphora/launch-nimbus.sh
In terminal 2:
tests/amphora/check-merge-test-vectors.sh
* Redesign of BaseVMState descriptor
why:
BaseVMState provides an environment for executing transactions. The
current descriptor also provides data that cannot generally be known
within the execution environment, e.g. the total gasUsed which is
available not before after all transactions have finished.
Also, the BaseVMState constructor has been replaced by a constructor
that does not need pre-initialised input of the account database.
also:
Previous constructor and some fields are provided with a deprecated
annotation (producing a lot of noise.)
* Replace legacy directives in production sources
* Replace legacy directives in unit test sources
* fix CI (missing premix update)
* Remove legacy directives
* chase CI problem
* rebased
* Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation
why:
Constructing a new 'AccountsCache' descriptor can be avoided sometimes
when the current state root is properly positioned already. Such a
feature existed already as the update function 'initStateDB()' for the
'BaseChanDB' where the accounts cache was linked into this desctiptor.
The function 'initStateDB()' was removed and re-implemented into the
'BaseVmState' constructor without optimisation. The old version was of
restricted use as a wrong accounts cache state would unconditionally
throw an exception rather than conceptually ask for a remedy.
The optimised 'BaseVmState' re-initialisation has been implemented for
the 'persistBlocks()' function.
also:
moved some test helpers to 'test/replay' folder
* Remove unused & undocumented fields from Chain descriptor
why:
Reduces attack surface in general & improves reading the code.
previously, every time the VMState was created, it will also create
new stateDB, and this action will nullify the advantages of cached accounts.
the new changes will conserve the accounts cache if the executed blocks
are contiguous. if not the stateDB need to be reinited.
this changes also allow rpcCallEvm and rpcEstimateGas executed properly
using current stateDB instead of creating new one each time they are called.
this is a preparation for migration to confutils based config
although there is still some getConfiguration usage in tests code
it will be removed after new config arrived
instead of using stdlib/json, now we switch to json_serialization
the result is much tidier code and more robust when parsing
optional fields.
fixes#635
instead of using header as input param, now getReceipts using
receiptRoot hash, the intention is clearer and less data passed around
when we only using receiptRoot instead of whole block header.
- dynamically generated copyright year interval
- added the db backend to the header
- documented the db-backend-changing define, made it case insensitive
and ensured wrong values would trigger compilation errors
- added some useful `chronicles` defines in the top-level nim.cfg
- converted some assert() calls to error codes to avoid a reported
segfault with -d:release
- the recoverable errors are being recovered from, in persistWorkItem()
- the only case that's a show stopper raises an exception
Also implements transactional block persistence. Two issues
in the transaction processing code have been discovered that
might affect other usages such as the CALL instruction.
The main fix gets us past block 49000.
You may need to clean up your database.
By default, the database files will be written in the
platform-specific application data folder:
$HOME/AppData/Roaming/Nimbus/DB
$HOME/Library/Application Support/Nimbus/DB
$HOME/.cache/nimbus/db
* partly fix state_db getCode to get 20 new working GeneralStateTests; remove 2 functions which existed as workarounds; switch all remaining setBalance calls in GeneralStateTestRunner to subBalance (addBalance calls already changed)
* two of the 20 new tests don't work in 32-bit builds
This blindly changes logging to nim-chronicles - issues that ensue:
* keeps gas cost computation logs hidden behind flag
* unclear if logScope is practical - for example, since vm is split over
many files, topics get lost when using simple top-level per-module
topics
* when passing named object around, scope should incliude the name of
the object but this is caught neither by logScope nor by dynamicLogScope
* move forks constants, rename errors
* Move vm/utils to vm/interpreter/utils
* initial opcodes refactoring
* Add refactored Comparison & Bitwise Logic Operations
* Add sha3 and address, simplify macro, support pop 0
* balance, origin, caller, callValue
* fix gas copy opcodes gas costs, add callDataLoad/Size/Copy, CodeSize/Copy and gas price opcode
* Update with 30s, 40s, 50s opcodes + impl of balance + stack improvement
* add push, dup, swap, log, create and call operations
* finish opcode implementation
* Add the new dispatching logic
* Pass the opcode test
* Make test_vm_json compile
* halt execution without exceptions for Return, Revert, selfdestruct (fix#62)
* Properly catch and recover from EVM exceptions (stack underflow ...)
* Fix byte op
* Fix jump regressions
* Update for latest devel, don't import old dispatch code as quasiBoolean macro is broken by latest devel
* Fix sha3 regression on empty memory slice and until end of range slice
* Fix padding / range error on expXY_success (gas computation left)
* update logging procs
* Add tracing - expXY_success is not a regression, sload stub was accidentally passing the test
* Reuse the same stub as OO implementation
* Delete previous opcode implementation
* Delete object oriented fork code
* Delete exceptions that were used as control flows
* delete base.nim 🔥, yet another OO remnants
* Delete opcode table
* Enable omputed gotos and compile-time gas fees
* Revert const gasCosts -> generates SIGSEGV
* inline push, swap and dup opcodes
* loggers are now template again, why does this pass new tests?
* Trigger CI rebuild after rocksdb fix https://github.com/status-im/nim-rocksdb/pull/5
* Address review comment on "push" + VMTests in debug mode (not release)
* Address review comment: don't tag fork by default, make opcode impl grepable
* Static compilation fixes after rebasing
* fix the initialization of the VM database
* add a missing import
* Deactivate balance and sload test following #59
* Reactivate stack check (deactivated in #59, necessary to pass tests)
* Merge remaining opcodes implementation from #59
* Merge callDataLoad and codeCopy fixes, todo simplify see #67
Merge note: currently cannot compile due to `quasiBoolean` (#63). This will be solved by https://github.com/status-im/nimbus/pull/65
----
* Implemented most of the stubbed out state handling instructions
The code compiles, but still fails at the moment due to incorrect
initialization of the VM. Don't merge yet. More commits will be
pushed in the coming days.
* Fixed crash
* trie put and del are void now
* getBlockTransactionData and getReceipts
* Working code for extcodesize0.json
* fix origin.json
* fix calldatasize1
* fix calldataloadSizeTooHighPartial
* fix calldataloadSizeTooHigh
* more efficient PushX implementation
* fix and, or, xor
* `usedBytes` was not named appropriately. it indicates the last
populated position in the data (the last valid index) for slicing
purposes (i.e. the position is passed to `toOpenArray`)
* Fixed the `==` operator for DBKeys
* Use one-byte key for the 'canonical head hash'
* Move and cleanup interpreter files - prepare for redesign of VM
* fix call comment aobut recursive dependencies
* memory: use a template again and avoid (?) a cstring-> string conversion
* Fix stack test regression
* Fix recursive dependency on logging_ops, test_vm_json compiles but regression :/
* Fix signextend regression
* Fix 3 signed test and sha3 test
* Decoupling op logic and gas - introduce gasometer, rework opcode declaration
* Remove gas constants for gas opcode computation
* Remove gas constants for precompiled contracts
* make vm_types compile
* Make opcode, call and computation compile
* Distinguish between dynamic and complex gas costs, fix arithmetic
* Fix context and sha3
* update memory and storage ops
* Log opcode uses memory expansion code
* update/stub system_ops with gas costs
* Make test compile. Deactivate stub test_vm
* all tests compiles, opcode fails due to https://github.com/nim-lang/Nim/issues/8007 (const object variant in tables reset at runtime)
* Create an enum without holes - workaround: https://github.com/nim-lang/Nim/issues/8007
* Use arrays instead of tables for GasCosts, remove some unused imports - passing all basic tests!
* Make test_vm_json compile
* Fix test_vm_json - workaround https://github.com/nim-lang/Nim/issues/8015
* fix memory expansion cost bug
* Remove leftover special handling from before GckMemExpansion
* cleanup outdated comment, better align =
* Fix sha3 gas cost not taking memory expansion into account
* Improve gas error reporting of test_vm_json
* Fix gas computation regression due to mem expansion
* mass replace for memExpansion->RequestedMemSize was too eager
* fix log gas cost (no tests :/)
* missed a static FeeSchedule
* static as expression is fickle
Jordan Hrycaj
Adam Spitz
jangko
Kim De Mey
Jacek Sieka
KonradStaniec
Zahary Karadjov
Ștefan Talpalaru
Yuriy Glukhov
Yuriy Glukhov
Dustin Brody
coffeepots
coffeepots
Mamy Ratsimbazafy