Root encoding is on the hot path for block verification both in the
consensus (when syncing) and execution clients and oddly consititutes a
significant part of resource usage even though it is not that much work.
While the trie code is capable of producing a transaction root and
similar feats, it turns out that it is quite inefficient - even for
small work loads.
This PR brings in a helper for the specific use case of building tries
of lists of values whose key is the RLP-encoded index of the item.
As it happens, such keys follow a particular structure where items end
up "almost" sorted, with the exception for the item at index 0 which
gets encoded as `[0x80]`, ie the empty list, thus moving it to a new
location.
Armed with this knowledge and the understanding that inserting ordered
items into a trie easily can be done with a simple recursion, this PR
brings a ~100x improvement in CPU usage (360ms vs 33s) and a ~50x
reduction in memory usage (70mb vs >3gb!) for the simple test of
encoding 1000000 keys.
In part, the memory usage reduction is due to a trick where the hash of
the item is computed as the item is being added instead of storing it in
the value.
There are further reductions possible such as maintaining a hasher per
level instead of storing hash values as well as using a direct-to-hash
rlp encoder.
Since these types were written, we've gained an executable spec:
https://github.com/ethereum/execution-specs
This PR aligns some of the types we use with this spec to simplify
comparisons and cross-referencing.
Using a `distinct` type is a tradeoff between nim ergonomics, type
safety and the ability to work around nim quirks and stdlib weaknesses.
In particular, it allows us to overload common functions such as `hash`
with correct and performant versions as well as maintain control over
string conversions etc at the cost of a little bit of ceremony when
instantiating them.
Apart from distinct byte types, `Hash32`, is introduced in lieu of the
existing `Hash256`, again aligning this commonly used type with the spec
which picks bytes rather than bits in the name.
* Restricting exception catcher
why:
`CatchableError` is not needed here
* Check data length before converting to `openArray[]`
why:
Getting the first entry of an `openArray[]` crashes with `IndexDefect`.
This is particularly annoying when decoding messages in rlpx.
* Added unit test using rlpx message that causes this problem to detect
A first step in cleaning up RLP, which has lots of interesting issues -
the next step would be to clean up the exception handling as well
(Resultify?)
* remove `RlpNode` (unused)
* single-pass parsing for most functionality via RlpItem
* stricter conformance to spec
* remove float support
* warn about signed integers
* raise on invalid RLP earlier
* avoid several pointless allocations, in particular in `listLen`,
`listElem` etc
* include spec docs
This is more according to the structure of the code itself.
And also, currently some of these tests live under tests/rlp but
some under tests/common. Could use some further re-organisation
within the different tests, but this move is a first step.
1. Fix Assertion error when receipt is not a List nor has a single
byte value. Receiving such garbage data would cause a crash.
2. Fix decoding of Receipt list by adding the missing Blob
encapsulation
Also added tests for these scenarios.
* Refactor p2pProtocol internals
* Attempt to fix rlp crash with Nim v2 (#658)
* Attempt to fix rlp crash with Nim v2
* Fix test_ecies for nim v2
* Reduce compiler warnings
* Resolve ambiquity in testutils
* Disable nim devel continue-on-error
`blobGasUsed` was not mentioned in `test_rlp_codec` test names, despite
being used. Further, update the idea of `genOptionalFieldsValidation`
to also check for `blobGasUsed`, and fix the check to catch unsupported
combinations of optional fields.
Instead of patching BlockHeader or BlockBody codec
each time it get additional optional fields,
this PR make the rlp codec automatically handle
optional fields. Thus rlp codec overloading of
EthBlock, BlockHeader, and BlockBody can be removed.
Using exception in rlp.append turn out creates more problem than it solve.
`blockHash` operation is literally everywhere in nimbus-eth1, nimbus-eth2, and fluffy-code.
The compiler begins to complain about unlisted exception.
Rlp bytes coming from network and test vectors already verified by the decoder.
So, any invalid optional fields of blockheader that come into rlp.append means
programming error. Assertion will be more apropiate in this situation and less problematic.
including:
- fix rlp.append of BlockHeader
- add tests for rlp.readRecordType of BlockBody
- add tests for rlp.append of BlockHeader
- add tests for EthBlock roundtrip
Adds `Withdrawal` type according to EIP-4895, and extends `BlockHeader`
accordingly. Also adds RLP encoding support for `Withdrawal` to enable
building `BlockHeader` (used by Nimbus-CL in empty block prod fallback).
RLP Enum deserialization would currently not check if "hole values"
were attempted to be converted to the enum type. Now use
checkedEnumAssign and fail with RlpTypeMismatch on invalid values.
There is at least one occurance of an enum with holes in rlpx p2p:
DisconnectionReason. For this enum the issue could occur.
Also:
- Added enum RLP tests and rlpx p2p disconnect message tests to
test the DisconnectionReason with enum hole value.
- Fixed worse custom DisconnectionReason decoding occurance
in rlpx in waitSingleMsg proc where this issue could occur.
Currently only setting `--styleCheck:hint` as there are some
dependency fixes required and the compiler seems to trip over the
findnode MessageKind, findnode Message field and the findNode
proc. Also over protocol.Protocol usage.
* Add build_dcli target and add it to CI
* Fix local imports for dcli
* And use local imports for all other files too
* Use local imports in tests and rlpx protocols
* port kvstore from nim-beacon-chain
* remove old database backends
* use kvstore in trie database
* add sqlite dep
* avoid template param double evaluation
* clean up heterogenous lookup todo
Jacek Sieka
Jordan Hrycaj
Kim De Mey
andri lim
Etan Kissling
tersec
Jordan Hrycaj
Ștefan Talpalaru
Zahary Karadjov
Yuriy Glukhov