Commit Graph

767 Commits

Author SHA1 Message Date
Kim De Mey 2ad07a6010
Fix two bugs in Receipts RLP encoding/decoding (#672)
1. Fix Assertion error when receipt is not a List nor has a single
byte value. Receiving such garbage data would cause a crash.
2. Fix decoding of Receipt list by adding the missing Blob
encapsulation

Also added tests for these scenarios.
2024-02-05 23:22:15 +01:00
Kim De Mey 21f3014f54
Remove usage of stew/shim/net (#670)
* Remove usage of stew/shim/net and replace with std/net where needed

* Remove chronosStrictException compile flag
2024-01-24 09:09:11 +01:00
Kim De Mey 19965bab95
Fix chronos related deprecated warnings in uTP code part II (#669)
This also makes the uTP SendCallBack not returning a Future any
more as it is not used in sendData anyhow. And in case of uTP
over discv5, discv5 send call is already not async.

This gives quite a noticable throughput benchmark improvement over
with uTP over UDP, and a slightly noticable with uTP over discv5
2024-01-23 18:41:38 +01:00
Kim De Mey 496bcdab74
Fix chronos related deprecated warnings in discv5 and uTP (#667)
* Fix chronos related deprecated warnings in discv5

* Fix chronos related deprecated warnings in uTP code

* Improve discv5 closeWait order of events and add noCancel
2024-01-22 17:08:08 +01:00
andri lim b75ee36576
Refactor p2pProtocol internals (#657)
* Refactor p2pProtocol internals

* Attempt to fix rlp crash with Nim v2 (#658)

* Attempt to fix rlp crash with Nim v2
* Fix test_ecies for nim v2
* Reduce compiler warnings
* Resolve ambiquity in testutils

* Disable nim devel continue-on-error
2024-01-22 16:47:46 +07:00
Kim De Mey 2c6049b126
Fix a future .callback usage in rlpx (#666) 2024-01-19 12:29:11 +01:00
andri lim c57cebfd81
Fix discv4.send: use addCallback instead of .callback (#665) 2024-01-19 11:31:17 +01:00
Kim De Mey 159ae697c5
Remove last leftovers of ValidIpAddress (#664) 2024-01-19 11:15:57 +01:00
Kim De Mey 974a995b21
Add uTP over discv5 test and small uTP performance improvements (#663)
- Add a multiple sockets use test for uTP over discv5
- Use assign2 for the biggest consumer of genericAssignAux in uTP
- Avoid calling exists on the growable buffer when there is no
place in the socket window.
2024-01-18 12:07:03 +01:00
Etan Kissling a6f79a6000
Adjust test names and comments for `blobGasUsed` field (#661)
`blobGasUsed` was not mentioned in `test_rlp_codec` test names, despite
being used. Further, update the idea of `genOptionalFieldsValidation`
to also check for `blobGasUsed`, and fix the check to catch unsupported
combinations of optional fields.
2024-01-16 22:02:06 +01:00
Jacek Sieka e5e695c396
better async timeout wait (#659)
* don't call timeout code if future finishes before getting cancelled
* avoid extra raises effect resulting from `read` (vs `await`)
2024-01-16 18:12:47 +01:00
Kim De Mey cbcd1fd307
Fix for uTP issues with latest chronos (#655)
* Fix for uTP issues with latest chronos

* Fix for discv5 issue with latest chronos

* Increase amount of data send in uTP and uTP over discv5 tests
2024-01-12 17:35:44 +01:00
Kim De Mey a6942e3a30
Fix missing std/times import for the metrics 0.0.1 case (#656) 2024-01-12 17:35:14 +01:00
Kim De Mey 11049fb87d
Fix a bunch of compiler hints and warnings in uTP and discv5 (#652)
* Fix a bunch of compiler hints and warnings in uTP and discv5

* Leave in the cancel() for now
2024-01-12 09:49:06 +01:00
Jacek Sieka 0fc5dc03c5
support nim-metrics 0.1.0+ (#651) 2024-01-05 15:17:27 +01:00
Kim De Mey e5c2b1784e
Replace registerCustomScalarFunction with createCustomFunction (#649)
* Replace registerCustomScalarFunction with createCustomFunction

The nice thing about `registerCustomScalarFunction` and
`registerCustomScalarFunction` was that it kept the sqlite calls
of type sqlite3_value_xxx inside kvstore_sqlite.

The big downside however is that the "custom" call is awfully
specific due to the specific function signature it demands.
It is likely that for each different use case, a new version
needs to be added.

In this PR the code gets altered to `createCustomFunction` which
allows to just create a custom function with directly xFunc
signature of sqlite. The downside of this is that you still need
to import sqlite_abi in your local code to have access to each of
the sqlite3_value_xxx and sqlite3_result_xxx calls. These could of
course also be wrapped.

* Fix custom sql func test
2023-11-21 14:39:24 +01:00
tersec ca4898e24a
switch NAT module from ValidIpAddress to IpAddress (#647)
* switch NAT module from ValidIpAddress to IpAddress

* fix dcli build

* switch discv5 from ValidIpAddress to IpAddress

* fix fuzzers build

* mostly finish nim-eth to IpAddress transition
2023-11-10 09:28:21 +03:00
Kim De Mey 700360fde2
Rename routing_table_nodes metric for discv4 routing table (#646) 2023-10-31 18:28:21 +01:00
andri lim 4b00b230d0
Remove unused bootnodes and add Holesky bootnodes (#644)
* Remove unused bootnodes and add Holesky bootnodes

* Fix enode test
2023-10-25 11:07:55 +07:00
andri lim 1bae1d8426
Fix rlpHash of Transaction: remove EIP-4844 blobs before hashing (#643) 2023-10-22 14:37:49 +07:00
jangko e34a9c4e5d
Fix EthTime.now from func to proc due to sideeffects 2023-10-17 17:20:34 +07:00
jangko 2be8c183d1
Change EthTime from Time to distinct uint64
Cancun test vector demand a uint64 of block timestamp.
Therefore using std.Time will not able to pass test.
2023-10-17 16:00:57 +07:00
Jacek Sieka 34ab6bd986
rlp: avoid aliasing casts (#637)
* also get rid of unused concept
2023-09-25 11:30:47 +02:00
andri lim fe88d9e6b7
Change maxFeePerBlobGas from GasInt to UInt256 (#640)
* Change maxFeePerBlobGas from GasInt to UInt256

per latest spec:
https://eips.ethereum.org/EIPS/eip-4844#blob-transaction

* fix eip-4844 test
2023-09-24 14:33:24 +07:00
Jacek Sieka 02c8a1276b
avoid use of shallow (#636)
this probably has some tiny overhead but it shouldn't practically matter
2023-09-20 09:24:24 +02:00
jangko ac680ed79b
Add closeWait to EthereumNode 2023-09-08 21:45:41 +07:00
Kim De Mey 0d7e7448c4
Allow passing along the handshake ENR through talkresp handler (#634)
This allows for protocols build on top of discv5 to use the ENR
provided in the handshake directly, instead of having to rely
on requesting it from the discv5 routing table.
2023-08-30 17:44:05 +02:00
Jacek Sieka 22df101fcf
keyfile: reduce imports (#633) 2023-08-21 11:11:52 +02:00
Etan Kissling 894ec07d9f
fix `{.raises.}` annotation to `writeValue` (#631)
`writeValue` doesn't raise `SerializationError`, so the `{.push.}`
is not optimal. Move `{.raises.}` to each `proc`, same as other modules.
2023-08-19 19:03:10 +02:00
jangko 074edff1b4
EIP-4788: Add parentBeaconBlockRoot field to BlockHeader 2023-08-18 17:52:55 +07:00
Etan Kissling 946ffe0a4a
arc/orc lacks `shallowCopy` --> use `move` (#630)
`shallowCopy` is not available in `--mm:arc/orc`, but our usage can be
replaced with `move`.
2023-08-15 14:48:41 +02:00
andri lim 2ed8e991b5
add mixin env to sqlite3 generic procs (#627) 2023-08-03 22:00:47 +07:00
tersec d217d309eb
revert some func back to proc; unused by nim-eth tests (#629) 2023-08-02 23:00:49 +00:00
tersec 01ae82fd70
specify exceptions of callback called by exception-free function (#628) 2023-08-01 06:05:22 +00:00
jangko 521db96a65
rename data gas to blob gas 2023-07-28 10:19:00 +07:00
Jacek Sieka 15a09fab73
sqlite3: better error message (#622)
* sqlite3: better error message

* fix dispose
2023-07-06 06:53:29 +00:00
Kim De Mey 26ae539598
Use results Opt for discv5 getNode and resolve (#625) 2023-06-26 11:30:33 +02:00
jangko 6b8a7b009e
implement EIP-4844: Shard Blobs Transactions 2023-06-23 13:48:51 +07:00
Kim De Mey c5dd5e4115
Enable tests with also json sink and fix failures (#624)
And a few minor cleanups
2023-06-22 16:13:16 +00:00
Kim De Mey d74dc40bee
Style fixes and comment improvements on uTP code (#623) 2023-06-22 11:31:30 +00:00
jangko 6dacb2ca5c
fixes related to stint breaking changes 2023-06-21 17:55:00 +07:00
Kim De Mey c3860244b1
Fix potentential uTP socket leak for socket in SynSent state (#620)
Socket leak could because ST_DATA (and ST_FIN) packets for sockets
not in connected stated were only dropped after appling acks.
2023-06-21 12:44:52 +02:00
Kim De Mey cea23299be
Avoid double socket destroy on timed-out uTP SYN-ACK (#619) 2023-06-15 12:26:22 +02:00
tersec c608426d03
add/change Deneb fields per https://github.com/ethereum/execution-apis/pull/417 (#616) 2023-06-03 18:47:55 +00:00
Jacek Sieka 55b9da0bea
fix cancellation syntax (#615)
* fix cancellation syntax

* simplify
2023-06-02 14:33:06 +02:00
andri lim 91b2b9d2ed
rlp codec support optional fields (#613)
Instead of patching BlockHeader or BlockBody codec
each time it get additional optional fields,
this PR make the rlp codec automatically handle
optional fields. Thus rlp codec overloading of
EthBlock, BlockHeader, and BlockBody can be removed.
2023-05-30 16:02:02 +03:00
jangko 67bbd88616
Replace exception with assertion in rlp.append of BlockHeader
Using exception in rlp.append turn out creates more problem than it solve.
`blockHash` operation is literally everywhere in nimbus-eth1, nimbus-eth2, and fluffy-code.
The compiler begins to complain about unlisted exception.
Rlp bytes coming from network and test vectors already verified by the decoder.
So, any invalid optional fields of blockheader that come into rlp.append means
programming error. Assertion will be more apropiate in this situation and less problematic.
2023-05-25 15:27:37 +07:00
jangko 7ff6de2367
fix rlp.readRecordType of BlockBody and friends
including:
- fix rlp.append of BlockHeader
- add tests for rlp.readRecordType of BlockBody
- add tests for rlp.append of BlockHeader
- add tests for EthBlock roundtrip
2023-05-23 14:37:47 +07:00
jangko e639dc1e14
fix missing withdrawals field in EthBlock 2023-05-22 16:11:30 +07:00
Ivan Folgueira Bande 285da12bf3
nat.nim: breaking dependency with 'confutils' (#609)
* nat.nim: breaking dependency with 'confutils'

The main purpose of this change is to break the dependency with
`confutils`.
2023-05-16 13:43:31 +02:00
Jacek Sieka 6e5ee490c2
avoid ptr_arith (#608) 2023-05-11 07:39:21 +00:00
Kim De Mey d57a1094b1
Remove all Defect raises to avoid unnecessary warnings (#607)
As this module dropped support for Nim versions < 1.6, all Defect
raises can be removed to avoid unnecessary warnings
2023-05-10 15:50:04 +02:00
tersec 7564b21ade
support only Nim 1.6+ (#606) 2023-05-09 20:19:44 +00:00
jangko 607539d821
upgrade github actions to v3 2023-05-05 17:25:15 +07:00
Adam Spitz 4b818e8307
In the incomplete-db node-existence check, don't use contains. (#603)
* In the incomplete-db node-existence check, don't use contains.

(Using contains led to a problem with CaptureDB.)

* In the incomplete-db check, just checking len > 0 isn't right.

* Oh, I needed the AssertionDefect thing too.

* Need this when compiling under older versions of Nim.

* Sometimes we want missing nodes to be errors, sometimes not.
2023-04-28 12:04:33 -04:00
Adam Spitz f5dd26eac0
Some changes to make hexary.nim better able to handle incomplete DBs (#602)
* Added maybeGet, for working with incomplete DBs.

* Made trie.del throw an exception if it encounters a missing node.
2023-04-21 07:59:15 -04:00
jangko 917888356e
no more wild constants and rename ecdhSecret to ecdhSharedSecret 2023-04-21 16:08:57 +07:00
jangko 9bf2e1786b
remove ecdhRaw usage and replace it with ecdh+custom hash function 2023-04-20 15:45:23 +07:00
Jordan Hrycaj 9a1d35f803
Fix block body encoding (#598)
why:
  List wrapper was missing in `append()` mixin.
2023-04-19 13:26:41 +01:00
Kim De Mey 25b0da0280
Fix JSON writer for logging in devp2p code (#597) 2023-04-18 13:51:02 +02:00
Kim De Mey ea3c164a00
uTP clarification on SYN-ACK seqnr and some cleanup (#596)
- Clarify why on SYN-ACK seqNr gets -1
- Cleanup comments
- Style fixes
- Reuse WriteResult
2023-04-04 17:11:36 +02:00
Adam Spitz 4754543605
Added maybeGet, for working with incomplete DBs. (#595) 2023-04-03 13:01:25 -04:00
Adam Spitz 9e89f0dccc
Made rlp.readRecordType(BlockBody, false) include withdrawals. (#594) 2023-03-17 10:41:38 -04:00
Kim De Mey d2ba753792
Add rlpx metrics and avoid immediate peer reconnections (#585)
* Add metrics related to devp2p peer connections

* Avoid reconnecting to peers that just failed connection

- Add SeenTable to avoid reconnecting to peers immediately after
a failed connect. Depending on the failure, the amount of time is
different. This is similar to what is done in nimbus-eth2.
- Attempt to rework rlpxConnect at the same time, in order to
make sure that errors are properly handled. The current structure
is far from ideal, but it is hopefully a small step in the right
direction. To many oddities in there right now to really rework
rlpxConnect properply.

* Fix rlpx thunk fuzzer
2023-03-16 16:45:12 +01:00
Kim De Mey 29b14749fa
Improve uTP decoded packet logs + style and comment clean-up (#593)
* Improve uTP decoded packet logs + style and comment clean-up

* Don't test for the exact error strings in uTP decode + clean-up
2023-03-14 16:17:39 +00:00
Jacek Sieka 72c9858927
kvstore: remove unnecessary EH (#592) 2023-02-22 09:03:13 +00:00
Jacek Sieka ca1c205441
fix missing gcsafe (#591) 2023-02-22 08:11:32 +00:00
Jacek Sieka ade6d19691
Defect fixes in kvstore (#589)
* Defect fixes in kvstore

* compile fixes
2023-02-21 18:54:30 +01:00
Kim De Mey d2a3727c4c
Use chronos isGlobal to verify public IPs for net/nat and discv5 (#588) 2023-02-21 09:34:26 +01:00
Kim De Mey 4d55bfc9e8
Move dcli tool to a tools folder (#584) 2023-01-27 18:40:40 +01:00
Adam Spitz 20d2d318f0
Fixed a bug that I just introduced in the BlockBody RLP code. (#583)
The nimbus-eth1 tests seem to think that this is the right thing to do.
2023-01-21 14:41:30 +01:00
Adam Spitz 5b189ce5e2
EIP-4895: added withdrawals to BlockBody. (#581) 2023-01-20 12:18:39 +01:00
Adam Spitz 71b148a5b0
Propagating the del-returns-a-bool change through the trie DB code. (#578)
See here:
8f0ae55353
2023-01-17 16:39:28 -05:00
tersec 68f4c53828
withdrawals amount in uint64 gwei: https://github.com/ethereum/execution-apis/pull/354 (#577) 2023-01-14 14:39:23 +00:00
zah 0807378e7f
Migrate some functions from nimbus-eth1 in order to make them available in nimbus-eth2 (#576) 2023-01-10 18:56:27 +02:00
Adam Spitz 58284ffeda
Attempt to connect to many peers concurrently, not in sequence. (#575)
The old code, which did it in sequence, made it take a very long
time to accumulate peers in the peer pool, particularly because
occasionally a connection-attempt would hang for a couple of minutes
before timing out.
2023-01-04 05:56:23 -05:00
Jacek Sieka 8f0ae55353
kvstore: return bool from del, add clear (#574)
This allows making decisions based on whether an element was present
during the `del`
2023-01-03 16:38:27 +01:00
Kim De Mey 40ec601d07
Fix accidential usage of default rlp decode for DisconnectionReason (#572) 2022-12-14 10:57:23 +01:00
Etan Kissling 2b5f2a27e3
fix `excessDataGas` type (`GasInt` --> `UInt256`) (#571)
Used incorrect type in original PR. Fixing.
2022-12-14 00:25:27 +01:00
Etan Kissling c9fcab8052
extend `BlockHeader` for EIP-4844 (#570)
Extends `BlockHeader` with `excessDataGas` according to EIP-4844
(used by Nimbus-CL in empty block prod fallback).
2022-12-13 20:53:05 +01:00
Lorenzo Delgado 5c46220e72
add eth topic to all the logs scope (#568) 2022-12-06 14:54:03 +01:00
tersec 571b98d7f7
avoid invalid codegen/UB by avoiding case object discriminant assignment (#566) 2022-12-01 13:55:42 +01:00
Lorenzo Delgado 833818e9c7
Make discv5 `protocol-id` configurable at compile time
Co-authored-by: ksr <kaiserd@users.noreply.github.com>
2022-11-30 12:59:41 +01:00
KonradStaniec 22d0ac81e1
Fix defect when writing over send buffer (#564)
* Fix defect when writing over send buffer
2022-11-30 09:34:08 +01:00
Jacek Sieka 8f4ef19fc9
sqlite: support read-only kvstores (#563)
in a read-only database, we cannot create the table but we can still
reason about elements in it - they simply don't exist
2022-11-28 21:15:53 +01:00
Etan Kissling 6499ee2bc5
extend `BlockHeader` for Capella (#562)
Adds `Withdrawal` type according to EIP-4895, and extends `BlockHeader`
accordingly. Also adds RLP encoding support for `Withdrawal` to enable
building `BlockHeader` (used by Nimbus-CL in empty block prod fallback).
2022-11-25 11:07:51 +01:00
Mikel Cortes f4ef9181c7
persist discovered ENRs from the discv5 into CSV (#560)
* persist ENRs (parsed) from the discovery5 protocol

* filter fork_digest from eth2 data

* Update eth/p2p/discoveryv5/dcli.nim

Co-authored-by: Kim De Mey <kim.demey@gmail.com>

* apply reviewer's suggestions

Co-authored-by: Kim De Mey <kim.demey@gmail.com>
2022-11-23 23:48:44 +01:00
Kim De Mey 9ba1eb99e3
Fix possible crashes due to RangeErrors in rlpx auth decoding (#558)
- Fix two possible RangeErrors, due to negative seq allocation
in decodeAuthMessageEIP8 and in decodeAckMessageEIP8
- Set the minimum auth message size to AuthMessageEIP8Length,
in case there are clients that no longer add padding
- Add tests for invalid length cases
2022-11-17 08:46:27 +01:00
Justin Traglia e7d3de6ebf
Fix some typos (#557) 2022-11-16 17:44:00 +01:00
Justin Traglia ff0b1a330a
Allow distance of 256 in lookupDistances (#556) 2022-11-15 12:36:16 +01:00
Kim De Mey 4b22fcdce4
Proc to func in enr code and style guide related clean-up (#555)
Should not have any functional changes. Clean-up related to
avoiding result usage (also implicit), and other style
guide items.
2022-11-15 10:34:56 +01:00
Kim De Mey 522db295f2
Fix RLP deserialzation for Enum with holes (#554)
RLP Enum deserialization would currently not check if "hole values"
were attempted to be converted to the enum type. Now use
checkedEnumAssign and fail with RlpTypeMismatch on invalid values.

There is at least one occurance of an enum with holes in rlpx p2p:
DisconnectionReason. For this enum the issue could occur.

Also:
- Added enum RLP tests and rlpx p2p disconnect message tests to
test the DisconnectionReason with enum hole value.
- Fixed worse custom DisconnectionReason decoding occurance
in rlpx in waitSingleMsg proc where this issue could occur.
2022-11-14 15:49:37 +01:00
Kim De Mey 9b0f054b04
Tackle some Nim warnings new since Nim 1.4 & 1.6 (#553) 2022-11-10 16:32:57 +01:00
Kim De Mey 70b83a4efb
Improve the discv5 initialization logs (#552)
- Unify the debug log and info log into one. Removing the
redundant information
- Log the custom ENR fields more pretty
- Make the JSON format logging more pretty for several types
2022-11-10 10:52:24 +01:00
Kim De Mey 86b37bf72e
Default compile all tests and binaries with TRACE log level (#549)
In order to avoid unused warning and more importantly to make
sure all log statements work.
2022-11-10 09:01:58 +01:00
andri lim 10870d8b15
Revert #544 "Fix related to nim devel branch" (#551)
This reverts commit:
5d13052dd9
e1bdf1741a
d238693571
2022-11-09 18:57:04 +01:00
jangko 5d13052dd9
refactor: compile with nim-devel 2022-11-08 11:08:25 +07:00
jangko e1bdf1741a
refactor p2pProtocol internals
Nim devel brach(1.7.1) introduce gc=orc as default mode.
Because the p2p protocol using unsafe pointer operations
for it's ProtocolInfo and using global variables scattered
around, the orc mistakenly(or maybe correctly) crash the protocol.
2022-11-08 09:41:45 +07:00
jangko d238693571
refactor: replace shallowCopy with something else 2022-11-08 09:41:35 +07:00
Kim De Mey 64b56d866c
Add own node checks before sending ping (#548)
Seems there are two occasions possible where we try to ping the
own local node which causes an assert

- One via the bounding
- One via a received ping, which is strange in the first place but
notice in a stack trace
2022-11-07 14:25:11 +01:00