Commit Graph

534 Commits

Author SHA1 Message Date
KonradStaniec fcb0ff521c
Improve logging in uTP (#457)
* Improve logging
2022-01-07 10:38:19 +01:00
KonradStaniec 9c8e9d9f64
Handle selective acks (#456)
* Handle selective acks
2022-01-04 09:52:38 +01:00
KonradStaniec 664072fff7
Minor improvements from pr comments (#455) 2021-12-28 08:08:05 +01:00
Kim De Mey 42e257796f
Add discoveryv5 session metrics (#454) 2021-12-21 15:09:46 +01:00
Kim De Mey 8b7d41f596
Don't use exceptions for enr get call (#453)
The ENR code used to be solely exception based, and these
exceptions where a left-over of that. They are useless as later
calls use Result anyhow.

Additionally, they cause quite the performance loss because they
are used in the "common path" for the toTypedRecord call, e.g.
when reading the fields of ip6, tcp6 and udp6.
2021-12-21 15:09:32 +01:00
Kim De Mey 2c236f6495
Style fixes according to --styleCheck:usages (#452)
Currently only setting `--styleCheck:hint` as there are some
dependency fixes required and the compiler seems to trip over the
findnode MessageKind, findnode Message field and the findNode
proc. Also over protocol.Protocol usage.
2021-12-20 13:14:50 +01:00
KonradStaniec 0cfe7df817
Handle packets with selective acks (#451)
* Handle packets with selective acks
2021-12-15 13:35:17 +01:00
Etan Kissling 5655bd035c
Merge pull request #450 from etan-status/reject-whoareyou-len
reject WHOAREYOU packets with non-empty message
2021-12-14 16:22:25 +01:00
Ștefan Talpalaru 2088d7568d
CI: test with multiple Nim version (#429)
* CI: test with multiple Nim version

* clean up the testing tree a little

* replace "unittest" with "unittest2"
2021-12-11 19:12:55 +01:00
Etan Kissling 45387ad4d2
reject WHOAREYOU packets with non-empty message
This changes the `discv5` parser to reject malformed WHOAREYOU packets
that have a non-0 message length. The extra data used to be ignored.
The `message` part of WHOAREYOU packets is always empty.
See https://github.com/ethereum/devp2p/blob/master/discv5/discv5-wire.md
2021-12-11 15:55:14 +01:00
Etan Kissling fb7ea69eb4
Merge pull request #449 from etan-status/encryptgcm-type
more specific type check in `encryptGCM`
2021-12-11 14:50:35 +01:00
Etan Kissling 172dad7968
more specific type check in `encryptGCM`
Narrowed the type of `encryptGCM`'s `key` parameter from
`openarray[byte]` to `AesKey`, same as already used for `decryptGCM`.
2021-12-11 12:41:18 +01:00
Etan Kissling 923fc428f2
Merge pull request #447 from etan-status/rm-ssz
remove outdated and incorrect SSZ code
2021-12-10 16:57:31 +01:00
KonradStaniec b56e19a590
Improve handling of timestamps (#446)
* Improve handling of timestamps
2021-12-10 16:28:00 +01:00
Etan Kissling 3ce2d9a58e
remove outdated and incorrect SSZ code
This removes the outdated copy of the SSZ code. It became incorrect
over time (e.g., empty SSZ list elements), and is no longer in use by
GitHub projects: https://github.com/search?q=extension%3Anim+eth%2Fssz

The canonical SSZ implementation resides at `nim-ssz-serialization`.
Compared to `nim-eth`, these changes were made meanwhile:
- `bitseqs` was extended with JSON serialization support,
  and with the new functions `isZero` and `countOnes`.
- `bytes_reader` was renamed to `codec`, extended with a few additional
  SSZ type conversions as well as support for `SingleMemberUnion`.
- The simplified merkle tree implementation in `merkle_tree.nim`
  was removed. It was not used by other projects.
- `merkleization` was extended with support for `HashArray`, `HashList`
  and `SingleMemberUnion`. The `isValidProof` functionality has been
  moved to `nimbus-eth2` and replaced with the EF defined function
  `is_valid_merkle_branch`. The test was also moved to `nimbus-eth2`.
  There are no other GitHub projects using `isValidProof`:
  https://github.com/search?q=extension%3Anim+isValidProof
  Furthermore, a definition for `GeneralizedIndex` was added.
- `ssz_serialization` was moved one directory up, and improved with
  bug fixes and `HashArray`, `HashList` and `SingleMemberUnion` support.
- `types` was extended with JSON serialization and new type support for
  `Uint128`, `Uint256`, `HashArray`, `HashList` and `SingleMemberUnion`.
  There is also a new `getBit` function for `BitList`.
2021-12-10 15:51:22 +01:00
Kim De Mey 41d2d3c991
Utp top level raises & some clean-up (#445)
* Add few missing top level raises Defect in uTP

- Add top level {.push raises: [Defect].}
- remove some local raises, including some unneeded
CatchableErrors.
- Don't export messageHandler (avoiding annoying naming collisions)
- export utp_router as those connection callbacks are in the API

* Add some missing copyright clauses

* Some ident and max line length cleanup

* Rename utp_discv5_protocol.nim to be more consistent
2021-12-10 11:12:24 +01:00
KonradStaniec 09959d2a3f
Update timestamp and ackNr on packet resend (#444)
* Update timestamp and ackNr on packet resend
2021-12-10 10:18:00 +01:00
KonradStaniec b4066a5688
Ledbat congestion control (#443)
* Return delay to remote peer

* Initial ledbat window calculation

* Add tests for window grow and decay

* Add delay histograms

* Add calculation of clock drift
2021-12-09 10:52:21 +01:00
Kim De Mey ae0920d40d
Remove hashData usage on objects (#441)
* Remove hashData usage on objects

* Add hash func for NodeId to avoid using the one of stint
2021-12-06 15:24:07 +01:00
tersec 307b4e51b4
Point to current name of Ethereum consensus specs repo (#442) 2021-12-06 09:08:51 +01:00
KonradStaniec e7bc10ab00
Add config for max snd buffer size (#440)
* Add config for max snd buffer size
2021-12-02 16:51:44 +01:00
KonradStaniec 3c8915cae1
Track send buffer and properly handle back pressoure when window is to small to process data (#437)
* Add separate datastructure to keep track of window

* Asynchronously block write until until new space in snd buffer

* Introduce write loop

* Properly handle write cancellation

* Proper handling of sending fin packet

* Reset remote window after configured amount of time
2021-12-02 15:46:18 +01:00
Kim De Mey 6e21b32f0d
Allow a node to self resolve (#439) 2021-12-02 11:10:26 +01:00
Jamie Lokier 6a8d49e4c0 Security/RLPx: Fix crash when peer sends out of bounds message id
Closes [nimbus-eth1#767](https://github.com/status-im/nimbus-eth1/issues/767).

Crashes occur when certain invalid RLPx messages are received from a peer.
Specifically, `msgId` out of range.  Because any peer can easily trigger this
crash, we'd consider it a DOS vulnerability if Nimbus-eth1 was in general use.

We noticed when syncing to Goerli, there were some rare crashes with this
exception.  It turned out one peer with custom code, perhaps malfunctioning,
was sending these messages if we were unlucky enough to connect to it.

`invokeThunk` is called from `dispatchMessages` and checks the range of
`msgId`.  It correctly recognise that it's out of range, raises and exception
and produces a message.  Job done.

Except the code in `dispatchMessage` treats all that as a warning instead of
error, and continues to process the message.  A bit lower down, `msgId` is used
again without a range check.

The trivial fix is to check array bounds before access.

--

ps. Here's the stack trace ("reraised" sections hidden):

```
WRN 2021-11-08 21:29:33.238+00:00 Error while handling RLPx message          topics="rlpx" tid=2003472 file=rlpx.nim:607 peer=Node[<IP>:45456] msg=45 err="RLPx message with an invalid id 45 on a connection supporting eth,snap"
/home/jamie/Status/nimbus-eth1/nimbus/p2p/chain/chain_desc.nim(437) main
/home/jamie/Status/nimbus-eth1/nimbus/p2p/chain/chain_desc.nim(430) NimMain
/home/jamie/Status/nimbus-eth1/nimbus/nimbus.nim(258) process
/home/jamie/Status/nimbus-eth1/vendor/nim-chronos/chronos/asyncloop.nim(279) poll
/home/jamie/Status/nimbus-eth1/vendor/nim-chronos/chronos/asyncmacro2.nim(74) colonanonymous
/home/jamie/Status/nimbus-eth1/vendor/nim-eth/eth/p2p/rlpx.nim(1218) rlpxAccept
/home/jamie/Status/nimbus-eth1/vendor/nim-chronos/chronos/asyncmacro2.nim(101) postHelloSteps
/home/jamie/Status/nimbus-eth1/vendor/nim-chronos/chronos/asyncmacro2.nim(74) colonanonymous
/home/jamie/Status/nimbus-eth1/vendor/nim-eth/eth/p2p/rlpx.nim(985) postHelloSteps
/home/jamie/Status/nimbus-eth1/vendor/nim-chronos/chronos/asyncmacro2.nim(101) dispatchMessages
/home/jamie/Status/nimbus-eth1/vendor/nim-chronos/chronos/asyncmacro2.nim(77) colonanonymous
/home/jamie/Status/nimbus-eth1/vendor/nim-eth/eth/p2p/rlpx.nim(614) dispatchMessages
/home/jamie/Status/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/chcks.nim(23) raiseIndexError2
/home/jamie/Status/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/fatal.nim(49) sysFatal
[[reraised from: ... ]]
[[reraised from: ... ]]
[[reraised from: ... ]]
[[reraised from: ... ]]
Error: unhandled exception: index 45 not in 0 .. 40 [IndexError]
```

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-11-30 20:03:01 +02:00
Kim De Mey ae0574fe61
Adjust logging when node is not reachable but enrAutoUpdate is on (#436)
Separate the logging when the node is not reachable and
enrAutoUpdate is on or off to avoid confusion whether or not the
node might still become reachable.
2021-11-29 22:13:08 +01:00
Jacek Sieka 2baa4c02a1
avoid allocation in `hash(ValidIpAddress)` (#433)
* avoid allocation in `hash(ValidIpAddress)`

While casually browsing the profiler output, to my great surprise I
found that an allocating string conversion function (gasp!) in the hash
function for ip addresses - this PR carefully excises this evil
construct from the codebase.

* bump nim version
2021-11-29 20:58:45 +01:00
KonradStaniec 139c6fa2a8
Track current bytes in flight (#434) 2021-11-24 17:49:13 +01:00
Kim De Mey 84f755d792
Revert the useless async change for queryRandom (#432) 2021-11-22 23:14:37 +01:00
Kim De Mey 086162183c
Make queryRandom async and add exports (#431)
queryRandom was currently only async for the `enrField` version.
However the basic queryRandom is also exported and thus gets
changed so it can be properly used as async proc.

Also added exports for the modules of which objects are used in
the discovery public API.
2021-11-22 19:45:38 +01:00
Kim De Mey e606d8c79e
Export discovery routing table and its buckets nodes (#430) 2021-11-22 18:53:52 +01:00
KonradStaniec ce296ff76e
Make initial state of socket configurable (#428)
* Make initial state of socket configurable
2021-11-19 11:36:46 +01:00
KonradStaniec d5e5ec9f90
Add possibility to connect with requested conneciton id (#425)
* Improve error handling when initiating connection

* Add api to connect with requested id

* Add callback to allow only specific incoming peers
2021-11-18 10:05:56 +01:00
Kim De Mey 22757db83b
Fix logDistance for BE arch and remove toBytes for NodeId (#427)
toBytes for NodeId wasn't selected by compiler byt if it does
get selected, it will fail on the test cases due to the
countdown that is done in logDistance.
Set to toBytesBE properly now and do countup, that should make
it correct also for BE architecture.

Removed toBytes to avoid confusion and avoid this one being
selected ever. The only place toBytes for NodeId was used is in
sessions.nim makeKey func and there also the stint one
(thus native endianness) was selected in Nim 1.2.x.
Native endianness is fine there as it is only an internal
representation.
2021-11-17 22:55:19 +01:00
Zahary Karadjov fe1bb4c4e7
Allow Sqlite keystores to be used in read-only mode
This is useful for tools such as `ncli_db` that can work with the database
of a running Nimbus instances.
2021-11-16 13:45:46 +02:00
KonradStaniec 73d9bf4c80
Add ackNr validation (#424) 2021-11-15 11:32:00 +01:00
KonradStaniec 8139aae346
Advertise correct rcv buffer size (#423)
* Advertise correct rcv buffer size
2021-11-12 10:58:49 +01:00
KonradStaniec b671f6c901
Handling of fin packet (#421)
* Handling of connection finalization by sending and receiving FIN packets
2021-11-09 15:29:59 +01:00
KonradStaniec 7a113ffa48
Add handling of reset packet (#420)
* Add handling of reset packet
2021-11-05 09:41:41 +01:00
KonradStaniec d4cc42241d
Add handling of out of order packets (#418)
* Add handling of out of order packets
2021-11-04 07:38:46 +01:00
KonradStaniec 34bac6e703
Utp code cleanup (#417)
* Refactor tests and move socket to separate file

* Move sockets handling to separate class

* Abstract over underlying transport

* Fix bug with receiving duplicated SYN packet

* Fix race condition in connect
2021-10-28 11:41:43 +02:00
KonradStaniec fd4f78d1c0
Add timeout loop (#416)
* Modify outbuffer

Each element of outbuffer keeps encoded packet ,number
of transmissions of givern packet and information if
given packet needs to be re-send.

* Add initial handling of timeouts

* Add tests for syn re-sends
2021-10-25 09:58:13 +02:00
Etan Kissling d34d3409da
avoid func call when merkleizing UintN arrays (#413)
This gets rid of an unnecessary function call when merkleizing `UintN`
arrays on `littleEndian` architectures.
2021-10-21 10:40:40 +02:00
Etan Kissling 33f548186d
allow `ElemType(XXX)` on inputs not called `T` (#414)
Currently, `ElemType` can only be called on `List` types when they are
first assigned to a variable called `T`. The template has been adjusted
so that different variable names may be used. The template can now also
be applied to in-line computed types, e.g., `ElemType(typeof(x))`.
2021-10-21 09:43:42 +02:00
Etan Kissling 004ea06b80
fix big-endian merkleization for UintN arrays (#412)
UintN arrays were incorrectly merkleized on big-endian. This was fixed
by making sure to use the correct buffer to store the final chunk.
2021-10-21 09:43:14 +02:00
Etan Kissling 3d78c66119
fix compile error due to cyrillic T (#411)
There was a cyrillic T in some big-endian specific code that broke the
compilation on such platforms. This replaces that T with an ASCII T to
fix the build.
2021-10-20 14:21:06 +02:00
Etan Kissling 6272eaa6cd
fix `markleizer` typo (-> `merkleizer`) (#410)
Fix for a typo in a variable name.
2021-10-20 14:20:58 +02:00
Kim De Mey 9a1bb5e125
Make ENR $ call print the IP address prettier (#409) 2021-10-19 14:13:09 +02:00
KonradStaniec 88795c6477
Add sending and receiving data procedures (#407)
* Add sending and receiving data procedures
2021-10-19 13:36:57 +02:00
Kim De Mey f101c83626
Enr rlp lists (#408)
* Don't fail ENR decoding when value is an RLP list

* Store RLP raw list in the ENR field pair instead

* Add ENR kList FieldKind so lists can be treated differently

Treated differently now when printing out the ENR, mentioning
that it is a raw RLP list
2021-10-19 09:26:14 +02:00
KonradStaniec 6fbf129ba9
Add initial handling of acks (#406)
* Add initial handling of acks

Add implemetaion of circular buffer based on reference implementation
Add way to test number of packet in flight
Add acking of initial syn packet
2021-10-15 13:38:51 +02:00