Commit Graph

559 Commits

Author SHA1 Message Date
Kim De Mey eb0908e33f
Push raises Defect to rlpx and accompanying changes (#373) 2021-07-14 10:35:35 +02:00
Kim De Mey 79911ed5d8
Log distance to uint16 and add public neighbours calls (#371)
* Use uint16 instead of uint32 for discv5 log distance

* Make neighboursAtDistances and neighbours calls available
2021-07-13 10:05:46 +02:00
Kim De Mey 41127eaee8
Remove portal wire code which was moved to nimbus-eth1 repo (#370) 2021-07-09 22:14:31 +02:00
Ștefan Talpalaru 2dfd352fd0 metrics: initialise lock for custom collector 2021-07-06 17:46:23 +03:00
jangko a10d301085
implement EIP-1559 Transaction and BlockHeader 2021-06-27 20:12:47 +07:00
kdeme 1c400e3f0e
Improve Portal message tracing 2021-06-09 14:57:35 +02:00
kdeme 880b753ad2
Add Portal wire readme doc 2021-06-09 14:57:35 +02:00
kdeme e2e30247bf
Add implementation of Portal wire protocol 2021-06-09 14:57:30 +02:00
Kim De Mey d18ebaa570
Slightly improved logging traces for error on message responses (#364) 2021-06-09 14:55:00 +02:00
Jacek Sieka 3514ee6484
sqlite: quick exec with result (#361) 2021-05-27 11:31:34 +02:00
Jacek Sieka 68e6aadc29 sqlite3: support option type
.. and any integer when writing but don't support int32 on reads -
internally sqlite will truncate on overflow which isn't nice.
2021-05-26 13:32:31 +03:00
Jacek Sieka 2a292cfb62
sqlite: fix readonly mode (#359)
* sqlite: fix readonly mode

* document kvstore versions
2021-05-25 20:57:28 +02:00
Kim De Mey 8abe6b7144
Add support for discv5 talk protocols (#357) 2021-05-20 09:49:46 +02:00
Jacek Sieka 1995afb87e
kvstore fixes (#350)
Storing large blobs in a "WITHOUT ROWID" table turns out to be extremely
slow when the tree must be rebalanced.

* Split out keystore capability into separate interface, making each
keystore a separate instance
* Disable "WITHOUT ROWID" optimization by default
* Implement prefix lookup that allows iterating over all values with a
certain prefix in their key
2021-05-17 15:55:57 +02:00
jangko ea8530f6a0
EIP-2718: fixes BlockBody rlp encoding 2021-05-17 10:07:05 +07:00
jangko d0eb2de328
eip2718: protect against malformed rlp in tx sequence decoder 2021-05-16 19:15:48 +07:00
jangko ef49a6f50a
fixes EIP2718/EIP2930 misinterpretation
the new Tx format (txType || txPayload) needs envelope
encoding at EthBlock level, not at the Tx level itself.
2021-05-16 12:10:09 +07:00
jangko 343d4f25e1
fix eip2718 bug
- fixes bug in rlp decoder of Receipt
- fixes bug in rlp decoder of Transaction
- refactor rlpEncode for AccessListTx in eth/common/transaction.nim
2021-05-15 14:34:43 +07:00
jangko b6b6f3dec7
implement txHashNoSignature for EIP2817's AccessListTx 2021-05-15 08:18:19 +07:00
jangko 61d5327f55
implement eip2718: transaction type envelope
also add more test cases for both tx and receipt
rlp encoding decoding
2021-05-15 08:18:08 +07:00
Jacek Sieka 8890175b6a remove WITHOUT ROWID
This is a minimal performance hotfix for storing large blobs in kvstore
2021-05-14 20:04:14 +03:00
Kim De Mey d05cb5d3bd
Fix raw Exceptions in hexary caused by forward declarations (#349)
* Fix raw Exceptions in hexary caused by forward declarations

* Fix raw Exceptions in trie/db caused by forward declarations

* And now we can remove those db Proc CatchableError raises
2021-05-11 17:32:47 +02:00
kdeme 00a45a7b91
Remove inline pragmas 2021-05-11 09:59:58 +02:00
kdeme 81f0a56ebd
Add/update bunch of license headers 2021-05-11 09:37:33 +02:00
kdeme 755729c6a1
Fix several compiler warnings
Mostly replacing deprecated calls
2021-05-11 09:24:23 +02:00
kdeme bcb58216d1
Add CatchableErrors where needed because of db backends used
In nim-eth this will not fail, as they are base, not implemented
methods. In for example Nimbus-eth1 it will.
2021-05-07 16:28:48 +02:00
kdeme 90b4724492
Adjust for chronosStrictException usage in rest of eth/p2p 2021-05-06 17:20:54 +02:00
kdeme e10ef19f81
Move push raises to top and add/update license info where needed 2021-04-28 16:20:05 +02:00
kdeme b0474c0d40
Add raises annotations to discovery.nim
And add push raises Defect, remove unneeded gcsafe and remove
all usage of inline.
2021-04-27 21:11:54 +02:00
kdeme a1da5d5e59
Use asyncSpawn instead of asyncCheck so chronos strict makes sense
And additional cleanup:
- Push raises Defect at top
- remove inlines
- remove unneeded gcsafe
- remove usage of deprecated calls
2021-04-27 11:30:08 +02:00
kdeme 9fed10de88
Allow for discv4 chronos strict usage
And group p2p tests that can be run with strict usage along the way.
2021-04-27 10:09:54 +02:00
Jamie Lokier 0f3bb61678
Fix import syntax error in `les_protocol.nim` added by PR #344
`les_protocol.nim` failed to build, due to very silly Nim bugs
nim-lang/Nim#8792 and nim-lang/Nim#17102.

    import
      ../../[rlp, keys], ../../common/eth_types,
      ../[rlpx, kademlia, blockchain_utils], ../private/p2p_types,

The silly part is `../` has to be quoted if it's before a group of files, but
not before a single file.  Most places in PR #344 / 7624153 use the workaround
`".."/` but it was missed in `les_protocol.nim`:

    nimbus-eth1/vendor/nim-eth/eth/p2p/rlpx_protocols/les_protocol.nim(14, 3)
        Error: cannot open file: ../../[rlp,keys]

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-07 18:07:00 +01:00
Jamie Lokier 209171d97d
discv4: Refactor findNode so it can be called directly from a test
Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-06 18:41:47 +01:00
Kim De Mey 762415319c
Add build_dcli target and add it to CI (#344)
* Add build_dcli target and add it to CI

* Fix local imports for dcli

* And use local imports for all other files too

* Use local imports in tests and rlpx protocols
2021-04-06 13:33:24 +02:00
Jamie Lokier e4b4b7f4af
discv4: Fix Kademlia crash when trying to sync (#342)
Fixes status-im/nim-eth#341, status-im/nimbus-eth1#489.

When using discv4 (Kademlia) to find peers, there is a crash after a few
minutes.  It occurs for most of us on Eth1 mainnet, and everyone on Ropsten.

The cause is `findNodes` being called twice in succession to the same peer,
within about 5 seconds of each other.  ("About" 5 seconds, because Chronos does
not guarantee to run the timeout branch at a particular time, due to queuing
and clock reading delays.)

Then `findNodes` sends a duplicate message to the peer and calls
`waitNeighbours` to listen for the reply.  There's already a `waitNeighbours`
callback in a shared table, so that function hits an assert failure.

Ignoring the assert would be wrong as it would break timeout logic, and sending
`FindNodes` twice in rapid succession also makes us a bad peer.

As a simple workaround, just skip `findNodes` in this state and return a fake
empty `Neighbours` reply.  This is a bit of a hack as `findNodes` should not be
called like this; there's a logic error at a higher level.  But it works.

Tested for about 4 days constant operation on Ropsten.  The crash which occured
every few minutes no longer occurs, and discv4 keeps working.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
2021-04-02 23:29:02 +02:00
Kim De Mey c5dace27ca
Use chronos http server for dcli metrics and remove insecure compile flag (#343)
And add cfg file to default have runtime log filtering on
2021-04-02 17:29:38 +02:00
Kim De Mey ae4177851f
Remove no longer needed try/except (#339) 2021-03-26 17:49:03 +01:00
Jacek Sieka 698b3bea88
Exception tracking (#334)
* Exception tracking

https://github.com/status-im/nim-chronos/pull/166

* Remove redundant raises annotation

Co-authored-by: kdeme <kim.demey@gmail.com>
2021-03-25 18:12:11 +01:00
Kim De Mey 8c27f291f5
Add raises annotation to the FilterMsgHandler proc type (#337) 2021-03-25 15:06:12 +01:00
Kim De Mey 0be863d2de
Add raises annotations to make exception tracking work (#336)
See https://github.com/status-im/nim-chronos/pull/166
2021-03-24 12:52:09 +01:00
Jacek Sieka 16802c0e52
Fix minor exception effect issues (#335)
Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.
2021-03-23 07:47:55 +01:00
Kim De Mey be5e088b21
Move parsing of nat cli option out of setupAddress (#331)
* Move parsing of nat cli option out of setupAddress

* Change some proc to func
2021-03-05 21:23:54 +01:00
Kim De Mey 0700ec770f
Improve network address discovery / NAT setup (#323)
* Add search for best route and refactor setupNat to setupAddress

* Update setupAddress and make enr ports in discovery optional

* Add specific error log when no route is found
* Use bindIP if it is public
* Adjust some log levels
2021-03-02 17:13:29 +01:00
jangko dda2bec8aa turn networkId into distinct uint 2021-02-13 17:43:17 +07:00
Zahary Karadjov e8c9691b35
awaitWithTimeout for Future[void] 2021-02-04 16:45:50 +02:00
Kim De Mey a339944bcf
Some discv5 cleanup / refactor (#326)
* Use Address in HandshakeKey

* Refactor + comments + dcli use queryRandom

* Rename types.nim to messages.nim
2021-02-02 22:47:21 +01:00
kdeme 4e58eb48ce Add comments on enrAutoUpdate and add metric 2021-02-01 20:30:55 +02:00
kdeme bfbad64d83 Add update of ENR on newly discovered address 2021-02-01 20:30:55 +02:00
kdeme 689eef2c11 Move IpAddress into the pong message 2021-02-01 20:30:55 +02:00
kdeme d33d27c2ee Small improvements and cleanup 2021-02-01 20:30:55 +02:00
kdeme e43ee6ef9c Add ip address voting through pong responses 2021-02-01 20:30:55 +02:00
Jacek Sieka 51a8795e56
one upnp is enough (#324)
we shouldn't have to recreate the upnp interface every time
2021-01-26 14:12:04 +01:00
kdeme 181bbadcbc
Add more message related metrics 2021-01-13 21:44:17 +01:00
kdeme aa7442c9ff
dcli: actually use the provided listen address 2021-01-11 15:01:02 +01:00
kdeme 9cedbc0cc8 Move code into seedTable and populateTable proc 2021-01-07 10:20:40 +01:00
kdeme ac58a1f35c Only do discovery queries to refresh the table 2021-01-07 10:20:40 +01:00
Jacek Sieka 0f48ccecc0 sqlite: manual checkpointing 2020-12-18 19:30:53 +02:00
kdeme 12ec608efd
Put a limit on the nodes returned on a findnode request 2020-12-17 17:06:42 +01:00
kdeme 21423fca42
Use of discv5.1 findnode request with multiple distances 2020-12-17 15:42:04 +01:00
kdeme a506739b23
Add query proc and use this for recurrent queries
This query proc is similar to the original (faulty) lookup proc.
But as we don't need to look for specific targets, it can be
used still as it gives a quicker and broader search resulting
in more nodes.
2020-12-17 15:23:02 +01:00
kdeme e1acc1ae2d
Avoid sending more requests to node that timed out + request metrics 2020-12-16 00:07:49 +01:00
kdeme 68c9b7b3ad
Fix lookup to sort and query closest nodes 2020-12-15 13:24:57 +01:00
Kim De Mey 0820dbba46
Use one instead of deprecated oneIndex (#317) 2020-12-15 12:41:53 +01:00
Kim De Mey d024ea8e85
Change invalid distance warning to debug (#316)
And also:

* move connected_peers metric so it does get added where it
shouldn't.

* imports cleanup
2020-12-14 12:21:03 +01:00
Mamy Ratsimbazafy 3b15abab10 Address review comment 2020-11-29 11:00:09 +02:00
Mamy André-Ratsimbazafy 482ea988c0 Add support for array[N, byte] in SQLite queries 2020-11-29 11:00:09 +02:00
Zahary Karadjov 49c40c9b95
Restore a commit accidentally omitted from a merged branch 2020-11-27 20:49:08 +02:00
Kim De Mey b4c1391be9
Discv5 ip limits for routing table (#308)
* Add ip limits to routing table and routing table buckets

* Fix order of ip limit check and duplicate check for replacement

* Fix ip limit for node with updated ip in ENR

* Fix bug where address wouldn't update on ENR update

and update some comments

* Reuse some add/remove code in routing table

* Fix seen bug on ENR update in routing table

* Rework addNode to make sure to do address check always

and adjust some logs.

* More documentation on the ip limits in routing table [skip ci]
2020-11-26 18:20:15 +01:00
Kim De Mey b88fef203b
Add info log line in case of no external IP (#313) 2020-11-26 18:19:13 +01:00
Jacek Sieka 114680453f
wait for cancellation to be processed on timeout (#311) 2020-11-17 15:36:41 +01:00
kdeme 18430ef127
Add empty talkreq command to dcli 2020-11-13 12:45:39 +01:00
kdeme 44637cdd8e
Remove support for discovery v5.0 2020-11-13 12:33:07 +01:00
Zahary Karadjov 5dff021cbc
Add a helper type used in web3 and elsewhere to denote blocks 2020-11-12 20:56:31 +02:00
Kim De Mey 71d6dbd1b4
Discv5 small adjustments (#307)
* Use checkedEnumAssign in discv5.1

* Some renaming according to discv5 spec nomenclature
2020-10-29 22:04:37 +01:00
Kim De Mey 3626755529
Improve discovery v5.1 logging (#306) 2020-10-23 16:41:44 +02:00
Kim De Mey 6bdf1b4b0f
Simplify UseDiscv51 compile flag (#305) 2020-10-19 21:04:53 +02:00
Zahary Karadjov 345dac702b Don't use destructors 2020-10-16 21:21:00 +03:00
kdeme cf586c363f Random message size and message size checks 2020-10-16 21:20:47 +03:00
kdeme 9b971a0e14 Improve error logging messages
As we don't need to differentate between errors with an enum
anymore thanks to refactor.
2020-10-16 21:20:47 +03:00
kdeme c28cba3589 Limit request id to maximum 8 bytes 2020-10-16 21:20:47 +03:00
kdeme 301067c869 Fix RequestId encoding/decoding and improved tests
- RequestId to variable rlp byte array
- Do encoding and decoding steps in the message tests
- Do sign and verify in the nonce sign test
2020-10-16 21:20:47 +03:00
kdeme 76c10c9b79 Minimum size of random message fix for interop 2020-10-16 21:20:47 +03:00
kdeme 17ef0b25e0 Update to support the latest discv5.1 specification 2020-10-16 21:20:47 +03:00
kdeme 820a73f96f Store session secrets only on succesful decodeHandshakePacket 2020-10-16 21:20:47 +03:00
kdeme a10ee4cc42 Fix topicquery value 2020-10-16 21:20:47 +03:00
kdeme 44b30ef0cb Add all message types and a basic implementation of talkreq/talkresp 2020-10-16 21:20:47 +03:00
kdeme ce2cd2323c Support findnode with multiple distances in discv5.1 2020-10-16 21:20:47 +03:00
kdeme 8042d72711 Add discovery v5.1 implementation 2020-10-16 21:20:47 +03:00
Clyybber 13089cad4c Fix compilation for 1.4
Since proc params are now turned into syms during sem
we need to "refresh" them for use in another context
2020-10-16 20:06:59 +03:00
Zahary Karadjov cc0d15ccac
Richer Sqlite API
* Adds a strongly typed API for creating custom SQL queries and
  executing them

* Uses destructors to simplify the error handling in the init logic
2020-10-13 21:44:42 +03:00
Zahary Karadjov c103721391
Add support for multiple keyspaces in the Sqlite keystore 2020-10-12 04:02:33 +03:00
Zahary Karadjov de2d43a7e7 Remove some unused code from NBC by making it RLPx-specific 2020-10-05 17:28:58 +03:00
Kim De Mey 6b17531d48
Improvements on dropping of challenges and handling of too large distance (#296)
- drop handshake challenge on invalid handshake
- send empty nodes reponse when distance is > 256
- misc
2020-09-30 09:43:51 +02:00
jangko 6c0bc31ad6
add 'BloomFilter.init' implementation 2020-09-30 12:04:23 +07:00
jangko 0194fb85eb
make 'bloom.incl(h256)' public 2020-09-30 11:57:57 +07:00
jangko 2f37870b6b
make 'bloom.contains' public 2020-09-30 11:01:50 +07:00
Jacek Sieka 3ddb498f2a
close sqlite transactions earlier (#294)
If the database is locked for reading, as it is when step returns ROW,
writes cannot checkpoint the wal leading to ever-increasing wal sizes
and a long delay at shutdown.

By resetting the transaction early, writes become more independent of
reads, memory is released earlier and wal can be checkpointed.
2020-09-21 08:21:47 +02:00
tersec ea0dbb256e
support closing kvstore databases (#293)
* support closing kvstore databases

* clear MemStore databases on close
2020-09-11 13:05:52 +00:00
Kim De Mey c9caafb2a4
Replace memory db with lrucache for temporary storage of sessions (#292) 2020-09-10 14:49:48 +02:00
Dustin Brody 64d880b24d
tweak phrasing to trigger CI to run again 2020-08-24 13:34:45 +02:00
Dustin Brody 9c9b9d5e2e
WAL pragma setting for in-memory databases results in "memory", not "wal" 2020-08-24 13:08:35 +02:00
Dustin Brody e2ccbbb418
use sqlite3 write-ahead logging 2020-08-24 12:37:40 +02:00
Zahary Karadjov 73c1bb817c
Adapt to latest snappy; Avoid decompression bombs 2020-08-19 14:14:59 +03:00
Zahary Karadjov 5eff071c8e Delegate traceAsyncErrors to Chronicles 2020-08-16 00:52:22 +03:00
jangko f2f988d6d7
fix keyfile using newest scrypt api 2020-08-03 09:28:46 +07:00
jangko 7afa22cf41
add scrypt kdf to keyfile and implement test for it 2020-07-27 19:34:36 +07:00
kdeme f0638eb3f1
Add rlp_decode fuzz test + some corpus generate code 2020-07-20 17:09:15 +02:00
kdeme 4e4836a0fe
Add discv5 fuzzing test targets 2020-07-20 14:17:44 +02:00
kdeme ec260dcfe4
Add & update tests 2020-07-17 22:48:14 +02:00
kdeme 1eae8f93f9
Fix empty ENR list in auth-response properly + potentially request new ENR on revalidation 2020-07-17 16:18:50 +02:00
kdeme 84fd39a8f8
Update enrs in buckets and replacement caches and fix empty ENR list in AuthResponse 2020-07-17 11:14:08 +02:00
kdeme c7ac9f54b3
Refactor the findNode code + add tests 2020-07-15 15:27:22 +02:00
kdeme 33f72e77bc
Add duplicate and distance checks in Nodes message 2020-07-15 11:15:50 +02:00
Kim De Mey 7febbec673
Use brHmacDrbgGenerate for all random operations in discovery (#272) 2020-07-13 14:34:53 +02:00
Kim De Mey f3de959261
Discv5 readme (#278)
* Add discovery v5 readme + test_discv5 task

* Move hkdf tests to test file and add to nimble task
2020-07-12 23:53:27 +02:00
Kim De Mey 0888667ac0
Comments and cleanup (#276)
* Fix dcli + add more comments

* Fix pong enr sequence number + varia cleanup

* Send randomData in case no handshake was done yet. Fix #277
2020-07-12 17:25:18 +02:00
Kim De Mey 28a8d52308
Fix enr deserialization crash + more tests + fuzz test (#274)
* Fix enr deserialization crash + more tests + fuzz test

* CI: Install depsOnly
2020-07-11 01:28:53 +02:00
Eugene Kabanov e64efc5dbb
Fix LSWAP problem. (#275) 2020-07-10 23:30:34 +02:00
Kim De Mey 7f77ba2308
Fix formatFloat issue (#273) 2020-07-10 14:17:06 +02:00
kdeme eeb958e834
Add comments [skip ci] 2020-07-08 14:56:56 +02:00
kdeme 4f3df3c9b2
Change update back to use var parameter of Record 2020-07-08 14:28:54 +02:00
kdeme d3db83fa0a
Extra test on newProtocol 2020-07-08 13:16:07 +02:00
kdeme 95a09fdf7f
Get rid of some duplicate code 2020-07-08 12:14:00 +02:00
kdeme 0fb21e72d7
Add updateEnr and allow for table constructor usage in newProtocol 2020-07-08 11:45:58 +02:00
kdeme 72420d7f17
Allow for passing in previous enr at discovery protocol creation 2020-07-07 23:39:32 +02:00
kdeme 57302fcf52
Allow for multiple enr fields to be inserted + enr update proc 2020-07-07 22:48:26 +02:00
kdeme 3bbe757fe3
Add record size check on init + add insertFieldPair call + tests 2020-07-07 17:19:15 +02:00
Jacek Sieka 484fbcab1b
use bearssl rng throughout (#265)
* use bearssl rng throughout

* seeder can fail

* imports and exports

* modules, sigh

* one more try

* move var

* even fewer thread vars

* remove out-of-date genrated files
2020-07-07 10:56:26 +02:00
jangko b5cabdfde4
add getForkId interface to AbstractChainDB 2020-07-02 11:09:48 +07:00
kdeme ffb7923370
Use different try/except construct to enforce returning on error 2020-06-30 21:58:23 +02:00
kdeme 23aa13b03d
Return on getMessage failure and do address conversion afterwards 2020-06-30 15:33:33 +02:00
Kim De Mey 9a467225fb
Add routing table metrics + tweaks + fixes (#261)
- routing table metrics + option in dcli
- only forward "seen" nodes on a findNode request
- setJustSeen & replace on ping AND findnode
- self lookup only at start
- revalidate 10x more
- use bitsPerHop (b) of 5
- small fix in resolve
- small fix in bucket split
2020-06-30 13:35:15 +02:00
Kim De Mey 0d591c6423
Merge pull request #247 from status-im/routing-table
Routing table work
2020-06-24 22:29:41 +02:00
narimiran ebabd6f76d more fixes 2020-06-24 22:20:50 +02:00
Miran 0424a66aa2
fix deprecated 'rand' (#259) 2020-06-24 16:06:07 +02:00
kdeme e28f3f987c
routing table: resolve some TODOs 2020-06-24 12:29:59 +02:00
kdeme ba19465892
Functional replacement cache 2020-06-23 16:11:58 +02:00
kdeme ceb4a20463 Fix depth calculation for bucket splitting 2020-06-23 09:35:12 +02:00
kdeme 2d7b3440f2 make bitsPerHop configurable + add routing table tests 2020-06-23 09:35:12 +02:00
kdeme 6c85a48b4c Fix bucket ordering and add comments on this 2020-06-23 09:35:12 +02:00
kdeme 7e35b329b4 Add limit to the replacement cache 2020-06-23 09:35:12 +02:00
Jacek Sieka 61feae0f21
secp: requiresInit updates (#258)
* secp: requiresInit updates

* fixup!

* clean up mapconverts
2020-06-22 18:07:48 +02:00
Zahary Karadjov 5c6d8ccb36 Add a helper from NBC 2020-06-22 17:32:32 +03:00
Zahary Karadjov 964a380191 Stricter error handling 2020-06-22 17:32:32 +03:00
Ștefan Talpalaru 66e2dc7ba3
ignoreSignalsInThread() 2020-06-21 01:32:13 +02:00
jangko f85c8303ee add goerli bootnodes 2020-06-19 12:15:05 +03:00
Kim De Mey 225a9ad41c
Add predicate filter option for randomNodes (#251)
* Add predicate filter option for randomNodes

* Further ValidIpAddress fixes

* Add gcsafe/noSideEffect and add test case
2020-06-11 21:24:52 +02:00
kdeme 67decc3a85 Clear after aes encrypt/decrypt 2020-06-09 11:10:16 +02:00
kdeme 1e1fd70649 Metrics with labels 2020-06-09 11:10:16 +02:00
Kim De Mey 1c77cc1d3d
Fix discovery after introduction of ValidIpAddress (#248) 2020-06-09 11:09:35 +02:00