status-im
/
nim-eth
mirror of https://github.com/status-im/nim-eth.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
648 Commits 25 Branches 0 Tags 6.6 MiB
Commit Graph

499 Commits

Author SHA1 Message Date
Jacek Sieka 68e6aadc29 sqlite3: support option type 
.. and any integer when writing but don't support int32 on reads -
internally sqlite will truncate on overflow which isn't nice.
 2021-05-26 13:32:31 +03:00
Jacek Sieka 2a292cfb62
sqlite: fix readonly mode (#359) 
* sqlite: fix readonly mode

* document kvstore versions
 2021-05-25 20:57:28 +02:00
Kim De Mey 8abe6b7144
Add support for discv5 talk protocols (#357) 2021-05-20 09:49:46 +02:00
Jacek Sieka 1995afb87e
kvstore fixes (#350) 
Storing large blobs in a "WITHOUT ROWID" table turns out to be extremely
slow when the tree must be rebalanced.

* Split out keystore capability into separate interface, making each
keystore a separate instance
* Disable "WITHOUT ROWID" optimization by default
* Implement prefix lookup that allows iterating over all values with a
certain prefix in their key
 2021-05-17 15:55:57 +02:00
jangko ea8530f6a0
EIP-2718: fixes BlockBody rlp encoding 2021-05-17 10:07:05 +07:00
jangko d0eb2de328
eip2718: protect against malformed rlp in tx sequence decoder 2021-05-16 19:15:48 +07:00
jangko ef49a6f50a
fixes EIP2718/EIP2930 misinterpretation 
the new Tx format (txType || txPayload) needs envelope
encoding at EthBlock level, not at the Tx level itself.
 2021-05-16 12:10:09 +07:00
jangko 343d4f25e1
fix eip2718 bug 
- fixes bug in rlp decoder of Receipt
- fixes bug in rlp decoder of Transaction
- refactor rlpEncode for AccessListTx in eth/common/transaction.nim
 2021-05-15 14:34:43 +07:00
jangko b6b6f3dec7
implement txHashNoSignature for EIP2817's AccessListTx 2021-05-15 08:18:19 +07:00
jangko 61d5327f55
implement eip2718: transaction type envelope 
also add more test cases for both tx and receipt
rlp encoding decoding
 2021-05-15 08:18:08 +07:00
Jacek Sieka 8890175b6a remove WITHOUT ROWID 
This is a minimal performance hotfix for storing large blobs in kvstore
 2021-05-14 20:04:14 +03:00
Kim De Mey d05cb5d3bd
Fix raw Exceptions in hexary caused by forward declarations (#349) 
* Fix raw Exceptions in hexary caused by forward declarations

* Fix raw Exceptions in trie/db caused by forward declarations

* And now we can remove those db Proc CatchableError raises
 2021-05-11 17:32:47 +02:00
kdeme 00a45a7b91
Remove inline pragmas 2021-05-11 09:59:58 +02:00
kdeme 81f0a56ebd
Add/update bunch of license headers 2021-05-11 09:37:33 +02:00
kdeme 755729c6a1
Fix several compiler warnings 
Mostly replacing deprecated calls
 2021-05-11 09:24:23 +02:00
kdeme bcb58216d1
Add CatchableErrors where needed because of db backends used 
In nim-eth this will not fail, as they are base, not implemented
methods. In for example Nimbus-eth1 it will.
 2021-05-07 16:28:48 +02:00
kdeme 90b4724492
Adjust for chronosStrictException usage in rest of eth/p2p 2021-05-06 17:20:54 +02:00
kdeme e10ef19f81
Move push raises to top and add/update license info where needed 2021-04-28 16:20:05 +02:00
kdeme b0474c0d40
Add raises annotations to discovery.nim 
And add push raises Defect, remove unneeded gcsafe and remove
all usage of inline.
 2021-04-27 21:11:54 +02:00
kdeme a1da5d5e59
Use asyncSpawn instead of asyncCheck so chronos strict makes sense 
And additional cleanup:
- Push raises Defect at top
- remove inlines
- remove unneeded gcsafe
- remove usage of deprecated calls
 2021-04-27 11:30:08 +02:00
kdeme 9fed10de88
Allow for discv4 chronos strict usage 
And group p2p tests that can be run with strict usage along the way.
 2021-04-27 10:09:54 +02:00
Jamie Lokier 0f3bb61678
Fix import syntax error in `les_protocol.nim` added by PR #344 
`les_protocol.nim` failed to build, due to very silly Nim bugs
nim-lang/Nim#8792 and nim-lang/Nim#17102.

    import
      ../../[rlp, keys], ../../common/eth_types,
      ../[rlpx, kademlia, blockchain_utils], ../private/p2p_types,

The silly part is `../` has to be quoted if it's before a group of files, but
not before a single file.  Most places in PR #344 / 7624153 use the workaround
`".."/` but it was missed in `les_protocol.nim`:

    nimbus-eth1/vendor/nim-eth/eth/p2p/rlpx_protocols/les_protocol.nim(14, 3)
        Error: cannot open file: ../../[rlp,keys]

Signed-off-by: Jamie Lokier <jamie@shareable.org>
 2021-04-07 18:07:00 +01:00
Jamie Lokier 209171d97d
discv4: Refactor findNode so it can be called directly from a test 
Signed-off-by: Jamie Lokier <jamie@shareable.org>
 2021-04-06 18:41:47 +01:00
Kim De Mey 762415319c
Add build_dcli target and add it to CI (#344) 
* Add build_dcli target and add it to CI

* Fix local imports for dcli

* And use local imports for all other files too

* Use local imports in tests and rlpx protocols
 2021-04-06 13:33:24 +02:00
Jamie Lokier e4b4b7f4af
discv4: Fix Kademlia crash when trying to sync (#342) 
Fixes status-im/nim-eth#341, status-im/nimbus-eth1#489.

When using discv4 (Kademlia) to find peers, there is a crash after a few
minutes.  It occurs for most of us on Eth1 mainnet, and everyone on Ropsten.

The cause is `findNodes` being called twice in succession to the same peer,
within about 5 seconds of each other.  ("About" 5 seconds, because Chronos does
not guarantee to run the timeout branch at a particular time, due to queuing
and clock reading delays.)

Then `findNodes` sends a duplicate message to the peer and calls
`waitNeighbours` to listen for the reply.  There's already a `waitNeighbours`
callback in a shared table, so that function hits an assert failure.

Ignoring the assert would be wrong as it would break timeout logic, and sending
`FindNodes` twice in rapid succession also makes us a bad peer.

As a simple workaround, just skip `findNodes` in this state and return a fake
empty `Neighbours` reply.  This is a bit of a hack as `findNodes` should not be
called like this; there's a logic error at a higher level.  But it works.

Tested for about 4 days constant operation on Ropsten.  The crash which occured
every few minutes no longer occurs, and discv4 keeps working.

Signed-off-by: Jamie Lokier <jamie@shareable.org>
 2021-04-02 23:29:02 +02:00
Kim De Mey c5dace27ca
Use chronos http server for dcli metrics and remove insecure compile flag (#343) 
And add cfg file to default have runtime log filtering on
 2021-04-02 17:29:38 +02:00
Kim De Mey ae4177851f
Remove no longer needed try/except (#339) 2021-03-26 17:49:03 +01:00
Jacek Sieka 698b3bea88
Exception tracking (#334) 
* Exception tracking

https://github.com/status-im/nim-chronos/pull/166

* Remove redundant raises annotation

Co-authored-by: kdeme <kim.demey@gmail.com>
 2021-03-25 18:12:11 +01:00
Kim De Mey 8c27f291f5
Add raises annotation to the FilterMsgHandler proc type (#337) 2021-03-25 15:06:12 +01:00
Kim De Mey 0be863d2de
Add raises annotations to make exception tracking work (#336) 
See https://github.com/status-im/nim-chronos/pull/166
 2021-03-24 12:52:09 +01:00
Jacek Sieka 16802c0e52
Fix minor exception effect issues (#335) 
Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.
 2021-03-23 07:47:55 +01:00
Kim De Mey be5e088b21
Move parsing of nat cli option out of setupAddress (#331) 
* Move parsing of nat cli option out of setupAddress

* Change some proc to func
 2021-03-05 21:23:54 +01:00
Kim De Mey 0700ec770f
Improve network address discovery / NAT setup (#323) 
* Add search for best route and refactor setupNat to setupAddress

* Update setupAddress and make enr ports in discovery optional

* Add specific error log when no route is found
* Use bindIP if it is public
* Adjust some log levels
 2021-03-02 17:13:29 +01:00
jangko dda2bec8aa turn networkId into distinct uint 2021-02-13 17:43:17 +07:00
Zahary Karadjov e8c9691b35
awaitWithTimeout for Future[void] 2021-02-04 16:45:50 +02:00
Kim De Mey a339944bcf
Some discv5 cleanup / refactor (#326) 
* Use Address in HandshakeKey

* Refactor + comments + dcli use queryRandom

* Rename types.nim to messages.nim
 2021-02-02 22:47:21 +01:00
kdeme 4e58eb48ce Add comments on enrAutoUpdate and add metric 2021-02-01 20:30:55 +02:00
kdeme bfbad64d83 Add update of ENR on newly discovered address 2021-02-01 20:30:55 +02:00
kdeme 689eef2c11 Move IpAddress into the pong message 2021-02-01 20:30:55 +02:00
kdeme d33d27c2ee Small improvements and cleanup 2021-02-01 20:30:55 +02:00
kdeme e43ee6ef9c Add ip address voting through pong responses 2021-02-01 20:30:55 +02:00
Jacek Sieka 51a8795e56
one upnp is enough (#324) 
we shouldn't have to recreate the upnp interface every time
 2021-01-26 14:12:04 +01:00
kdeme 181bbadcbc
Add more message related metrics 2021-01-13 21:44:17 +01:00
kdeme aa7442c9ff
dcli: actually use the provided listen address 2021-01-11 15:01:02 +01:00
kdeme 9cedbc0cc8 Move code into seedTable and populateTable proc 2021-01-07 10:20:40 +01:00
kdeme ac58a1f35c Only do discovery queries to refresh the table 2021-01-07 10:20:40 +01:00
Jacek Sieka 0f48ccecc0 sqlite: manual checkpointing 2020-12-18 19:30:53 +02:00
kdeme 12ec608efd
Put a limit on the nodes returned on a findnode request 2020-12-17 17:06:42 +01:00
kdeme 21423fca42
Use of discv5.1 findnode request with multiple distances 2020-12-17 15:42:04 +01:00
kdeme a506739b23
Add query proc and use this for recurrent queries 
This query proc is similar to the original (faulty) lookup proc.
But as we don't need to look for specific targets, it can be
used still as it gives a quicker and broader search resulting
in more nodes.
 2020-12-17 15:23:02 +01:00
kdeme e1acc1ae2d
Avoid sending more requests to node that timed out + request metrics 2020-12-16 00:07:49 +01:00
kdeme 68c9b7b3ad
Fix lookup to sort and query closest nodes 2020-12-15 13:24:57 +01:00
Kim De Mey 0820dbba46
Use one instead of deprecated oneIndex (#317) 2020-12-15 12:41:53 +01:00
Kim De Mey d024ea8e85
Change invalid distance warning to debug (#316) 
And also:

* move connected_peers metric so it does get added where it
shouldn't.

* imports cleanup
 2020-12-14 12:21:03 +01:00
Mamy Ratsimbazafy 3b15abab10 Address review comment 2020-11-29 11:00:09 +02:00
Mamy André-Ratsimbazafy 482ea988c0 Add support for array[N, byte] in SQLite queries 2020-11-29 11:00:09 +02:00
Zahary Karadjov 49c40c9b95
Restore a commit accidentally omitted from a merged branch 2020-11-27 20:49:08 +02:00
Kim De Mey b4c1391be9
Discv5 ip limits for routing table (#308) 
* Add ip limits to routing table and routing table buckets

* Fix order of ip limit check and duplicate check for replacement

* Fix ip limit for node with updated ip in ENR

* Fix bug where address wouldn't update on ENR update

and update some comments

* Reuse some add/remove code in routing table

* Fix seen bug on ENR update in routing table

* Rework addNode to make sure to do address check always

and adjust some logs.

* More documentation on the ip limits in routing table [skip ci]
 2020-11-26 18:20:15 +01:00
Kim De Mey b88fef203b
Add info log line in case of no external IP (#313) 2020-11-26 18:19:13 +01:00
Jacek Sieka 114680453f
wait for cancellation to be processed on timeout (#311) 2020-11-17 15:36:41 +01:00
kdeme 18430ef127
Add empty talkreq command to dcli 2020-11-13 12:45:39 +01:00
kdeme 44637cdd8e
Remove support for discovery v5.0 2020-11-13 12:33:07 +01:00
Zahary Karadjov 5dff021cbc
Add a helper type used in web3 and elsewhere to denote blocks 2020-11-12 20:56:31 +02:00
Kim De Mey 71d6dbd1b4
Discv5 small adjustments (#307) 
* Use checkedEnumAssign in discv5.1

* Some renaming according to discv5 spec nomenclature
 2020-10-29 22:04:37 +01:00
Kim De Mey 3626755529
Improve discovery v5.1 logging (#306) 2020-10-23 16:41:44 +02:00
Kim De Mey 6bdf1b4b0f
Simplify UseDiscv51 compile flag (#305) 2020-10-19 21:04:53 +02:00
Zahary Karadjov 345dac702b Don't use destructors 2020-10-16 21:21:00 +03:00
kdeme cf586c363f Random message size and message size checks 2020-10-16 21:20:47 +03:00
kdeme 9b971a0e14 Improve error logging messages 
As we don't need to differentate between errors with an enum
anymore thanks to refactor.
 2020-10-16 21:20:47 +03:00
kdeme c28cba3589 Limit request id to maximum 8 bytes 2020-10-16 21:20:47 +03:00
kdeme 301067c869 Fix RequestId encoding/decoding and improved tests 
- RequestId to variable rlp byte array
- Do encoding and decoding steps in the message tests
- Do sign and verify in the nonce sign test
 2020-10-16 21:20:47 +03:00
kdeme 76c10c9b79 Minimum size of random message fix for interop 2020-10-16 21:20:47 +03:00
kdeme 17ef0b25e0 Update to support the latest discv5.1 specification 2020-10-16 21:20:47 +03:00
kdeme 820a73f96f Store session secrets only on succesful decodeHandshakePacket 2020-10-16 21:20:47 +03:00
kdeme a10ee4cc42 Fix topicquery value 2020-10-16 21:20:47 +03:00
kdeme 44b30ef0cb Add all message types and a basic implementation of talkreq/talkresp 2020-10-16 21:20:47 +03:00
kdeme ce2cd2323c Support findnode with multiple distances in discv5.1 2020-10-16 21:20:47 +03:00
kdeme 8042d72711 Add discovery v5.1 implementation 2020-10-16 21:20:47 +03:00
Clyybber 13089cad4c Fix compilation for 1.4 
Since proc params are now turned into syms during sem
we need to "refresh" them for use in another context
 2020-10-16 20:06:59 +03:00
Zahary Karadjov cc0d15ccac
Richer Sqlite API 
* Adds a strongly typed API for creating custom SQL queries and
  executing them

* Uses destructors to simplify the error handling in the init logic
 2020-10-13 21:44:42 +03:00
Zahary Karadjov c103721391
Add support for multiple keyspaces in the Sqlite keystore 2020-10-12 04:02:33 +03:00
Zahary Karadjov de2d43a7e7 Remove some unused code from NBC by making it RLPx-specific 2020-10-05 17:28:58 +03:00
Kim De Mey 6b17531d48
Improvements on dropping of challenges and handling of too large distance (#296) 
- drop handshake challenge on invalid handshake
- send empty nodes reponse when distance is > 256
- misc
 2020-09-30 09:43:51 +02:00
jangko 6c0bc31ad6
add 'BloomFilter.init' implementation 2020-09-30 12:04:23 +07:00
jangko 0194fb85eb
make 'bloom.incl(h256)' public 2020-09-30 11:57:57 +07:00
jangko 2f37870b6b
make 'bloom.contains' public 2020-09-30 11:01:50 +07:00
Jacek Sieka 3ddb498f2a
close sqlite transactions earlier (#294) 
If the database is locked for reading, as it is when step returns ROW,
writes cannot checkpoint the wal leading to ever-increasing wal sizes
and a long delay at shutdown.

By resetting the transaction early, writes become more independent of
reads, memory is released earlier and wal can be checkpointed.
 2020-09-21 08:21:47 +02:00
tersec ea0dbb256e
support closing kvstore databases (#293) 
* support closing kvstore databases

* clear MemStore databases on close
 2020-09-11 13:05:52 +00:00
Kim De Mey c9caafb2a4
Replace memory db with lrucache for temporary storage of sessions (#292) 2020-09-10 14:49:48 +02:00
Dustin Brody 64d880b24d
tweak phrasing to trigger CI to run again 2020-08-24 13:34:45 +02:00
Dustin Brody 9c9b9d5e2e
WAL pragma setting for in-memory databases results in "memory", not "wal" 2020-08-24 13:08:35 +02:00
Dustin Brody e2ccbbb418
use sqlite3 write-ahead logging 2020-08-24 12:37:40 +02:00
Zahary Karadjov 73c1bb817c
Adapt to latest snappy; Avoid decompression bombs 2020-08-19 14:14:59 +03:00
Zahary Karadjov 5eff071c8e Delegate traceAsyncErrors to Chronicles 2020-08-16 00:52:22 +03:00
jangko f2f988d6d7
fix keyfile using newest scrypt api 2020-08-03 09:28:46 +07:00
jangko 7afa22cf41
add scrypt kdf to keyfile and implement test for it 2020-07-27 19:34:36 +07:00
kdeme f0638eb3f1
Add rlp_decode fuzz test + some corpus generate code 2020-07-20 17:09:15 +02:00
kdeme 4e4836a0fe
Add discv5 fuzzing test targets 2020-07-20 14:17:44 +02:00
kdeme ec260dcfe4
Add & update tests 2020-07-17 22:48:14 +02:00
kdeme 1eae8f93f9
Fix empty ENR list in auth-response properly + potentially request new ENR on revalidation 2020-07-17 16:18:50 +02:00