Commit Graph

142 Commits

Author SHA1 Message Date
Adam Babik 913dbfca9b
Support Mail Server data synchronization (#1302)
These changes add a support for syncing data between two Mail Servers. It does not give an easy way to start syncing. This will be solved in the next PR.
2018-12-06 10:48:28 +01:00
Dmitry Shulyak a609b468fe
Mail peer store and connection manager (#1295)
This change implements connection manager that monitors 3 types of events:
1. update of the selected mail servers
2. disconnect from a mail server
3. errors for requesting mail history

When selected mail servers provided we will try to connect with as many as possible, and later disconnect the surplus. For example if we want to connect with one mail server and 3 were selected, we try to connect with all (3), and later disconnect with 2. It will to establish connection with live mail server faster.

If mail server disconnects we will choose any other mail server from the list of selected. Unless we have only one mail server. In such case we don't have any other choice and we will leave things as is.

If request for history was expired we will disconnect such peer and try to find another one. We will follow same rules as described above.

We will have two components that will rely on this logic:
1. requesting history

If target peer is provided we will use that peer, otherwise we will request history from any selected mail server that is connected at the time of request.

2. confirmation from selected mail server

Confirmation from any selected mail server will bee used to send a feedback that envelope was sent.

I will add several extensions, but probably in separate PRs:
1. prioritize connection with mail server that was used before reboot
2. disconnect from mail servers if history request wasn't expired but failed.
3. wait some time in RequestsMessage RPC to establish connection with any mail server

Currently this feature is hidden, as certain changes will be necessary in status-react. 

partially implements: https://github.com/status-im/status-go/issues/1285
2018-12-05 15:57:05 +02:00
Pedro Pombeiro cddf2e1c6c Change `NotifyUsers` to allow only data JSON element. Part of status-im/status-react#6772 2018-11-30 13:26:27 +01:00
Dmitry Shulyak aac706fe4c
Use confirmation that bundle is delivered for sending feedback to user (#1284)
* Pull whisper confirmations changes

* Use batch confirmations as a signal that envelope was sent into the network
2018-11-27 08:30:15 +02:00
Adam Babik 52a1bdfed6
Upgrade geth 1.8.17 plus add metrics during compilation time (#1273)
This commit updates geth to 1.8.17 and adds a possibility to enable metrics during compilation time.

The cascade of issues forced us to upgrade geth to 1.8.17 in order to allow enabling metrics during compilation time. 1.8.17 introduced `NodeID` refactoring and `enode` package which affected our peers pool and integration with Discovery V5.
2018-11-14 08:03:58 +01:00
Andrea Maria Piana ee3c05c79b
Change handling of skipped/deleted keys & add version (#1261)
- Skipped keys

The purpose of limiting the number of skipped keys generated is to avoid a dos
attack whereby an attacker would send a large N, forcing the device to
compute all the keys between currentN..N .

Previously the logic for handling skipped keys was:

- If in the current receiving chain there are more than maxSkip keys,
throw an error

This is problematic as in long-lived session dropped/unreceived messages starts
piling up, eventually reaching the threshold (1000 dropped/unreceived
messages).

This logic has been changed to be more inline with signals spec, and now
it is:

- If N is > currentN + maxSkip, throw an error

The purpose of limiting the number of skipped keys stored is to avoid a dos
attack whereby an attacker would force us to store a large number of
keys, filling up our storage.

Previously the logic for handling old keys was:

- Once you have maxKeep ratchet steps, delete any key from
currentRatchet - maxKeep.

This, in combination with the maxSkip implementation, capped the number of stored keys to
maxSkip * maxKeep.

The logic has been changed to:

- Keep a maximum of MaxMessageKeysPerSession

and additionally we delete any key that has a sequence number <
currentSeqNum - maxKeep

- Version

We check now the version of the bundle so that when we get a bundle from
the same installationID with a higher version, we mark the previous
bundle as expired and use the new bundle the next time a message is sent
2018-11-05 20:00:04 +01:00
Igor Mandrigin ba504e99c4
Mailserver: return error response. (#1244) 2018-10-18 12:25:00 +02:00
b00ris 820217ea7a added ulc patch (#1239) 2018-10-16 14:27:11 +03:00
Igor Mandrigin 1136176f4a Make statusd compile with go 1.11 (#1225)
* upgrading github.com/rjeczalik/notify to the latest release
2018-10-04 17:57:39 +02:00
Dmitry 5b551c67fa Use status-im whisper/whisperv6 module everywhere 2018-10-03 09:27:51 +03:00
Adam Babik ac8da3c0f0
Use patched go-ethereum from our fork instead of patching it on-the-fly (#1184)
* update to 1.8.16
2018-09-27 21:16:15 +02:00
Dmitry Shulyak 529e18af33
Update geth to v1.8.15 (#1213)
* Update geth to v1.8.15

* Apply patches
2018-09-25 11:55:54 +03:00
Andrea Maria Piana dcaf8caed0
Add x3dh key exchange (#1127)
* Add x3dh key exchange

* Encrypt using the double ratchet

* Multi device with auto-pairing

* Add pfs enabled flag
2018-09-24 20:07:34 +02:00
Dmitry aab84e53dc Ephemeral rendezvous client generates key using seckp256k1 curve 2018-09-24 13:37:06 +03:00
Dmitry 6f937d4ddf Update rendezvous client to a latest version 2018-09-13 09:50:38 +03:00
Adam Babik bf1d1ee0ea
Remove EnvelopeTranced and add Whisper metrics as a patch (#1177) 2018-08-31 10:09:38 +02:00
Adam Babik a571eeb7a4
add patch 0036-fix-whisper-expire-loop (#1175) 2018-08-28 10:49:08 +02:00
Dmitry Shulyak f150d678de
Update geth to 1.8.14 (#1171)
* Update to geth v1.8.14

* Remove patches that were merged upstream

* Apply patches before 0016

* Fix 0016 and apply it

* Apply everything else

* Pass gas limit as a second argument to simulated backend
2018-08-27 11:22:21 +03:00
Dmitry Shulyak 2cfcde0283
Remove patches to discovery v5 (#1144) 2018-08-24 11:24:22 +03:00
Dmitry Shulyak d8b5ba7fbf
Add generated cht for ropsten and mainnet (#1166) 2018-08-24 06:05:06 +03:00
Dmitry bfdf823628 Add delay to refresh cycle with patch 2018-08-15 04:25:18 +03:00
Dmitry 6c6fb4d0b3 Remove patch 0031 2018-08-15 04:25:18 +03:00
Adam Babik ecc39735f5
Upgrade geth to 1.8.13 (#1140) 2018-08-07 15:31:06 +02:00
Dmitry eeca435064 Add rendezvous implementation for discovery interface
Update vendor

Integrate rendezvous into status node

Add a test with failover using rendezvous

Use multiple servers in client

Use discovery V5 by default and test that node can be started with rendezvous discovet

Fix linter

Update rendezvous client to one with instrumented stream

Address feedback

Fix test with updated topic limits

Apply several suggestions

Change log to debug for request errors because we continue execution

Remove web3js after rebase

Update rendezvous package
2018-07-25 15:10:57 +03:00
Ivan Daniluk 801053a17d
Remove jail package (#1106) 2018-07-24 12:05:21 +02:00
Ivan Daniluk 93210061ad
Add goroutines count to geth metrics (#1077) 2018-07-03 20:15:16 +02:00
Adrià Cidre f19e3eae8e
Move TestSentEnvelope to status-scale repo (#1071) 2018-07-02 22:29:03 +02:00
Andrea Franz 809db97e54
mailserver pagination (#1039)
* mailserver sends envelopes in descending order

* add limit value in mailserver request payload

* mailserver sends messages up to the limit specified in the request

* update Archive method to return key and error

* processRequest returns the next page cursor

* add cursor to mailserver request

* add limit and cursor to request payload

* fix request limit encoding

* wait for request completed event in TrackerSuite/TestRequestCompleted

* add cursor to mailserver response

* fix cursor position in payload

* add e2e test for mail server pagination

* validate mail server response size

* remove old limitReached var

* fix lint warnings

* add whisper patch

* fix tests after rebase

* check all return values to avoid lint warnings

* check that all messages have been retrieved after 2 paginated requests

* fix lint warnings

* rename geth patch

* merge mailserver patches into one

* add last envelope hash to mailserver response and EventEnvelopeAvailable event

* update whisper patch

* add docs to MailServerResponse

* update whisper patch

* fix tests and lint warnings

* send mailserver response data on EventMailServerRequestCompleted signal

* update tracker tests

* optimise pagination test waiting for mailserver to archive only before requesting

* rollback mailserver interface changes

* refactoring and docs changes

* fix payload size check to determine if a limit is specified

* add more docs to the processRequest method

* add constants for request payload field lengths

* add const noLimits to specify that limit=0 means no limits
2018-07-02 09:38:10 +02:00
Andrea Franz b913c98573
remove formatting from metrics.ResettingTimer when requested in raw format (#1052) 2018-06-27 11:46:50 +02:00
Pedro Pombeiro 3b052098fe Update to geth 1.8.11. Closes #1033
- Replace deprecated common.Hex with hexutil.Encode.
- Remove upstreamed 0010-geth-17-fix-npe-in-filter-system.patch.
- Remove upstreamed 0020-discv5-metrics.patch.
- Remove upstreamed 0026-ethdb-error-deadlock.patch.
- Update goleveldb to same version used by geth 1.8.11.
- Update PublicTransactionPoolAPI.GasPrice return type to match type in internal geth interface.
2018-06-22 18:04:13 +02:00
Adrià Cidre 51ab9053ab
[#1040] Patch go-ethereum reducing from Warn to Info "message loop" logs (#1048) 2018-06-22 11:57:52 +02:00
Andrea Franz fa390a52ae
RequestHistoricMessages response (#1009)
* refactor TestRequestMessageFromMailboxAsync to use s.requestHistoricMessages helper

* send p2pRequestResponseCode from mailserver

* send p2p message response to after sending all historic messages

* mailserver sends `whisper.NewSentMessage` as response

* add mailserver Client and p2pRequestAckCode watchers

* send event with envelopeFeed when p2pRequestAckCode is received

* test request completed event in tracker

* rename mailserver response events and code to RequestCompleteCode

* wait for mailserver response in e2e test

* use SendHistoricMessageResponse method name for mailserver response

* fix lint warnings

* add mailserver request expiration

* send mailserver response without envelope

* add `ttl` to Request struct in shhext_requestMessages

* test that tracker calls handler.MailServerRequestExpired

* add geth patch

* rename TTL to Timeout

* split tracker.handleEvent in multiple methods
2018-06-15 17:12:31 +02:00
Adrià Cidre 8141657ad7
[#1018] Add a flag to disable shell colors and move verbose logs to debug level (#1020) 2018-06-12 09:49:38 +02:00
Pedro Pombeiro 3f1b2fa57f Update go-duktape.v3 revision to include submitted patch
- Gets rid of noisy build warnings
2018-06-11 15:53:56 +02:00
Adam Babik b5f05b1ab5
Fix flaky TestSingleTopicDiscoveryWithFailover and others (#992)
Other changes:
* needed to patch that loop implementation in Discover V5 implementation in go-ethereum,
* fixed TestStatusNodeReconnectStaticPeers,
* fixed TestBackendAccountsConcurrently.
2018-05-26 09:37:13 +02:00
Dmitry Shulyak ed1cdf0418
Custom status bootnodes (#968)
This change makes invalidation mechanism more aggressive. With a primary goal to invalidate short living nodes faster. In current setup any node that became known in terms of discovery will stay in this state until it will fail to respond to 5 queries. Removing them earlier from a table allows to reduce latency for finding required nodes.

The second change, one adds a version for discovery, separates status dht from ethereum dht.
After we rolled out discovery it became obvious that our boot nodes became spammed with irrelevant nodes. And this made discovery process very long, for example with separate dht discovery takes ~2s, with mutual dht - it can take 1m-10m and there is still no guarantee to find a max amount of peers, cause status nodes is a very small part of whole ethereum infra.

In my understanding, we don't need to be a part of ethereum dht, and lower latency is way more important for us.

Closes: #941
Partially closes: #960 (960 requires futher investigations on devices)
2018-05-18 16:43:07 +03:00
Adam Babik 6da469140f
Refactor in-app metrics (#963)
Remove Prometheus and other metric types and use metrics package from go-ethereum.
2018-05-16 17:36:59 +02:00
Dmitry Shulyak ca962ddaa3 Whisper broadcast loop spams way too hard with debug level 2018-05-16 17:54:40 +03:00
Dmitry Shulyak 5aae87aba8
Prevent sending messages to flaky peers (#917) 2018-05-15 20:08:31 +03:00
Adrià Cidre 08b4d515c6
[#ISSUE-937] : Move mail server implementation to status-go repo (#942)
* [#ISSUE-937] : Move mail server implementation to status-go repo

* Applying dep-ensure
2018-05-11 21:43:07 +02:00
Dmitry Shulyak 7aa508765e Set timesource from config (#915) 2018-05-08 13:38:54 +03:00
Dmitry Shulyak 84cb5ca917 Ensure that wg.Add is called before wg.Wait
Now if Add is to be called it will be called before Wait, this is achieved
by doing following:
- if cancel() gets lock first and closes channelCh before spawnSync is called
we will exit right away
- if not than we will ensure that we hold a lock until syncers are spawned
so that cancel() will be blocked for this time and it will prevent whole Terminate() from
progressing
2018-05-04 10:28:15 +02:00
Dmitry Shulyak 707221954f
Make whisper tolerant to local time skews (#864)
This change adds adds an ability to use different source of time for whisper:

when envelope is created it is used to set expiry
to track when envelope needs to be expired
This time is then used to check validity of the envelope when it is received. Currently If we receive an envelope that is sent from future - peer will get disconnected. If envelope that was received has an expiry less then now it will be simply dropped, if expiry is less than now + 10*2 seconds peer will get dropped.

So, it is clear that whisper depends on time. And any time we get a skew with peers that is > 20s reliability will be grealy reduced.

In this change another source of time for whisper will be used. This time source will use ntp servers from pool.ntp.org to compute offset. When whisper queries time - this offset will be added/substracted from current time.

Query is executed every 2 mins, queries 5 different servers, cut offs min and max and the computes mean value. pool.ntp.org is resolved to different servers and according to documentation you will rarely hit the same.

Closes: #687
2018-05-04 11:23:38 +03:00
Pedro Pombeiro acbf251a3c Fix race condition in `LDBDatabase.Close()`
- https://jenkins.status.im/job/status-go/job/race-check/54/consoleFull
2018-05-03 20:39:42 +03:00
Frank Mueller 123382cdf1
LevelDB vendor has now the same version as used in go-ethereum 1.8.5 (#883)
* LevelDB version is now same as in go-ethereum

* make dep-ensure untracked LevelDB storage.go
2018-05-01 19:09:19 +02:00
Pedro Pombeiro 7074048fa5 Add patch to fix race condition in `Downloader.Cancel`
- Submitted patch upstream in https://github.com/ethereum/go-ethereum/pull/16585
- Fixes Jenkins build in https://jenkins.status.im/job/status-go/job/race-check/48/
2018-05-01 16:29:27 +02:00
Frank Mueller 30cf19e0ae
Wrong command order (#879) 2018-04-30 23:36:28 +02:00
Igor Mandrigin f4cd8d27b5 Add `shhext_getNewFilterMessages` function to RPC APIs.
This function returns only the new messages from the filter, never
returns the same message for the same user twice.
2018-04-27 15:24:37 +02:00
Pedro Pombeiro a23b607597 Fix race condition in `0016-fix-leveldb-issue.patch`
https://jenkins.status.im/job/status-go/job/race-check/45/
2018-04-26 17:19:32 +02:00
Dmitry Shulyak b37fda7731
Update go-ethereum to 1.8.5 (#854)
* Rebase on 1.8.5

* Remove outdated patches and apply all others

* Use shh_post that returns hash

* Use bloom filter for request to mailserver

* Remove tests for sending messages without subbing first

* Fix deadlock in ethdb

* Expect null if receipt is not yet created

* Subscribe to messages before sending them in whisper test
2018-04-24 18:50:26 +03:00